Changes between Version 1 and Version 2 of Import-Export
- Timestamp:
- 02/29/12 15:55:10 (9 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Import-Export
v1 v2 1 [[PageOutline]] 2 1 3 = [wiki:Earthworm Earthworm]: Import-Export Overview = 2 4 … … 32 34 This new version of export expects to receive an acknowledgment packet for every packet it writes to the socket. Creation of this new version was desired because we observed during the export of continuous trace data that during a communication failure, export was able to successfully write several messages to the (broken?) socket, but import never received them. When the socket was re-established, those messages were lost forever. 33 35 34 Acknowledgments are received asynchronously by the socket-reading thread. A circular buffer, SendQueue, tracks the status of msgs after they been pulled from the MessageStacker queue. If the socket is broken/reconnected, export*ack re-sends any msgs in the SendQueue which haven't been acknowledged. In normal operation, if export*ack laps theSendQueue and finds a message that has been sent but not ACKed, it will sleep 10ms and then check status again, looping until an ACK is received before preparing to send the next message.36 Acknowledgments are received asynchronously by the socket-reading thread. A circular buffer, !SendQueue, tracks the status of msgs after they been pulled from the !MessageStacker queue. If the socket is broken/reconnected, export*ack re-sends any msgs in the !SendQueue which haven't been acknowledged. In normal operation, if export*ack laps the !SendQueue and finds a message that has been sent but not ACKed, it will sleep 10ms and then check status again, looping until an ACK is received before preparing to send the next message. 35 37 36 The SendQueue length is configurable, from 1 to 254 messages (default=100). Using a shorterSendQueue can cause a decrease in the rate at which export will send messages. A slower rate might be desireable so that an import system is not flooded by data after a prolonged communication break. In testing with export and import on the same machine, these are export's approximate maximum data rates:38 The !SendQueue length is configurable, from 1 to 254 messages (default=100). Using a shorter !SendQueue can cause a decrease in the rate at which export will send messages. A slower rate might be desireable so that an import system is not flooded by data after a prolonged communication break. In testing with export and import on the same machine, these are export's approximate maximum data rates: 37 39 38 40 {{{ … … 42 44 }}} 43 45 44 Using the default SendQueueLength, export*ack's data rate was not observably different than export_generic's data rate.46 Using the default !SendQueueLength, export*ack's data rate was not observably different than export_generic's data rate. 45 47 46 export*ack requires using the new import_ack to communicate properly. Since the receipt of ACKs is asynchronous, we may now have the opposite problem to that we had before (duplicate data instead of lost data). It is now possible that import_ack actually received a packet, but export_ack didn't get the ACK before the socket broke. In this is case, export_ack 47 will re-send the packet and import_ack will see the same packet more than once. The system running import_ack must be prepared for duplicate data. 48 export*ack requires using the new import_ack to communicate properly. Since the receipt of ACKs is asynchronous, we may now have the opposite problem to that we had before (duplicate data instead of lost data). It is now possible that import_ack actually received a packet, but export_ack didn't get the ACK before the socket broke. In this is case, export_ack will re-send the packet and import_ack will see the same packet more than once. The system running import_ack must be prepared for duplicate data. 48 49 49 50 == import_ack ==