Changes between Version 1 and Version 2 of Import-Export


Ignore:
Timestamp:
02/29/12 15:55:10 (8 years ago)
Author:
branden
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Import-Export

    v1 v2  
     1[[PageOutline]] 
     2 
    13= [wiki:Earthworm Earthworm]: Import-Export Overview = 
    24 
     
    3234  This new version of export expects to receive an acknowledgment packet  for every packet it writes to the socket. Creation of this new version  was desired because we observed during the export of continuous trace data that during a communication failure, export was able to successfully write several messages to the (broken?) socket, but import never received them. When the socket was re-established, those messages were lost forever.  
    3335 
    34   Acknowledgments are received asynchronously by the socket-reading thread.  A circular buffer, SendQueue, tracks the status of msgs after they been  pulled from the MessageStacker queue. If the socket is broken/reconnected,  export*ack re-sends any msgs in the SendQueue which haven't been  acknowledged. In normal operation, if export*ack laps the SendQueue and  finds a message that has been sent but not ACKed, it will sleep 10ms and  then check status again, looping until an ACK is received before preparing  to send the next message.  
     36  Acknowledgments are received asynchronously by the socket-reading thread.  A circular buffer, !SendQueue, tracks the status of msgs after they been  pulled from the !MessageStacker queue. If the socket is broken/reconnected,  export*ack re-sends any msgs in the !SendQueue which haven't been  acknowledged. In normal operation, if export*ack laps the !SendQueue and  finds a message that has been sent but not ACKed, it will sleep 10ms and  then check status again, looping until an ACK is received before preparing  to send the next message.  
    3537 
    36   The SendQueue length is configurable, from 1 to 254 messages (default=100).   Using a shorter SendQueue can cause a decrease in the rate at which export will send messages. A slower rate might be desireable so  that an import system is not flooded by data after a prolonged  communication break. In testing with export and import on the same  machine, these are export's approximate maximum data rates: 
     38  The !SendQueue length is configurable, from 1 to 254 messages (default=100).   Using a shorter !SendQueue can cause a decrease in the rate at which export will send messages. A slower rate might be desireable so  that an import system is not flooded by data after a prolonged  communication break. In testing with export and import on the same  machine, these are export's approximate maximum data rates: 
    3739 
    3840{{{ 
     
    4244}}} 
    4345 
    44 Using the default SendQueueLength, export*ack's data rate was not observably different than export_generic's data rate.   
     46Using the default !SendQueueLength, export*ack's data rate was not observably different than export_generic's data rate.   
    4547 
    46   export*ack requires using the new import_ack to communicate properly. Since the receipt of ACKs is asynchronous, we may now have the opposite  problem to that we had before (duplicate data instead of lost data). It  is now possible that import_ack actually received a packet, but export_ack didn't get the ACK before the socket broke. In this is case, export_ack  
    47 will re-send the packet and import_ack will see the same packet more than  once. The system running import_ack must be prepared for duplicate data. 
     48export*ack requires using the new import_ack to communicate properly. Since the receipt of ACKs is asynchronous, we may now have the opposite  problem to that we had before (duplicate data instead of lost data). It  is now possible that import_ack actually received a packet, but export_ack didn't get the ACK before the socket broke. In this is case, export_ack will re-send the packet and import_ack will see the same packet more than  once. The system running import_ack must be prepared for duplicate data. 
    4849 
    4950== import_ack ==