wiki:import_ida
Last modified 8 years ago Last modified on 03/18/12 16:13:39

Earthworm Module: import_ida

Contributed by: David Chavez

Function

This program enables an Earthworm to retrieve and process continuous trace data from numerous IDA data servers.

Details

Import-Export Overview

Import_ida acquires data from one or more Project IDA data servers, reformats them into TracePackets and loads them into an Earthworm transport ring. The communication with the data server(s) is done using the idatap-1.2.1 toolkit available from the University of California, San Diego via

ftp://idahub.ucsd.edu:/pub/pickup/idatap-1.2.1.tar.Z

In the import_ida.d parameter file the user specifies a list of servers with which to connect, and the list of stations and channels to acquire from each server. The program forks separate processes to independently manage all communication with the various servers. These processes communicate through a pipe to distinct threads in the parent. As data packets are acquired, each child process passes them back to its parent thread, which loads the packets into a message queue that feeds the transport ring via the main thread. In addition, any child log messages are formatted into special packets that are also passed back to the main thread and logged using the standard Earthworm facilities.

Network timeouts are dealt with transparently to the application by the idatap library, which will retry and reconnect as necessary until successfull or until a more serious error occurs, at which point import_ida assumes the responsibility for reestablishing the connection. The import_ida.d parameter file includes an option for specifying the policy to follow when reconnecting after a loss of communication. One may chose to request the next packet in sequence (to minimize gaps in the acquired data set) or to request the most recently available data (to minimize delays).

Example Configuration File

# @(#)import_ida.d	1.1 12/26/97
# import_ida parameter file

#  Basic Earthworm setup:

MyModuleId         MOD_IMPORT_IDA # module id for this process
OutRing            WAVE_RING      # output ring
HeartBeatInterval  30             # seconds between heartbeats

# We need to assign unique pin numbers to each net/sta/chan triple that
# gets sent to the output ring.  Give the name of the file that contains
# this information.  The format of this file must be the same as what
# pick_ew expects.  You can create your own file with IDA specific
# entries (see ida.sta for an example), or you can add the required 
# information to the pick_ew station file.  The picker parameters are
# not required here, and as far as this program is concerned those fields
# may be left blank.

PinnoFile          ida.sta

# It is up to the user to insure that there are entries in the above file
# for all expected net/sta/chan triples, however the program will work even
# if the database is incomplete.  In such a case, an unresolved net/sta/chan
# will be assigned a default pin number of 12345.  You can override this
# default by specifying an alternate value.

DefaultPinno       12345

# Define the policy to follow when reconnecting after circuit timeouts.
# There are two choices: MINIMAL_GAP, or MINIMAL_DELAY.  If the former
# is selected then the program will request the flow start at the end of
# whatever has been received so far.  If the latter is selected then the
# program will request the flow start with the most recent packet available
# on the server.

Policy             MINIMAL_DELAY

# The network code for Project IDA stations is II, and that is the default
# code which gets loaded into the packet headers.  If you want to override
# this with some other code, enter it here.

Network            II

# List the server(s) and stream(s) that are desired.
# The station channel specifications should follow the syntax
# described in ftp://idahub.ucsd.edu:/pub/pickup/idatap-1.2.1.tar.Z
#
# You can give either the fully qualified host name, or its IP address
# in the usual decmial dot notation (eg, 111.22.333.44)
#
#          Hostname              Station/channel list

server   nnn.xxx.mmm.5        nna:bhz+pfo:bhz+sur:bhz

Helpful Hints