Changes between Version 3 and Version 4 of ew2mseed


Ignore:
Timestamp:
02/26/12 15:58:07 (9 years ago)
Author:
branden
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ew2mseed

    v3 v4  
    66ew2mseed is a standalone module that builds continuous miniSEED day files from a wave_serverV connection 
    77== Details == 
    8 ew2mseed  is a non interactive automatic Earthworm WaveServerV client for archiving continuous nov-overlapping time-ordered waveforms in MiniSEED format.  Like any  Earthworm program, it has a configuration file ew2mseed.d allowing for extensive configuration. This is where we tell it, among other things, which waveservers to interrogate, where and in which format to store the trace output.  
     8ew2mseed  is a non interactive automatic Earthworm !WaveServerV client for archiving continuous nov-overlapping time-ordered waveforms in MiniSEED format.  Like any  Earthworm program, it has a configuration file ew2mseed.d allowing for extensive configuration. This is where we tell it, among other things, which waveservers to interrogate, where and in which format to store the trace output.  
    99  
    1010 
     
    3838 
    3939The considerations for  a complex design described above are the following: [[BR]] 
    40 (a) ew2mseed is intended to grab as much data points from the WaveServer as possible in an automatic mode. [[BR]] 
     40(a) ew2mseed is intended to grab as much data points from the !WaveServer as possible in an automatic mode. [[BR]] 
    4141(b) It is assumes that if the data for the particular date are written into the file of it's directory structure, the data for  all previous days have are also processed if ew2mseed operates constantly. 
    4242 
     
    6565=== Using ew2mseed with multiple !WaveServers === 
    6666 
    67 1. Here we walk a reader through the steps of the ew2mseed program from the standpoint of multiple !WaveServers usage. Multiple !WaveServers are set in the configuration file as key word WaveServer followed by the IP:Port, where IP address can also be human readable computer name which will be resolved by gethostbyname() system call. 
     671. Here we walk a reader through the steps of the ew2mseed program from the standpoint of multiple !WaveServers usage. Multiple !WaveServers are set in the configuration file as key word !WaveServer followed by the IP:Port, where IP address can also be human readable computer name which will be resolved by gethostbyname() system call. 
    6868{{{ 
    6969# frame relay..  
     
    76762. All available !WaveServers are first registered in the configuration structure of ew2mseed.  If we follow the example configuration above, in particular, there will be created a WaveServer linked list containing 3 WaveServers. 
    7777 
    78 3.  ew2mseed calls a function  int processWsAppendMenu (RINGS *rn, WS_MENU_QUEUE_REC *menu) , which uses EW library function wsAppendMenu() in a loop for each WaveServer. wsAppendMenu() can either append a current WaveServer menu or return an error indicating that the connection to the WaveServer is not available. The main task of processWsAppendMenu () is to fill up the WaveServers structure WS_MENU_QUEUE_REC *menu. This structure is later used for getting data snippets. As long as at least a single WaveServer is available, the WS_MENU_QUEUE_REC *menu structure is not NULL and function processWsAppendMenu () returns the number of available WaveServers from the list. If no WaveServers are available, processWsAppendMenu () does not return. In this case it idles for 20 seconds and attempt to connect to WaveServers again. An operator can examine the log file and kill the instance of '''ew2mseed''' manually if he/she realizes that the !WaveServers declared in the configuration will not be available. 
     783.  ew2mseed calls a function  int processWsAppendMenu (RINGS *rn, WS_MENU_QUEUE_REC *menu) , which uses EW library function wsAppendMenu() in a loop for each !WaveServer. wsAppendMenu() can either append a current !WaveServer menu or return an error indicating that the connection to the !WaveServer is not available. The main task of processWsAppendMenu () is to fill up the !WaveServers structure WS_MENU_QUEUE_REC *menu. This structure is later used for getting data snippets. As long as at least a single !WaveServer is available, the WS_MENU_QUEUE_REC *menu structure is not NULL and function processWsAppendMenu () returns the number of available !WaveServers from the list. If no !WaveServers are available, processWsAppendMenu () does not return. In this case it idles for 20 seconds and attempt to connect to !WaveServers again. An operator can examine the log file and kill the instance of '''ew2mseed''' manually if he/she realizes that the !WaveServers declared in the configuration will not be available. 
    7979 
    80 4. Let us suppose that we passed the stage (3) and found out that, for example, two out of three !WaveServers declared in the configuration file are running and their data are available.  We now search through every available WaveServer and create a list of available PSCNLs. If no PSCNLs is available, the program quits. 
     804. Let us suppose that we passed the stage (3) and found out that, for example, two out of three !WaveServers declared in the configuration file are running and their data are available.  We now search through every available !WaveServer and create a list of available PSCNLs. If no PSCNLs is available, the program quits. 
    8181 
    82 5. If we are here, it means that more than 0 !WaveServers provide more than zero PSCNLs.  In other words we entered the main loop of ew2mseed. In the main loop, the core call is to int wsGetTraceBin (TRACE_REQ* getThis, WS_MENU_QUEUE_REC* menu_queue,  int timeout).  wsGetTraceBin() is a library function from WaveClient library and it is declared as being able to extract data from multiple waveServers. Here is an extract from wsGetTraceBin() documentation: 
     825. If we are here, it means that more than 0 !WaveServers provide more than zero PSCNLs.  In other words we entered the main loop of ew2mseed. In the main loop, the core call is to int wsGetTraceBin (TRACE_REQ* getThis, WS_MENU_QUEUE_REC* menu_queue,  int timeout).  wsGetTraceBin() is a library function from WaveClient library and it is declared as being able to extract data from multiple !WaveServers. Here is an extract from wsGetTraceBin() documentation: 
    8383{{{ 
    8484                 Retrieves the piece of raw trace data specified in the  
     
    9090                 in the menu queue will be tried. (http://www.cnss.org/EWAB/libsrc.html) 
    9191}}} 
    92 6. Once in while (when ew2mseed read all data until the endTime of the tank for a particular PSCNL),  ew2mseed calls a function int updateMenu (RINGS *rn, WS_MENU_QUEUE_REC *menu_queue), which updates local copies of menus of !WaveServers and includes/excludes from the WaveServer's linked list those !WaveServers which  became available/unavailable. 
     926. Once in while (when ew2mseed read all data until the endTime of the tank for a particular PSCNL),  ew2mseed calls a function int updateMenu (RINGS *rn, WS_MENU_QUEUE_REC *menu_queue), which updates local copies of menus of !WaveServers and includes/excludes from the !WaveServer's linked list those !WaveServers which  became available/unavailable. 
    9393 
    9494=== Catch-up algorithm (new in version 02-Apr-2002) === 
     
    9696The problem description: ew2mseed receives high volume of data from heterogeneous set of channels; due to the various reasons (connection speed is the most important), some channels are later than the others. We implemented an algorithm which forces  ew2mseed to request more information from the later channels. 
    9797 
    98   1) Each channel has a configuration structure. We add a new integer field "Priority" which indicates  a factor at  which we increase the parameter RecordNumber for a given station. That is, if  Priority  is 2 for a channel, it will poll twice  more data form the WaveServer relative to the channels with GetTraceTimes = 1. At the init stage each channel sets "Priority" to the default 1. 
     98  1) Each channel has a configuration structure. We add a new integer field "Priority" which indicates  a factor at  which we increase the parameter !RecordNumber for a given station. That is, if  Priority  is 2 for a channel, it will poll twice  more data form the !WaveServer relative to the channels with GetTraceTimes = 1. At the init stage each channel sets "Priority" to the default 1. 
    9999 
    100100 2)  We count number of loops over all channels.  After '''!LoopsBeforeService''' (configurable parameter, default value = 50)  production loops  for each channel we compute the TIME of the snippet we currently process and  AVERAGE TIME for all channels. 
    101101 
    102  4) Next for each channel we compute Priority as: if the processing time on the channel is less than  N days  later than the Average, we assign N+1 to GetTraceTimes, if the average time is later than the channel processing time, we set 1 to GetTraceTime. GetTraceTimes is bounded by 1 as the lowest limit and by the configurable parameter '''!PriorityHighWater''' (default value = 5). Next, we increase/decrease the requested time limits for a request to the WaveServerV proportinally to the "Priority" parameter. 
     102 4) Next for each channel we compute Priority as: if the processing time on the channel is less than  N days  later than the Average, we assign N+1 to !GetTraceTimes, if the average time is later than the channel processing time, we set 1 to !GetTraceTime. !GetTraceTimes is bounded by 1 as the lowest limit and by the configurable parameter '''!PriorityHighWater''' (default value = 5). Next, we increase/decrease the requested time limits for a request to the WaveServerV proportinally to the "Priority" parameter. 
    103103 
    104104 5) Goto 2. 
    105105 
    106106=== !StartLatency parameter (new in version 15-May-2003) === 
    107 !StartLatency parameter in hours is used to superseed !StartTime parameter.  The starttime is computed as the current time minus !StartLatency and the resultanant number is used to compute !StartTime. Either !StartTime or StartLatency must be present in the configuration file.  If both of them are present, the program will use the parameter which is below than the other one in the configuration file.  
     107!StartLatency parameter in hours is used to superseed !StartTime parameter.  The starttime is computed as the current time minus !StartLatency and the resultanant number is used to compute !StartTime. Either !StartTime or !StartLatency must be present in the configuration file.  If both of them are present, the program will use the parameter which is below than the other one in the configuration file.  
    108108  
    109109== Helpful Hints ==