Changes between Initial Version and Version 1 of trig2disk


Ignore:
Timestamp:
02/27/12 17:31:49 (9 years ago)
Author:
branden
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • trig2disk

    v1 v1  
     1[[PageOutline]] 
     2 
     3= [wiki:Earthworm Earthworm] Module: trig2disk = 
     4'''Contributed by: ''' 
     5 
     6== Function == 
     7Trig2disk module reads trigger messages and retrieves waveform data from wave_serverV into AH or SAC format files. 
     8 
     9== Details == 
     10 
     11On startup, trig2disk reads the configuration file named on the command line. Commands in this file set all the parameters used for reading triglist2k messages and writing sac, ah, suds, gse, seisan, or tankplayer format. In the control file, lines may begin with a valid trig2disk command (listed below) or with one of 2 special characters: 
     12{{{ 
     13#  marks the line as a comment (example: # This is a comment). 
     14    
     15@  allows control files to be nested; one control file can be  
     16   accessed from another with the command "@" followed by  
     17   a string representing the path name of the next control file  
     18   (example: @memphis.scnl). 
     19}}} 
     20Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!). 
     21 
     22 
     23=== EXAMPLE CONFIGURATION FILE === 
     24{{{ 
     25# 
     26# Configuration file for trig2disk: 
     27# I listen to trigger messages (TYPE_TRIGLIST), and stuff the implied pieces 
     28# of trace data into an Oracle server. I do this by going to any number 
     29# of WaveServerV's, as listed in this configuration file. 
     30# 
     31MyModuleId                 MOD_TRIG2DISK 
     32RingName                   HYPO_RING 
     33HeartBeatInt               30 
     34LogFile                    1   # 0 means don't create a disc log file. 1=> do. 
     35                               # 2 means log to disk file but not stderr/stdout 
     36# 
     37# What message logos to listen to. Can be more than one. 
     38# The type is hard coded to TYPE_TRIGLIST 
     39# 
     40GetEventsFrom              INST_MENLO  MOD_WILDCARD 
     41 
     42# 
     43# list of ip addresses and ports of the WaveServers we're to use 
     44# 
     45WaveServer                 111.222.33.444     16022 
     46 
     47  
     48 
     49# Optional NEW COMMAND in EW v7.3 Postproc allows running of a program or script that will process new files 
     50# no arguments are sent to the script...It should know where to run and what to do with new files: 
     51# 
     52#Postproc  myscript  # the myscript or program should be in the params directory or an absolute path must be provided. 
     53 
     54 
     55# 
     56# If a WaveServer doesn't talk to us in this  
     57# many seconds, we'll abort that request 
     58# 
     59TimeoutSeconds             30   
     60 
     61# 
     62# Max number of traces we'll ever see in one event 
     63# 
     64MaxTraces                  500   
     65 
     66# 
     67# kilobytes of the largest trace snippet we'll ever have to deal with 
     68# 
     69TraceBufferLen             1000          
     70 
     71# 
     72# Debug switch:  Debug will be logged if uncommented  
     73# 
     74Debug 
     75 
     76# 
     77# SCNL list of stations to write for each trigger message,  these get 
     78# written in addition to SCNLs in the trigger message. 
     79# 
     80#@memphis.scnl 
     81 
     82# 
     83# Minimum number of seconds to save for extra channels that we 
     84# are saving (i.e., those channels not in the TYPE_TRIGLIST message) 
     85# 
     86MinDuration                 30 
     87 
     88# 
     89# number of sample periods after which we declare a gap 
     90# 
     91GapThresh 100 
     92 
     93# Optional queue commands 
     94# Number of trigger messages to hold in queue; default is 10 
     95#QueueSize 10 
     96 
     97# Optional queue dumpfile name, for saving state of queue 
     98QueueFile trig2disk.que 
     99 
     100# Optional delay time: trig2disk will wait this many seconds from the 
     101# time it receives a trigger message until it starts to process the message 
     102DelayTime 5 
     103 
     104 
     105# 
     106# Select format of output data and the directory where it is written 
     107# Only one of the following pairs should be uncommented. 
     108# 
     109# 
     110# SUDS 
     111# 
     112#DataFormat                  suds 
     113#OutDir                      "/home/earthworm/SUDS" 
     114# 
     115# SAC 
     116# 
     117DataFormat                   sac 
     118OutDir                       "c:\earthworm\SAC" 
     119# 
     120# AH 
     121# 
     122#DataFormat                  ah 
     123#OutDir                     "/home/earthworm/AH" 
     124# 
     125# SEISAN 
     126# 
     127#DataFormat                  seisan 
     128#OutDir                     "/home/earthworm/seisan" 
     129# 
     130# GSE 
     131# 
     132#DataFormat                  gse_int 
     133#OutDir                     "/home/earthworm/gse" 
     134# 
     135# Tankplayer 
     136# 
     137#DataFormat                 tank 
     138#OutDir                     "./tanks/" 
     139# 
     140# Mini-SEED 
     141# Mini-SEED output format is currently only available on Solaris 
     142# 
     143#DataFormat                 mseed 
     144#OutDir                     "/earthworm/data/mseed" 
     145# 
     146# PSN4 
     147# 
     148# NOTE PSN4 output format is only avialable for WINDOWS! 
     149#  a station.lst file must also exist in the EW_PARAMS directory too (see example in docs) 
     150# 
     151#DataFormat                 psn4 
     152#OutDir                     "c:\earthworm\psn4" 
     153# 
     154# Specify on what platform the output files will be used: 
     155# intel or sparc - with this information, files will be written out 
     156# in the correct byte order. 
     157# 
     158OutputFormat                sparc 
     159}}} 
     160 
     161=== FUNCTIONAL COMMAND LISTING === 
     162 
     163Below are the commands recognized by trig2disk, grouped by the function they influence. All of the required commands must be specified in the control file in order for trig2disk to operate. 
     164{{{ 
     165   Earthworm system setup: 
     166                Debug   optional 
     167                GetEventsFrom   required 
     168                HeartBeatInt          required 
     169                MyModuleId              required 
     170                RingName          required 
     171 
     172        Waveform Control: 
     173                DelayTime       optional 
     174                GapThresh       required 
     175                MaxTraces       required 
     176                MinDuration     required 
     177                QueueSize       optional 
     178                TimeoutSeconds  required 
     179                TraceBufferLen  required 
     180                TrigStation     optional 
     181                WaveServer      required 
     182 
     183        Output Control: 
     184                DataFormat      required 
     185                LogFile                    required 
     186                OutDir  required 
     187                OutputFormat    required 
     188                QueueFile       optional 
     189                Postproc        optional 
     190}}} 
     191 
     192=== ALPHABETIC COMMAND LISTING & DESCRIPTION === 
     193 
     194In the following section, all configuration file commands are listed in alphabetical order. Listed along with the command (bold-type) are its arguments (in red), the name of the subroutine that processes the command, and the function within the module that the command influences. A detailed description of the command and is also given. Default values and the values used by Memphis are listed after each command description. 
     195 
     196The following list is organized by: 
     197 
     198command [argument here] 
     199 
     200'''!DataFormat [string]'''[[BR]] 
     201Processed by: putaway[[BR]] 
     202Function: Waveform control 
     203 
     204Sets the format of the output waveform files.  Valid strings defining format to use are: 
     205{{{ 
     206ah 
     207 
     208sac 
     209 
     210suds 
     211 
     212tank 
     213 
     214gse_int 
     215 
     216seisan 
     217  
     218mseed - solaris only 
     219 
     220psn4  - windows only 
     221 
     222wfdisc - linux only 
     223}}} 
     224 
     225Only one data format can be used for a given instance of trig2disk. The ah format is xdr. The sac format is not xdr. The suds format is the PC Suds standard. The tank format is suitable for playing through the earthworm tankplayer utility. The 
     226gse_int format is GSE 2.0. The seisan format is suitable for use by Seisan. The putaway routines used were contributed by various authors and it is assumed that the user is familiar with the details of individual formats and they are not  explained here. 
     227{{{ 
     228Default:  none 
     229Example   DataFormat sac 
     230}}} 
     231 
     232'''Debug [flag]''' 
     233 
     234Uncomment to turn on debugging information. 
     235 
     236'''!DelayTime [nsec]'''[[BR]] 
     237Processed by: SnippetMaker[[BR]] 
     238Function: Waveform control 
     239 
     240Sets the time (in seconds) to wait after dequeueing a type_triglist2k message and before attempting to process the message.  Some scnl's may have longer time length data packets or other latencies making it possible to generate a trigger message before all data are available in a waveserver. !DelayTime allows a mechanism to account for this latency.  Large delay times in a busy queue could cause missed messages.  
     241{{{ 
     242Default:  0 
     243Example   DelayTime 5 
     244}}} 
     245 
     246'''!GapThresh [nsamp]'''[[BR]] 
     247Processed by: PA_next[[BR]] 
     248Function: Waveform control 
     249 
     250Sets the gap threshold to G sample intervals. Trace data packets are timestamped with a start time and a stop time, the times of the first and last samples in that packet, respectively. Thus, the expected gap between end time of one packet and the start time of the next packet is one sample interval. Some data sources, such as older digitizers, produce data with slightly larger or smaller intervals between them. The gap threshold is intended to provide a means of detecting missing packets, without falsely declaring a missing packet because of sloppy timestamps. The preferred value for this gap threshold is 1.5. Set larger values only if you have a sloppy data source. It is not clear how wave_serverV should handle intervals of much more than one sample interval between packets if these are not caused by missing packets. 
     251{{{ 
     252Default:  none 
     253Example   GapThresh  1.5 
     254}}} 
     255 
     256'''!GetEventsFrom [logo]'''[[BR]] 
     257Processed by: trig2disk_config[[BR]] 
     258Function: Earthworm setup 
     259 
     260Controls the TYPE_TRIGLIST2K messages input to trig2disk. Trig2disk will only process TYPE_TRIGLIST2K messages that come from module mod_id at installation inst. inst and mod_id are character strings (valid strings are listed in earthworm_global.d and earthworm.d) which are related to single-byte numbers that uniquely identify each installation and module (please do not modify earthworm_global.d). Multiple "!GetEventsFrom" commands may be issued; wildcards (INST_WILDCARD and MOD_WILDCARD) will force trig2disk to process all trigger messages, regardless of their place of origin. 
     261{{{ 
     262Default:  none 
     263Example   GetEventsFrom  INST_WILDCARD  MOD_WILDCARD 
     264          GetEventsFrom  INST_MEMPHIS   MOD_CARLSUBTRIG 
     265}}} 
     266 
     267'''!HeartBeatInt [nsec]'''[[BR]] 
     268Processed by: trig2disk_config[[BR]] 
     269Function: Earthworm setup 
     270 
     271Defines the number of seconds nsec between TYPE_HEARTBEAT messages issued by trig2disk. 
     272{{{ 
     273Default:  none 
     274Example: HeartBeatInt 15 
     275}}} 
     276 
     277'''Logfile [switch]'''[[BR]] 
     278Processed by: trig2disk_config[[BR]] 
     279Function: Earthworm setup 
     280 
     281Sets the on-off switch for writing a log file to disk.  If switch is 0 , no log file will be written.  If switch is 1, trig2disk will log to stderr/stdout and to a daily log file(s) called paramfile_ccyymmdd.log where paramfile is the config file (so-called .d) of a particular instance of trig2disk and ccyymmdd is the current UTC date (ex: 20060123) on the system clock. The file(s) will be written in the EW_LOG directory (environment variable). If switch is 2, only the log file is used and not stderr/stdout. 
     282 
     283{{{ 
     284Default:  none  
     285Example: LogFile 1 
     286}}} 
     287 
     288'''!MaxTraces [int]'''[[BR]] 
     289Processed by: trig2disk_config[[BR]] 
     290Function: Waveform control 
     291 
     292This is the maximum number of traces in a given triglist2k message that we'll ever have to deal with (e.g. individual scnl's).  If the total number of traces from any incoming triglist2k message exceed this value, only !MaxTraces number of traces will be saved.  Excessively large values for this parameter may cause trig2disk to use an excessively large amount of memory as sufficient memory will be allocated to buffer this many traces. 
     293{{{ 
     294Default:  none 
     295Example: MaxTraces 1024 
     296}}} 
     297 
     298'''!MinDuration [int]'''[[BR]] 
     299Processed: trig2disk_config[[BR]] 
     300Function: Waveform control 
     301 
     302The value of the !MinDuration parameter defines the minimum number of seconds to save for each additional scnl trace configured using the !TrigStation parameter.  For added scnl's not in the triglist2k message, start-time will be the earliest value of all scnl's in the triglist message, and duration will be the longest value of all scnl's in the triglist message. If the resulting duration is less than !MinDuration seconds, then !MinDuration seconds will be saved. 
     303{{{ 
     304Default: 
     305Example: MinDuration 30 
     306}}} 
     307 
     308'''!MyModuleId [mod_id]'''[[BR]] 
     309Processed by: trig2disk_config[[BR]] 
     310Function: Earthworm setup 
     311 
     312Sets the module id for labeling all outgoing messages. mod_id is a character string (valid strings are listed in  earthworm.d) that relates (in earthworm.d) to a unique single-byte number. 
     313{{{ 
     314Default:  none 
     315Example:  MyModuleId MOD_TRIG2DISK 
     316}}} 
     317 
     318'''!OutDir [path]'''[[BR]] 
     319Processed by: trig2disk_config[[BR]] 
     320Function: Output control  
     321 
     322Trig2disk will create a separate subdirectory for each event within the directory configured by !OutDir.  Specific naming conventions for subdirectories and files are output format dependent but generally based on trigger time in the triglist2k message. 
     323{{{ 
     324Default:  none 
     325Example:  OutDir /export/home/ew/sac 
     326}}} 
     327 
     328'''!OutputFormat [string]'''[[BR]] 
     329Processed by: trig2disk_config[[BR]] 
     330Function: Output control 
     331 
     332Sets the byte order of binary waveform files that trig2disk will write. Either intel or sparc are acceptable strings.  This parameter should be configured for the platform on which the output datafiles are going to be read. 
     333{{{ 
     334Default: none 
     335Example: OutputFormat sparc 
     336}}} 
     337 
     338'''Postproc [string]'''[[BR]] 
     339Processed by: trig2disk_config[[BR]] 
     340Function: Output control 
     341 
     342Runs the script/program associated with string with no arguments AFTER a trigger is converted to disk files. Must be an absolute path to the script/program to be run. Otherwise, the script must be located in the params (EW_PARAMS) directory. The program or script must be marked as executable.  
     343{{{ 
     344Default: none 
     345Example: Postproc /home/ew/myscript_that_does_something_with_triggers 
     346}}} 
     347 
     348'''!QueueFile [file]'''[[BR]] 
     349Processed by: trig2disk_config [[BR]] 
     350Function: Output Control 
     351 
     352Optional parameter to save waveserver queue status.  This allows trig2disk to save its message queue if a fatal error or stop or restart message is received.  On next start, trig2disk will load the queue from this file and start message processing from there. 
     353{{{ 
     354Default: none 
     355Example: QueueFile trig2disk.que 
     356}}} 
     357 
     358'''!QueueSize [int]'''[[BR]] 
     359Processed by: Trig2disk_config[[BR]] 
     360Function: Output Control 
     361 
     362Controls the number of triglist2k messages to hold in the queue.  No more than this many messages will be dumped to the file configured in !QueueFile parameter and subsequently processed on restart. 
     363{{{ 
     364Default: 10 
     365Example: QueueSize 10 
     366}}} 
     367 
     368'''!RingName [ring]'''[[BR]] 
     369Processed by: trig2disk_config[[BR]] 
     370Function: Earthworm setup 
     371 
     372Tells trig2disk which shared memory region to use for input/output. ring is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique number for the key to the shared memory region. 
     373{{{ 
     374Default:  none 
     375Example: RingName HYPO_RING 
     376}}} 
     377 
     378'''!TimeoutSeconds [int]'''[[BR]] 
     379Processed by: trig2disk_config [[BR]] 
     380Function Waveform control 
     381 
     382If a wave_server doesn't respond to any interaction within !TimeoutSeconds number of seconds, bail and move on to the next waveserver in the list. 
     383{{{ 
     384Default: none 
     385Example: TimeoutSeconds 30 
     386}}} 
     387 
     388'''!TraceBufferLen [int]'''[[BR]] 
     389Processed by: trig2disk_config[[BR]] 
     390Function: Waveform control 
     391 
     392This parameter sets the number of kilobytes trig2disk will ever have to deal with for a given trace.  Make sure it is large enough for expected sample rates and durations (a single floating point sample is generally 4 bytes on most platforms). 
     393{{{ 
     394Default: none 
     395Example: TraceBufferLen    1000 
     396}}} 
     397 
     398'''!TrigStation [scnl]'''[[BR]] 
     399Processed by: trig2disk_config[[BR]] 
     400Function: Output control 
     401 
     402Optional SCNL list of channels to write for each trigger message.  These get get written in addition to those in the trigger message.  Trace start will be set to the earliest of any scnl in the trigger message and duration will be the longest of any scnl in the trigger message (but always at least !MinDuration seconds long).  
     403{{{ 
     404Default: none 
     405Example: TrigStation SWET HHZ ET -- 
     406}}} 
     407 
     408'''!WaveServer [ip port]'''[[BR]] 
     409Processed by: trig2disk_config[[BR]] 
     410Function: Waveform Control 
     411 
     412List of waveserver ip address and port from which to gather traces.  Can have multiple waveservers configured. 
     413{{{ 
     414Default: none 
     415Example: WaveServer   111.222.33.444     16022 
     416}}} 
     417 
     418== Helpful Hints ==