wiki:trig2disk
Last modified 9 years ago Last modified on 05/02/12 09:07:15

Earthworm Module: trig2disk

Contributed by:

Function

Trig2disk module reads trigger messages and retrieves waveform data from wave_serverV into AH or SAC format files.

Details

trig2disk allows an Earthworm to save waveform data to several formats: sac, ah, PCSUDS, Earthworm tankplayer, GSE, and Seisan. Trig2disk reads type_triglist2k messages from its message ring (e.g. those produced by arc2trig and by carltrig) and writes triggered data files to disk. It uses the same putaway routines as waveman2disk. Minimal location code support is provided but is naturally output data format dependent.

PSN4 format was added for windows in Earthworm 7.1 (contact www.webtronics.com for details).

MSEED format was added for solaris in Earthworm 7.1.

WFDISC format was added for Linux in Earthworm 7.4.
wfdisc format can take advantage of a wfdisc params config file which must be located at "EW_PARAMS/wfdisc_params.d" where EW_PARAMS is the EW_PARAMS env variable. The content of this file is very specific and allows one to specify station/channel specific information on chanid, calib, calper, instype and segtype. See the command document for file syntax and formatting

The ah data is in xdr format which is transparent between sun/solaris and intel/NT so long as the ioroutin.c interface is used.

The NT port of the xdr routines was accomplished by modifying the OpenBSD code from the ftp site at umich. Thus modifications must include this source should be careful not to remove the copyright statements and can't be sold. Cavalier commenting of pieces of this source that weren't needed to write local xdr is ubiquitous. Thus it is not rpc compatible.

Will get restarted if restartMe is in the .desc

This module is cloned from ora_trace_save. Thanks Alex. The putaway routines are cloned from those in earth2sac. Thanks Pete. Many others contributed the various format putaway routines.

Configuration File Commands

On startup, trig2disk reads the configuration file named on the command line. Commands in this file set all the parameters used for reading triglist2k messages and writing sac, ah, suds, gse, seisan, or tankplayer format. In the control file, lines may begin with a valid trig2disk command (listed below) or with one of 2 special characters:

#  marks the line as a comment (example: # This is a comment).
   
@  allows control files to be nested; one control file can be 
   accessed from another with the command "@" followed by 
   a string representing the path name of the next control file 
   (example: @memphis.scnl).

Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!).

EXAMPLE CONFIGURATION FILE

#
# Configuration file for trig2disk:
# I listen to trigger messages (TYPE_TRIGLIST), and stuff the implied pieces
# of trace data into an Oracle server. I do this by going to any number
# of WaveServerV's, as listed in this configuration file.
#
MyModuleId                 MOD_TRIG2DISK
RingName                   HYPO_RING
HeartBeatInt               30
LogFile                    1   # 0 means don't create a disc log file. 1=> do.
                               # 2 means log to disk file but not stderr/stdout
#
# What message logos to listen to. Can be more than one.
# The type is hard coded to TYPE_TRIGLIST
#
GetEventsFrom	           INST_MENLO  MOD_WILDCARD

#
# list of ip addresses and ports of the WaveServers we're to use
#
WaveServer                 111.222.33.444     16022

 

# Optional NEW COMMAND in EW v7.3 Postproc allows running of a program or script that will process new files
# no arguments are sent to the script...It should know where to run and what to do with new files:
#
#Postproc  myscript  # the myscript or program should be in the params directory or an absolute path must be provided.


#
# If a WaveServer doesn't talk to us in this 
# many seconds, we'll abort that request
#
TimeoutSeconds             30  

#
# Max number of traces we'll ever see in one event
#
MaxTraces                  500	

#
# kilobytes of the largest trace snippet we'll ever have to deal with
#
TraceBufferLen             1000  	

#
# Debug switch:  Debug will be logged if uncommented 
#
Debug

#
# SCNL list of stations to write for each trigger message,  these get
# written in addition to SCNLs in the trigger message.
#
#@memphis.scnl

#
# Minimum number of seconds to save for extra channels that we
# are saving (i.e., those channels not in the TYPE_TRIGLIST message)
#
MinDuration                 30

#
# number of sample periods after which we declare a gap
#
GapThresh 100

# Optional queue commands
# Number of trigger messages to hold in queue; default is 10
#QueueSize 10

# Optional queue dumpfile name, for saving state of queue
QueueFile trig2disk.que

# Optional delay time: trig2disk will wait this many seconds from the
# time it receives a trigger message until it starts to process the message
DelayTime 5


#
# Select format of output data and the directory where it is written
# Only one of the following pairs should be uncommented.
#
#
# SUDS
#
#DataFormat                  suds
#OutDir                      "/home/earthworm/SUDS"
#
# SAC
#
DataFormat                   sac
OutDir                       "c:\earthworm\SAC"
#
# AH
#
#DataFormat                  ah
#OutDir                     "/home/earthworm/AH"
#
# SEISAN
#
#DataFormat                  seisan
#OutDir                     "/home/earthworm/seisan"
#
# GSE
#
#DataFormat                  gse_int
#OutDir                     "/home/earthworm/gse"
#
# Tankplayer
#
#DataFormat                 tank
#OutDir                     "./tanks/"
#
# Mini-SEED
# Mini-SEED output format is currently only available on Solaris
#
#DataFormat                 mseed
#OutDir                     "/earthworm/data/mseed"
#
# PSN4
#
# NOTE PSN4 output format is only avialable for WINDOWS!
#  a station.lst file must also exist in the EW_PARAMS directory too (see example in docs)
#
#DataFormat                 psn4
#OutDir                     "c:\earthworm\psn4"
#
# Specify on what platform the output files will be used:
# intel or sparc - with this information, files will be written out
# in the correct byte order.
#
OutputFormat                sparc

station.lst

FUNCTIONAL COMMAND LISTING

Below are the commands recognized by trig2disk, grouped by the function they influence. All of the required commands must be specified in the control file in order for trig2disk to operate.

   Earthworm system setup:
                Debug	optional
                GetEventsFrom	required
                HeartBeatInt          required
                MyModuleId              required
                RingName          required

        Waveform Control:
                DelayTime	optional
                GapThresh	required
                MaxTraces	required
                MinDuration	required
                QueueSize	optional
                TimeoutSeconds	required
                TraceBufferLen	required
                TrigStation	optional
                WaveServer	required

        Output Control:
                DataFormat	required
                LogFile                    required
                OutDir	required
                OutputFormat	required
                QueueFile	optional
		Postproc        optional

ALPHABETIC COMMAND LISTING & DESCRIPTION

In the following section, all configuration file commands are listed in alphabetical order. Listed along with the command (bold-type) are its arguments (in red), the name of the subroutine that processes the command, and the function within the module that the command influences. A detailed description of the command and is also given. Default values and the values used by Memphis are listed after each command description.

The following list is organized by:

command [argument here]

DataFormat [string]
Processed by: putaway
Function: Waveform control

Sets the format of the output waveform files. Valid strings defining format to use are:

ah

sac

suds

tank

gse_int

seisan
 
mseed - solaris only

psn4  - windows only

wfdisc - linux only

Only one data format can be used for a given instance of trig2disk. The ah format is xdr. The sac format is not xdr. The suds format is the PC Suds standard. The tank format is suitable for playing through the earthworm tankplayer utility. The gse_int format is GSE 2.0. The seisan format is suitable for use by Seisan. The putaway routines used were contributed by various authors and it is assumed that the user is familiar with the details of individual formats and they are not explained here.

Default:  none
Example   DataFormat sac

Debug [flag]

Uncomment to turn on debugging information.

DelayTime [nsec]
Processed by: SnippetMaker
Function: Waveform control

Sets the time (in seconds) to wait after dequeueing a type_triglist2k message and before attempting to process the message. Some scnl's may have longer time length data packets or other latencies making it possible to generate a trigger message before all data are available in a waveserver. DelayTime allows a mechanism to account for this latency. Large delay times in a busy queue could cause missed messages.

Default:  0
Example   DelayTime 5

GapThresh [nsamp]
Processed by: PA_next
Function: Waveform control

Sets the gap threshold to G sample intervals. Trace data packets are timestamped with a start time and a stop time, the times of the first and last samples in that packet, respectively. Thus, the expected gap between end time of one packet and the start time of the next packet is one sample interval. Some data sources, such as older digitizers, produce data with slightly larger or smaller intervals between them. The gap threshold is intended to provide a means of detecting missing packets, without falsely declaring a missing packet because of sloppy timestamps. The preferred value for this gap threshold is 1.5. Set larger values only if you have a sloppy data source. It is not clear how wave_serverV should handle intervals of much more than one sample interval between packets if these are not caused by missing packets.

Default:  none
Example   GapThresh  1.5

GetEventsFrom [logo]
Processed by: trig2disk_config
Function: Earthworm setup

Controls the TYPE_TRIGLIST2K messages input to trig2disk. Trig2disk will only process TYPE_TRIGLIST2K messages that come from module mod_id at installation inst. inst and mod_id are character strings (valid strings are listed in earthworm_global.d and earthworm.d) which are related to single-byte numbers that uniquely identify each installation and module (please do not modify earthworm_global.d). Multiple "GetEventsFrom" commands may be issued; wildcards (INST_WILDCARD and MOD_WILDCARD) will force trig2disk to process all trigger messages, regardless of their place of origin.

Default:  none
Example   GetEventsFrom  INST_WILDCARD  MOD_WILDCARD
          GetEventsFrom  INST_MEMPHIS   MOD_CARLSUBTRIG

HeartBeatInt [nsec]
Processed by: trig2disk_config
Function: Earthworm setup

Defines the number of seconds nsec between TYPE_HEARTBEAT messages issued by trig2disk.

Default:  none
Example: HeartBeatInt 15

Logfile [switch]
Processed by: trig2disk_config
Function: Earthworm setup

Sets the on-off switch for writing a log file to disk. If switch is 0 , no log file will be written. If switch is 1, trig2disk will log to stderr/stdout and to a daily log file(s) called paramfile_ccyymmdd.log where paramfile is the config file (so-called .d) of a particular instance of trig2disk and ccyymmdd is the current UTC date (ex: 20060123) on the system clock. The file(s) will be written in the EW_LOG directory (environment variable). If switch is 2, only the log file is used and not stderr/stdout.

Default:  none 
Example: LogFile 1

MaxTraces [int]
Processed by: trig2disk_config
Function: Waveform control

This is the maximum number of traces in a given triglist2k message that we'll ever have to deal with (e.g. individual scnl's). If the total number of traces from any incoming triglist2k message exceed this value, only MaxTraces number of traces will be saved. Excessively large values for this parameter may cause trig2disk to use an excessively large amount of memory as sufficient memory will be allocated to buffer this many traces.

Default:  none
Example: MaxTraces 1024

MinDuration [int]
Processed: trig2disk_config
Function: Waveform control

The value of the MinDuration parameter defines the minimum number of seconds to save for each additional scnl trace configured using the TrigStation parameter. For added scnl's not in the triglist2k message, start-time will be the earliest value of all scnl's in the triglist message, and duration will be the longest value of all scnl's in the triglist message. If the resulting duration is less than MinDuration seconds, then MinDuration seconds will be saved.

Default:
Example: MinDuration 30

MyModuleId [mod_id]
Processed by: trig2disk_config
Function: Earthworm setup

Sets the module id for labeling all outgoing messages. mod_id is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique single-byte number.

Default:  none
Example:  MyModuleId MOD_TRIG2DISK

OutDir [path]
Processed by: trig2disk_config
Function: Output control

Trig2disk will create a separate subdirectory for each event within the directory configured by OutDir. Specific naming conventions for subdirectories and files are output format dependent but generally based on trigger time in the triglist2k message.

Default:  none
Example:  OutDir /export/home/ew/sac

OutputFormat [string]
Processed by: trig2disk_config
Function: Output control

Sets the byte order of binary waveform files that trig2disk will write. Either intel or sparc are acceptable strings. This parameter should be configured for the platform on which the output datafiles are going to be read.

Default: none
Example: OutputFormat sparc

Postproc [string]
Processed by: trig2disk_config
Function: Output control

Runs the script/program associated with string with no arguments AFTER a trigger is converted to disk files. Must be an absolute path to the script/program to be run. Otherwise, the script must be located in the params (EW_PARAMS) directory. The program or script must be marked as executable.

Default: none
Example: Postproc /home/ew/myscript_that_does_something_with_triggers

QueueFile [file]
Processed by: trig2disk_config
Function: Output Control

Optional parameter to save waveserver queue status. This allows trig2disk to save its message queue if a fatal error or stop or restart message is received. On next start, trig2disk will load the queue from this file and start message processing from there.

Default: none
Example: QueueFile trig2disk.que

QueueSize [int]
Processed by: Trig2disk_config
Function: Output Control

Controls the number of triglist2k messages to hold in the queue. No more than this many messages will be dumped to the file configured in QueueFile parameter and subsequently processed on restart.

Default: 10
Example: QueueSize 10

RingName [ring]
Processed by: trig2disk_config
Function: Earthworm setup

Tells trig2disk which shared memory region to use for input/output. ring is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique number for the key to the shared memory region.

Default:  none
Example: RingName HYPO_RING

TimeoutSeconds [int]
Processed by: trig2disk_config
Function Waveform control

If a wave_server doesn't respond to any interaction within TimeoutSeconds number of seconds, bail and move on to the next waveserver in the list.

Default: none
Example: TimeoutSeconds 30

TraceBufferLen [int]
Processed by: trig2disk_config
Function: Waveform control

This parameter sets the number of kilobytes trig2disk will ever have to deal with for a given trace. Make sure it is large enough for expected sample rates and durations (a single floating point sample is generally 4 bytes on most platforms).

Default: none
Example: TraceBufferLen    1000

TrigStation [scnl]
Processed by: trig2disk_config
Function: Output control

Optional SCNL list of channels to write for each trigger message. These get get written in addition to those in the trigger message. Trace start will be set to the earliest of any scnl in the trigger message and duration will be the longest of any scnl in the trigger message (but always at least MinDuration seconds long).

Default: none
Example: TrigStation SWET HHZ ET --

WaveServer [ip port]
Processed by: trig2disk_config
Function: Waveform Control

List of waveserver ip address and port from which to gather traces. Can have multiple waveservers configured.

Default: none
Example: WaveServer   111.222.33.444     16022

Helpful Hints