Version 3 (modified by branden, 10 years ago) (diff)


Earthworm Module: q2ew

Contributed by:


A Quanterra/COMSERV data feeding program.


q2ew is a Quanterra/COMSERV data feeding program for Earthworm. The utility was written for the Alaska Tsunami Warning Center's Quanterra Q730 digitizers, but should work with any Quanterra digitizer that is COMSERV compatible. COMSERV is a software freeware product developed by Quanterra and other Quanterra users. It is a middleware software layer between the digitizer and the data acquisition software (clients). In short, COMSERV insulates the developer/user from changes to the QSL communications protocol and acts as a multithreaded switch for client processes to access data from the digitizers.

COMSERV must be configured on the same host as the Earthworm system in order for this program to work. Berkeley is the ftp repository for the COMSERV software. Please refer to the COMSERV documentation on how to correctly install and configure this system.

q2ew is designed as a multi-threaded program. The main thread loops polling the COMSERV shared memory region, using COMSERV cs_scan() calls, for new packets arriving from digitizers. Once data arrives, the main thread converts the data into an earthworm trace buffer and ships the data off to the earthworm transport ring. The second thread, the heartbeat, checks that data are coming in from COMSERV in the main thread and issues regular heartbeats. Should there be a fatal problem with the COMSERV server for any of the stations, the heartbeat thread detects this and a message is sent to statmgr and the process dies. The cs_status() call is a critical one to adjust to fine tune the program to modify the severity of reactions to problems/changes observed with COMSERV. All of the fatal errors will cause q2ew to die, but some of the recoverable errors are now set to cause q2ew to die (in particular CSCR_DIED). As we gain more experience with the robustness of COMSERV, we can adjust the recoverable errors to simply provide status messages, but for now lets play it cautious.

The program q2ew has been tested on Solaris and Linux platforms.

Configuration File Commands

On startup, q2ew reads the configuration file named on the command line. As always, the configuraton file contains comments:

#  marks the line as a comment (example: # This is a comment).

Command names must be typed in the control file exactly as shown in this document (upper/lower case matters!). q2ew supports one diagnostic command line option, -v is for verbose mode so that you can see packets coming in.

Example Configuration File

# q2ew configuration file
# This code receives MiniSEED records from COMSERV, converts them into
# Earthworm trace buf messages, and stuffs them into a wave ring.
 ModuleId	MOD_Q2EW	# module id for this import,
 RingName	WAVE_RING	# transport ring to use for input/output,

 HeartbeatInt	10		# Heartbeat interval in seconds
				# this should be half the q2ew.desc heartbeat!

 LogFile	1		# If 0, don't write logfile at all,

 LOG2LogFile 	1		#  write LOG chans to log file
				# this is useful for debugging timing or
				# data logger problems.	

TimeoutNoSend	5		# If no data for X seconds from COMSERV for
				# ALL stations,  then kill process (HB and
				# main thread). 
				# NOTE: this timeout does not take effect until
				# a trace data packet comes from comserv. That
				# is, LOG channels do not count! 
				# Iff TimeoutNoSend == 0, then this data
				# check is DISABLED and q2ew will only die
				# if EW or COMSERV dies.

# For ATWC, set this next config param to CSQ_LAST
 ComservSeqBuf	CSQ_LAST	# This is a comserv sequence control value
				# that can have the following settings:
				# CSQ_NEXT  = Get newer than already received
				# CSQ_LAST  = Get first available data (latest)
				# CSQ_FIRST = Get all data in DAQ
				# This effects what data are returned.

# These are optional for those EW users who want to have a pin number
# instead of a SCNL. The keyword here is SCNL2pinmap. No duplicate checking
# occurs. Note that SCNL2pinmap replaces SCN2pinmap (q2ew now supports location code)

# Example
#		S	C    	N	L	pin
SCNL2pinmap 	Q003	HHZ  	AT	--	1
SCNL2pinmap 	Q003	HHN  	AT	00	2
SCNL2pinmap 	Q003	HHE  	AT	01	3

Functional Command Listing

Below are the configure commands recognized by q2ew, grouped by the function they influence. Most of the commands are required. Only SCNL2pinmap and LOG2LogFile are optional.

         Earthworm system setup:
 		MyModuleId	   	required
		RingName	   	required
		ComservSeqBuf	   	required
		HeartbeatInt	   	required
		TimeoutNoSend	   	required
	Output Control:
		LogFile		   	required
		LOG2LogFile		optional
		SCNL2pinmap		optional

Alphabetic Command Listing & Description

In the following section, all configuration file commands are listed in alphabetical order. Listed along with the command (bold-type) are its arguments (in red), the name of the subroutine that processes the command, and the function within the module that the command influences. A detailed description of the command and is also given. Default values and example commands are listed after each command description.

The following list is organized by:

command [argument here]

ComservSeqBuf [csqtype]
Processed by: main
Function: Main q2ew Thread

This command is used to define how q2ew gets data from COMSERV. csqtype is the name of a #define as found in COMSERV's comserv/include/service.h file. csqtype can have the following values:


CSQ_LAST is what is being used for ATWC since they are only interested in the latest available data from the digitizer or DA (digitizer=DA in Quanterras documents). CSQ_NEXT gets only the data that are newer than that already receieved by COMSERV and CSQ_FIRST gets all the data in the DA's buffer.

Default:  none
	ComservSeqBuf CSQ_LAST

HeartbeatInt [beat]
Processed by: HeartBeat
Function: Earthworm setup

Beat the q2ew heart every beat seconds. This is implemented as a separate thread in q2ew.

LOG2LogFile [switch]
Processed by: logLOGchans
Function: Main q2ew Thread

Sets the on-off switch for writing Quanterra LOG channels to the LogFile. If switch is 0, no LOG channels will be written to the log. If switch is 1, q2ew will write all LOG channels to the LogFile. Turning this on will automatically turn on writing of a LogFile.

Default:  0 - do not use logit() for LOG channels

LogFile [switch]
Processed by: GetConfig
Function: Earthworm Setup

Sets the on-off switch for writing a log file to disk. If switch is 0, no log file will be written. If switch is 1, q2ew will write a daily log file(s) called q2ewxx.log_yymmdd where xx is q2ew's module id (set with "MyModuleId" command) and yymmdd is the current UTC date (ex: 960123) on the system clock. The file(s) will be written in the EW_LOG directory (environment variable).

Default:  none

MyModuleId [mod_id]
Processed by: GetConfig
Function: Earthworm setup

Sets the module id for labeling all outgoing messages. mod_id is a character string (valid strings are listed in earthworm.d) that relates (in earthworm.d) to a unique single-byte number.

Default:  none				
Calnet:   MyModuleId MOD_Q2EW

RingName [ring]
Processed by: GetConfig
Function: Earthworm setup

This is the ring into which the waveforms and messages are sent.

Default:  none, required option				

SCNL2pinmap [S C N L pin]
Processed by: insertSCNL
Function: Main q2ew Thread

This command is used to define how q2ew maps a Station Channel Name to a pin number for those installations who wish to define pins....This is a throwback to old digitizers and is not recommended since Quanterra digitizers can properly be configured to know SCNL information. Data from S C N L is compared to the SEED header of the waveform packets coming off the DA and the pin number is substituted into the tracebuf struct passed to Earthworm. S is for station name, C is for channel identifier, and N is for network code. Note that station, channel, and network names in Quanterra follow the SEED convention and can be 5, 3, and 2 chars in length respectively.

Default:  none, optional command
SCNL2pinmap	Q003 HHZ AT --  1
SCNL2pinmap	Q003 HLN AT --  2
SCNL2pinmap	Q003 HLE AT --  3

TimeoutNoSend [timeout]
Processed by: HeartBeat
Function: Earthworm setup

If no waveform packets come from the DA after timeout seconds, the q2ew module will notify statmgr and die. If timeout is set to 0, then the q2ew module will never check for this timeout. If set to 0, q2ew will only die if COMSERV dies, or if the Earthworm system calls for its termination. This timeout begins once the first Waveform packet is received from ANY DA served by COMSERV. It is not specific to any one DA.

Default:  none, required option				
ATWC:   TimeoutNoSend 0

Descriptor File Example

Here is a copy of the q2ew.desc file as implemented. Note that all causes of death for q2ew feed through the function q2ew_die() which logs all causes of death using logit() and sends a message to the statmgr utility.

modName  q2ew
modId    MOD_Q2EW
instId   INST_ATWC

restartMe	# restart this sucker as it is critical to data collection
#    Heartbeat Specification.  If the status manager does not receive
#    a heartbeat message every  seconds from this module, an
#    error will be reported (client module dead).   is the maximum
#    number of pager messages that will be reported and  is the
#    maximum number of email messages that will be reported.  If the
#    page or mail limit is exceeded, no further errors will be reported
#    until the status manager is restarted.
tsec: 20  page: 0  mail: 99

# these are the statmgr messages that q2ew will send before dieing
err: 0  nerr: 1  tsec: 0  page: 5  mail: 20
text: "COMSERV has a problem and died"
err: 1  nerr: 1  tsec: 0  page: 5  mail: 20
text: "COMSERV Recv NO DATA TIMEOUT and died"
err: 2  nerr: 1  tsec: 0  page: 5  mail: 20
text: "SIGNAL caused q2ew to die"
err: 3  nerr: 1  tsec: 0  page: 5  mail: 20
text: "EW tport_putmsg() failed, q2ew dead"
err: 4  nerr: 1  tsec: 0  page: 5  mail: 20
text: "EW TERMINATE recv'd, q2ew dead"
err: 5  nerr: 1  tsec: 0  page: 5  mail: 20
text: "EW config problems on startup, q2ew dead"
err: 6  nerr: 1  tsec: 0  page: 5  mail: 20
text: "qlib2 decompression errors"

Helpful Hints