wiki:wftimefilter
Last modified 9 years ago Last modified on 03/25/12 14:04:26

Earthworm Module: wftimefilter

Contributed by: Lynn Dietz for the CISN

Function

Sanitizer of waveform packets to eliminate out of order and badly timed packets (overlaps and dups etc). (new in EW v7.0)

Details

This new module was created to "sanitize" incoming waveform data streams to the liking of Earthworm modules. It reads waveform data (compressed or uncompressed) from one transport ring and writes it to another ring, filtering out duplicate data, time overlaps and bogus future timestamped data. Timestamp checks are done for each channel independently. The check for bogus future timestamps requires that the system clock is set to network time.

Wftimefilter can be configured to output the messages using the original installation/moduleid or its own installation/moduleid. However, it does not alter the contents of the message in any way.

Systems which are importing data using import_ack/export*ack should probably use wftimefilter to remove possible duplicate data after an import/export reconnect. Systems importing waveform data from non-Earthworm data sources should also consider running wftimefilter to ensure that only chronological timeseries enter their own Earthworm processing.

Modifed to allow additional optional behavior:

  1. Rejecting packets that have a sample-rate different than the expected one for the channel.
  2. Limiting per-channel logging, to limit logfile size and increase readability.
  3. Adding per-channel daily summaries of errors and warnings.
  4. Adding the ability to issue a STATUS message to a ring as a part of the daily summary.
  5. Default behavior should be unchanged.

Example Configuration File

# wftimefilter configuration file

# Basic Earthworm setup:
#-----------------------
 MyModuleId    MOD_WFTIMEFILTER # module id for this instance of wftimefilter 
 InRing        RAW_RING         # shared memory ring for input
 OutRing       WAVE_RING        # shared memory ring for output
 HeartbeatInt  30               # seconds between heartbeats to statmgr
 LogFile       1                # 0 log to stderr/stdout only 
                                # 1 log to stderr/stdout and to disk file
                                # 2 log to disk file only

# List the message logos to grab from InRing, WILDCARDs permitted.
# Multiple "GetLogo" commands are allowed, with no hardcoded limit.
# The four valid message types are:
# TYPE_TRACEBUF, TYPE_TRACEBUF2, TYPE_TRACE_COMP_UA, TYPE_TRACE2_COMP_UA
#         Installation   Module        Message Types
#-----------------------------------------------------------------
 GetLogo  INST_WILDCARD  MOD_WILDCARD  TYPE_TRACEBUF2

# Waveform Filtering Parameters
#------------------------------
 MaxMessageSize    4096  # length (bytes) of largest msg to be processed

 UseOriginalInstId    1  # 0  apply wftimefilter's installationid to 
                         #    any messages that pass the test. 
                         # non-zero means apply the original instid.

 UseOriginalModId     0  # 0  apply wftimefilter's moduleid to 
                         #    any messages that pass the test 
                         # non-zero means apply the original instid.

 LogGap             0.0  # Minumum gap duration (seconds) for which we'll make
                         # a log entry.

 TimeJumpTolerance  600  # (in seconds) This term is used to catch packets with
                         # dubious timestamps.  If a time gap is detected in an
                         # incoming data stream, the new packet's timestamp is
                         # compared to the system clock.  If it is later than
                         # the current system time by more than TimeJumpTolerance
                         # seconds, wftimefilter assumes the packet timestamp is 
                         # bogus (it's in the future) and ignores the entire packet.
                         # NOTE: if you use this feature with small tolerances,
                         # the system clock must be kept on network time!!!
                         # 
                         # Set to -1 if you never want to compare packet times
                         # to the system clock.
                         # Valid values are -1 or any number >= 0.0


#AllowChangesInSampleRate 1
                         # Set to 0 to disallow packets with a different sample-rate
                         # than the rate already observed for that channel.

#################################################################################
## LOGGING And STATUS Commands
#################################################################################
#LimitLoggingOfBadChannels 0
                         # Set to 1 to 
                         # suppress log messages from channels that  
                         # exceed the MaxPackets daily log thresholds.
                         # This way, one (or more) bad channel(s) will not generate a
                         # huge logfile.

#MaxPacketsToLogPerChannelPerDay 3
                         # If LimitLoggingOfBadChannels is enabled, then
                         # this is this number is the MAXIMUM number of
                         # filtered packets that will be logged per channel per day.

#MaxPacketsToLogPerWarningPerDay 3
                         # If LimitLoggingOfBadChannels is enabled, then
                         # this is this number is the MAXIMUM number of
                         # packet warnings that will be logged per channel per day.
                         # Currently, warnings issued for gaps between packets, and
                         # possibly changes in sample rate depending on the value
                         # of AllowChangesInSampleRate.

                         
#LogDailySummary 0
                         # Set to 1 to log a daily summary of the packets processed
                         # and number of errors and warning generated per channel.

#IssueDailyStatusWithSummaryOfBadPackets 0
                         # Set to 1 to have program issue a STATUS ERROR message each day
                         # if bad packets are processed that day.

#LogOnlyChannelsWithProblems 0
                         # If LogDailySummary is set to 1, then set this param 
                         # to 1 to cause only channels with problems to be logged
                         # in the daily summary.

Helpful Hints