Changes between Initial Version and Version 1 of wftimefilter


Ignore:
Timestamp:
03/25/12 14:04:26 (10 years ago)
Author:
branden
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • wftimefilter

    v1 v1  
     1[[PageOutline]] 
     2 
     3= [wiki:Earthworm Earthworm] Module: wftimefilter = 
     4'''Contributed by: Lynn Dietz for the CISN''' 
     5 
     6== Function == 
     7Sanitizer of waveform packets to eliminate out of order and badly timed packets (overlaps and dups etc). (new in EW v7.0) 
     8 
     9== Details == 
     10This new module was created to "sanitize" incoming waveform data  streams to the liking of Earthworm modules. It reads waveform data (compressed or uncompressed) from one transport ring and writes it to another ring, filtering out duplicate data, time overlaps and bogus future timestamped data. Timestamp checks are done for each channel independently. The check for bogus future timestamps requires that the system clock is set to network time. 
     11 
     12Wftimefilter can be configured to output the messages using the original installation/moduleid or its own installation/moduleid. However, it does not alter the contents of the message in any way. 
     13 
     14Systems which are importing data using import_ack/export*ack should probably use wftimefilter to remove possible duplicate data after  an import/export reconnect. Systems importing waveform data from  non-Earthworm data sources should also consider running wftimefilter to ensure that only chronological timeseries enter their own Earthworm processing.  
     15 
     16Modifed to allow additional optional behavior: 
     17 1. Rejecting packets that have a sample-rate different than the expected one for the channel.   
     18 2. Limiting per-channel logging, to limit logfile size and increase readability. 
     19 3. Adding per-channel daily summaries of errors and warnings. 
     20 4. Adding the ability to issue a STATUS message to a ring as a part of the daily summary. 
     21 5. Default behavior should be unchanged. 
     22 
     23=== Example Configuration File === 
     24{{{ 
     25# wftimefilter configuration file 
     26 
     27# Basic Earthworm setup: 
     28#----------------------- 
     29 MyModuleId    MOD_WFTIMEFILTER # module id for this instance of wftimefilter  
     30 InRing        RAW_RING         # shared memory ring for input 
     31 OutRing       WAVE_RING        # shared memory ring for output 
     32 HeartbeatInt  30               # seconds between heartbeats to statmgr 
     33 LogFile       1                # 0 log to stderr/stdout only  
     34                                # 1 log to stderr/stdout and to disk file 
     35                                # 2 log to disk file only 
     36 
     37# List the message logos to grab from InRing, WILDCARDs permitted. 
     38# Multiple "GetLogo" commands are allowed, with no hardcoded limit. 
     39# The four valid message types are: 
     40# TYPE_TRACEBUF, TYPE_TRACEBUF2, TYPE_TRACE_COMP_UA, TYPE_TRACE2_COMP_UA 
     41#         Installation   Module        Message Types 
     42#----------------------------------------------------------------- 
     43 GetLogo  INST_WILDCARD  MOD_WILDCARD  TYPE_TRACEBUF2 
     44 
     45# Waveform Filtering Parameters 
     46#------------------------------ 
     47 MaxMessageSize    4096  # length (bytes) of largest msg to be processed 
     48 
     49 UseOriginalInstId    1  # 0  apply wftimefilter's installationid to  
     50                         #    any messages that pass the test.  
     51                         # non-zero means apply the original instid. 
     52 
     53 UseOriginalModId     0  # 0  apply wftimefilter's moduleid to  
     54                         #    any messages that pass the test  
     55                         # non-zero means apply the original instid. 
     56 
     57 LogGap             0.0  # Minumum gap duration (seconds) for which we'll make 
     58                         # a log entry. 
     59 
     60 TimeJumpTolerance  600  # (in seconds) This term is used to catch packets with 
     61                         # dubious timestamps.  If a time gap is detected in an 
     62                         # incoming data stream, the new packet's timestamp is 
     63                         # compared to the system clock.  If it is later than 
     64                         # the current system time by more than TimeJumpTolerance 
     65                         # seconds, wftimefilter assumes the packet timestamp is  
     66                         # bogus (it's in the future) and ignores the entire packet. 
     67                         # NOTE: if you use this feature with small tolerances, 
     68                         # the system clock must be kept on network time!!! 
     69                         #  
     70                         # Set to -1 if you never want to compare packet times 
     71                         # to the system clock. 
     72                         # Valid values are -1 or any number >= 0.0 
     73 
     74 
     75#AllowChangesInSampleRate 1 
     76                         # Set to 0 to disallow packets with a different sample-rate 
     77                         # than the rate already observed for that channel. 
     78 
     79################################################################################# 
     80## LOGGING And STATUS Commands 
     81################################################################################# 
     82#LimitLoggingOfBadChannels 0 
     83                         # Set to 1 to  
     84                         # suppress log messages from channels that   
     85                         # exceed the MaxPackets daily log thresholds. 
     86                         # This way, one (or more) bad channel(s) will not generate a 
     87                         # huge logfile. 
     88 
     89#MaxPacketsToLogPerChannelPerDay 3 
     90                         # If LimitLoggingOfBadChannels is enabled, then 
     91                         # this is this number is the MAXIMUM number of 
     92                         # filtered packets that will be logged per channel per day. 
     93 
     94#MaxPacketsToLogPerWarningPerDay 3 
     95                         # If LimitLoggingOfBadChannels is enabled, then 
     96                         # this is this number is the MAXIMUM number of 
     97                         # packet warnings that will be logged per channel per day. 
     98                         # Currently, warnings issued for gaps between packets, and 
     99                         # possibly changes in sample rate depending on the value 
     100                         # of AllowChangesInSampleRate. 
     101 
     102                          
     103#LogDailySummary 0 
     104                         # Set to 1 to log a daily summary of the packets processed 
     105                         # and number of errors and warning generated per channel. 
     106 
     107#IssueDailyStatusWithSummaryOfBadPackets 0 
     108                         # Set to 1 to have program issue a STATUS ERROR message each day 
     109                         # if bad packets are processed that day. 
     110 
     111#LogOnlyChannelsWithProblems 0 
     112                         # If LogDailySummary is set to 1, then set this param  
     113                         # to 1 to cause only channels with problems to be logged 
     114                         # in the daily summary. 
     115}}} 
     116 
     117== Helpful Hints ==