MADIS RT RUN-TIME STATUS for Wed Mar 29 15:32:01 GMT 2023 on vm-lnx-madis-process1a
     INFO RT VM STATUS for Node  vm-lnx-madis-process1a Time Wed Mar 29 15:32:01 GMT 2023

     PASS METAR processing tbl                     file:  128320 Mar 28 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  102  daily<1800min  102  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 13 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 768
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20230329_1400:452 20230329_1300:865  #PUBLIC-DP 20230329_1400:1888 20230329_1300:2011  #PUBLIC-DP-profile 20230329_1400:40 20230329_1300:38      
     PASS Load resource                 m15 < 9 cores 100% (goal > 60%)   -- % of the time M15 < 9  Samples: 932    #10=+ 0 #9 0 #8 54 #7 256
     PASS Open file resource                       check: open file < 5000 Max Files: 2846 #<1000: 355 #<10000 932
     PASS Process count                            check: processes < 1000 Max Processes: 499
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 186 #Delayed: 0
     INFO AO runtime avg sfcanl                     14  seconds   910/62
     INFO AO runtime avg sfcncdf                     1  seconds   49/44
     INFO AO runtime avg sfcver                     79  seconds   4840/61
     INFO AO runtime avg sfcqcmsg                   16  seconds   1037/62
     INFO AO runtime avg sfcqcout meso              32  seconds   5525/172
     INFO AO runtime avg sfcqcout_late meso         80  seconds   18484/229
     INFO AO runtime avg FSL_qcout5m_               41  seconds   7771/187
     INFO AO runtime avg FSL_analrun15m_           114  seconds   7080/62
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 935 #Delayed: 1 Competed timely:  869
     INFO IQC runtime avg qcstg1_2                   5  seconds   76314/14724
     INFO IQC runtime avg qcstg1_2 meso             18  seconds   35207/1863
     INFO IQC runtime avg qcstg1_2_late meso        20  seconds   26063/1263
     INFO IQC runtime avg sfcing                    28  seconds   26591/930
     INFO IQC runtime avg sfchqc                    60  seconds   56354/929
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(9999)
     PASS RT Throughput                            -32434 secs -- check: Avg time < 180 sec. #Hrs: 15 Max secs: 1258176
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20230329_1400: #CPYRMA: 0 #NJWXNET: 635
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.78 % OBS FAILED  CHECK ( 8161 OF 31661) AVGTHR 1.7987 RATIO 0.100E+06 AVGEPS 0.2611E-01 FRMSE 37.372 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              5.22 % OBS FAILED  CHECK ( 3296 OF 63179) AVGTHR 5.2625 RATIO 0.294 AVGEPS 0.1433E-01 FRMSE 18.450 TRMSE 1.5477
     PASS QC POTENTIAL TEMPERATURE (K)             5.01 % OBS FAILED  CHECK ( 3317 OF 66187) AVGTHR 3.6410 RATIO 0.100E+06 AVGEPS 0.2571E-01 FRMSE 1389.1 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             5.20 % OBS FAILED  CHECK ( 3287 OF 63179) AVGTHR 3.9324 RATIO 0.278 AVGEPS 0.1542E-01 FRMSE 22.322 TRMSE 1.0930
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.22 % OBS FAILED  CHECK (  753 OF 61849) AVGTHR 4.4170 RATIO 0.299 AVGEPS 0.1420E-01 FRMSE 8.1754 TRMSE 1.3226
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.34 % OBS FAILED  CHECK (  827 OF 61849) AVGTHR 4.5376 RATIO 0.307 AVGEPS 0.1425E-01 FRMSE 8.0692 TRMSE 1.3943
     PASS QC 3H ALTIMETER CHANGE (MB)              1.17 % OBS FAILED  CHECK (  332 OF 28305) AVGTHR 1.2114 RATIO 0.237 AVGEPS 0.1665E-01 FRMSE 64.731 TRMSE 0.28769
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.15 % OBS FAILED  CHECK (   43 OF   699) AVGTHR 1.5815 RATIO 0.414 AVGEPS 0.6214E-01 FRMSE 4.1702 TRMSE 0.65518
     PASS QC ALTIMETER (MB)                       24.47 % OBS FAILED  CHECK ( 7746 OF 31661) AVGTHR 1.9378 RATIO 0.100E+06 AVGEPS 0.2632E-01 FRMSE 38.293 TRMSE 99999.
     PASS MXST_P limit                             255472    MXST_P: 280000
     PASS AO Possible Seg fault                    0  #logs checked: 2265  
     PASS IQC Possible Seg fault                   0  #logs checked: 4629  
     PASS NONSFC Possible Seg fault                0  #logs checked: 791  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230329_411760.log
     INFO MADIS DB Connection cnts                 Info:486 [2023-Mar-29 15:31:33.208384] - LastUploadStats:mode:upload_sfc,pid:24523,aborted:0,fileCount:1,wholeFileErrors:0,lines:50048,inserts:35165,updates:14883,duplicates:0,errors:2,stationsAdded:22,stationsUpdated:18,publishLines:0,publishErrors:0,notes:./202303291530_receipt.sfcd2.txt,run ms:84193
     PASS AFWS files current                       # in last hour: 11
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_2:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_2:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcanl.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcanl.out.230881300_1:
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871100_1:

  COMPARE Mesonet data providers cnt                39     for 20230329_1300:
  COMPARE DECODE STN TBLS                           149        Newest:     2231 Mar 29 15:00 LDAD/data/DEDOTStation.txt  Oldest:     8848 Aug 21  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:     2231 Mar 29 15:00 LDAD/data/master/DEDOTStation.txt  Oldest:     1568 Aug 21  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (14): Newest: 7 Mar 29 15:27 madis/proc/sfc/ao/qc122/qcsumry15.dat Oldest:
  COMPARE sda IO Writes 2308813                     Local Disk MB/s: 23 RS MB/s: 0 PQ MB/s: 19+ Tot secs: 3475+
  COMPARE sda IO Reads(2308813                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3475+
  COMPARE NetCDF REC Writes 2308813                 Avg DECODE rec/s: 76 Avg AO rec/s: 79 Avg NON-SFC rec/s: 12 Tot secs: 3475
     INFO Network  2308813                          eth0 14 MB/s TX    3 MB/s RX     Tot secs: 3475  
  COMPARE usr1 Disk 2308813                         23 WRITE MB/s | 1 READ MB/s | 5815 Avg WRT TPS | 16559 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20230329_13  sfc/ao/qcms_data/buoy/20230329_1300                                844    1105028
AO-20230329_13  sfc/ao/qcms_data/crn/20230329_1300                                 243     431716
AO-20230329_13  sfc/ao/qcms_data/hcn/20230329_1300                                 102     199844
AO-20230329_13  sfc/ao/qcms_data/hfmetar/20230329_1300                           12037   18942844
AO-20230329_13  sfc/ao/qcms_data/hydro/20230329_1300                              8109   10345356
AO-20230329_13  sfc/ao/qcms_data/mesonet/20230329_1300                          236316  520473216
AO-20230329_13  sfc/ao/qcms_data/metar/20230329_1300                              8102   12056780
AO-20230329_13  sfc/ao/qcms_data/nepp/20230329_1300                                 31     117940
AO-20230329_13  sfc/ao/qcms_data/rwis/20230329_1300                               3340   12418568
AO-20230329_13  sfc/ao/qcms_data/urbanet/20230329_1300                            5144   14040632

NS-20230329_13  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1300                44    2256920
NS-20230329_13  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1330                34    1753480
NS-20230329_13  nonsfc/madis_data/LDAD/snow/netCDF/20230329_1300                  3376    3669860
NS-20230329_13  nonsfc/madis_data/point/acars/netcdf/20230329_1300               39306   45225080
NS-20230329_13  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_1300         410   59713276

AO-20230329_12  sfc/ao/qcms_data/buoy/20230329_1200                                982    1261796
AO-20230329_12  sfc/ao/qcms_data/crn/20230329_1200                                 118     250216
AO-20230329_12  sfc/ao/qcms_data/hcn/20230329_1200                                  75     165392
AO-20230329_12  sfc/ao/qcms_data/hfmetar/20230329_1200                           12049   18961516
AO-20230329_12  sfc/ao/qcms_data/hydro/20230329_1200                             10099   12637836
AO-20230329_12  sfc/ao/qcms_data/mesonet/20230329_1200                          245187  539953932
AO-20230329_12  sfc/ao/qcms_data/metar/20230329_1200                              8157   12137300
AO-20230329_12  sfc/ao/qcms_data/nepp/20230329_1200                                 63     164404
AO-20230329_12  sfc/ao/qcms_data/rwis/20230329_1200                               3262   12154928
AO-20230329_12  sfc/ao/qcms_data/urbanet/20230329_1200                            5239   14293332

NS-20230329_12  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1200                43    2206576
NS-20230329_12  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1230                34    1753480
NS-20230329_12  nonsfc/madis_data/LDAD/snow/netCDF/20230329_1200                  5313    5614608
NS-20230329_12  nonsfc/madis_data/point/acars/netcdf/20230329_1200               33729   38844992
NS-20230329_12  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_1200         317   46178056
NS-20230329_12  nonsfc/madis_data/point/raob/netcdf/20230329_1200                  383   26330700

DECODE          decode/data/fxa/LDAD/crn/netCDF/20230329_1300                      243     313364
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20230329_1300                      102     139672
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20230329_1300                12037   13335604
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20230329_1300                   8309    7069164
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20230329_1300               240596  297883712
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20230329_1300                      31      78208
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20230329_1300                    3371    9426800
DECODE          decode/data/fxa/LDAD/snow/netCDF/20230329_1300                    3406    2654284
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20230329_1300                 5144    4668600

IQC             sfc/iqc1/qcms_data/buoy/20230329_1300                              602     830116
IQC             sfc/iqc1/qcms_data/crn/20230329_1300                               150     296680
IQC             sfc/iqc1/qcms_data/hcn/20230329_1300                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20230329_1300                          3577    5779084
IQC             sfc/iqc1/qcms_data/hydro/20230329_1300                            3989    5599116
IQC             sfc/iqc1/qcms_data/mesonet/20230329_1300                         83678  185280168
IQC             sfc/iqc1/qcms_data/metar/20230329_1300                            3949    5976788
IQC             sfc/iqc1/qcms_data/nepp/20230329_1300                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20230329_1300                              605    3174268
IQC             sfc/iqc1/qcms_data/urbanet/20230329_1300                          1244    3666632
IQC             sfc/iqc2/qcms_data/buoy/20230329_1300                                6     153060
IQC             sfc/iqc2/qcms_data/crn/20230329_1300                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20230329_1300                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20230329_1300                          2847    4643204
IQC             sfc/iqc2/qcms_data/hydro/20230329_1300                            1965    3267468
IQC             sfc/iqc2/qcms_data/mesonet/20230329_1300                         58682  130388952
IQC             sfc/iqc2/qcms_data/metar/20230329_1300                            1478    2359244
IQC             sfc/iqc2/qcms_data/nepp/20230329_1300                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20230329_1300                             2168    8457208
IQC             sfc/iqc2/qcms_data/urbanet/20230329_1300                           504    1698232
IQC             sfc/iqc3/qcms_data/buoy/20230329_1300                               50     203044
IQC             sfc/iqc3/qcms_data/crn/20230329_1300                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20230329_1300                                78     169220
IQC             sfc/iqc3/qcms_data/hfmetar/20230329_1300                          2593    4247980
IQC             sfc/iqc3/qcms_data/hydro/20230329_1300                             883    2021004
IQC             sfc/iqc3/qcms_data/mesonet/20230329_1300                         33898   75963288
IQC             sfc/iqc3/qcms_data/metar/20230329_1300                            1307    2108900
IQC             sfc/iqc3/qcms_data/nepp/20230329_1300                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20230329_1300                              826    3921248
IQC             sfc/iqc3/qcms_data/urbanet/20230329_1300                          1483    4302372
IQC             sfc/iqc4/qcms_data/buoy/20230329_1300                              147     313236
IQC             sfc/iqc4/qcms_data/crn/20230329_1300                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20230329_1300                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20230329_1300                          2533    4154620
IQC             sfc/iqc4/qcms_data/hydro/20230329_1300                            1321    2525580
IQC             sfc/iqc4/qcms_data/mesonet/20230329_1300                         43663   97407228
IQC             sfc/iqc4/qcms_data/metar/20230329_1300                            1210    1966892
IQC             sfc/iqc4/qcms_data/nepp/20230329_1300                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20230329_1300                                0    1129368
IQC             sfc/iqc4/qcms_data/urbanet/20230329_1300                          1991    5653652
IQC             sfc/iqc5/qcms_data/buoy/20230329_1300                              121     283700
IQC             sfc/iqc5/qcms_data/crn/20230329_1300                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20230329_1300                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20230329_1300                          2285    3768732
IQC             sfc/iqc5/qcms_data/hydro/20230329_1300                            1050    2213388
IQC             sfc/iqc5/qcms_data/mesonet/20230329_1300                         44970  100277400
IQC             sfc/iqc5/qcms_data/metar/20230329_1300                            1029    1701908
IQC             sfc/iqc5/qcms_data/nepp/20230329_1300                               31     117940
IQC             sfc/iqc5/qcms_data/rwis/20230329_1300                              120    1534968
IQC             sfc/iqc5/qcms_data/urbanet/20230329_1300                          1218    3597472