MADIS RT RUN-TIME STATUS for Wed Mar 29 17:32:01 GMT 2023 on vm-lnx-madis-process1a
     INFO RT VM STATUS for Node  vm-lnx-madis-process1a Time Wed Mar 29 17:32:02 GMT 2023

     PASS METAR processing tbl                     file:  128320 Mar 28 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  102  daily<1800min  102  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 15 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 760
     FAIL RSAS analysis file missing:   madis/proc/sfc/ao/sfc_netcdf/20230329_1600   
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20230329_1600:51 20230329_1500:253  #PUBLIC-DP 20230329_1600:285 20230329_1500:1182  #PUBLIC-DP-profile 20230329_1600:9 20230329_1500:18      
     PASS Load resource                 m15 < 9 cores 100% (goal > 60%)   -- % of the time M15 < 9  Samples: 1052    #10=+ 0 #9 0 #8 69 #7 337
     PASS Open file resource                       check: open file < 5000 Max Files: 2846 #<1000: 403 #<10000 1052
     PASS Process count                            check: processes < 1000 Max Processes: 499
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 210 #Delayed: 0
     INFO AO runtime avg sfcanl                     15  seconds   1069/70
     INFO AO runtime avg sfcncdf                     1  seconds   49/44
     INFO AO runtime avg sfcver                     79  seconds   4840/61
     INFO AO runtime avg sfcqcmsg                   17  seconds   1196/70
     INFO AO runtime avg sfcqcout meso              32  seconds   6272/195
     INFO AO runtime avg sfcqcout_late meso         80  seconds   21049/261
     INFO AO runtime avg FSL_qcout5m_               42  seconds   8865/211
     INFO AO runtime avg FSL_analrun15m_           106  seconds   7427/70
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1055 #Delayed: 1 Competed timely:  989
     INFO IQC runtime avg qcstg1_2                   5  seconds   86888/16703
     INFO IQC runtime avg qcstg1_2 meso             19  seconds   40467/2103
     INFO IQC runtime avg qcstg1_2_late meso        20  seconds   29281/1423
     INFO IQC runtime avg sfcing                    28  seconds   30493/1052
     INFO IQC runtime avg sfchqc                    61  seconds   64739/1049
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(9998)
     PASS RT Throughput                            -32434 secs -- check: Avg time < 180 sec. #Hrs: 15 Max secs: 1258176
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20230329_1600: #CPYRMA: 0 #NJWXNET: 633
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.49 % OBS FAILED  CHECK ( 7876 OF 30902) AVGTHR 1.7928 RATIO 0.100E+06 AVGEPS 0.2532E-01 FRMSE 38.971 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.84 % OBS FAILED  CHECK ( 3735 OF 77090) AVGTHR 5.2652 RATIO 0.278 AVGEPS 0.1446E-01 FRMSE 19.376 TRMSE 1.4658
     PASS QC POTENTIAL TEMPERATURE (K)             4.23 % OBS FAILED  CHECK ( 3362 OF 79545) AVGTHR 3.6354 RATIO 0.100E+06 AVGEPS 0.2540E-01 FRMSE 1351.0 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             5.24 % OBS FAILED  CHECK ( 4036 OF 77090) AVGTHR 3.9362 RATIO 0.257 AVGEPS 0.1569E-01 FRMSE 21.542 TRMSE 1.0130
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.07 % OBS FAILED  CHECK (  808 OF 75575) AVGTHR 4.4118 RATIO 0.301 AVGEPS 0.1372E-01 FRMSE 7.6885 TRMSE 1.3295
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.36 % OBS FAILED  CHECK ( 1029 OF 75575) AVGTHR 4.5337 RATIO 0.316 AVGEPS 0.1398E-01 FRMSE 8.6619 TRMSE 1.4324
     PASS QC 3H ALTIMETER CHANGE (MB)              0.95 % OBS FAILED  CHECK (  261 OF 27400) AVGTHR 1.2097 RATIO 0.231 AVGEPS 0.1621E-01 FRMSE 49.322 TRMSE 0.27937
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.60 % OBS FAILED  CHECK (   47 OF   712) AVGTHR 1.5704 RATIO 0.403 AVGEPS 0.5944E-01 FRMSE 3.9147 TRMSE 0.63322
     PASS QC ALTIMETER (MB)                       24.46 % OBS FAILED  CHECK ( 7560 OF 30902) AVGTHR 1.9312 RATIO 0.100E+06 AVGEPS 0.2544E-01 FRMSE 39.627 TRMSE 99999.
     FAIL MXST_P sizing check dailed to no information in the log file madis/proc//sfc/ao/output/sfcanl.out.230881600_3
     PASS AO Possible Seg fault                    0  #logs checked: 2267  
     PASS IQC Possible Seg fault                   0  #logs checked: 4630  
     PASS NONSFC Possible Seg fault                0  #logs checked: 798  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230329_411786.log
     INFO MADIS DB Connection cnts                 Info:500 [2023-Mar-29 17:30:44.845324] - LastUploadStats:mode:upload_sfc,pid:44478,aborted:0,fileCount:1,wholeFileErrors:0,lines:34887,inserts:20866,updates:14021,duplicates:0,errors:0,stationsAdded:30,stationsUpdated:3,publishLines:0,publishErrors:0,notes:./202303291729_receipt.sfcd2.txt,run ms:62401
     FAIL AFWS files missing               # in last hour: 0
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_2:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_2:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcanl.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcanl.out.230881300_1:
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871100_1:

  COMPARE Mesonet data providers cnt                39     for 20230329_1500:
  COMPARE DECODE STN TBLS                           149        Newest:   214669 Mar 29 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Aug 21  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   214669 Mar 29 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Aug 21  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (16): Newest: 6 Mar 29 17:27 madis/proc/sfc/ao/qc122/qcsumry17.dat Oldest:
  COMPARE sda IO Writes 2308815                     Local Disk MB/s: 23 RS MB/s: 0 PQ MB/s: 18+ Tot secs: 3469+
  COMPARE sda IO Reads(2308815                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3469+
  COMPARE NetCDF REC Writes 2308815                 Avg DECODE rec/s: 96 Avg AO rec/s: 94 Avg NON-SFC rec/s: 8 Tot secs: 3469
     INFO Network  2308815                          eth0 14 MB/s TX    3 MB/s RX     Tot secs: 3469  
  COMPARE usr1 Disk 2308815                         23 WRITE MB/s | 1 READ MB/s | 5759 Avg WRT TPS | 16594 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P -1
AO-20230329_15  sfc/ao/qcms_data/buoy/20230329_1500                                874    1139108
AO-20230329_15  sfc/ao/qcms_data/crn/20230329_1500                                 246     436072
AO-20230329_15  sfc/ao/qcms_data/hcn/20230329_1500                                 101     198568
AO-20230329_15  sfc/ao/qcms_data/hfmetar/20230329_1500                           12020   18916392
AO-20230329_15  sfc/ao/qcms_data/hydro/20230329_1500                              3659    5218956
AO-20230329_15  sfc/ao/qcms_data/mesonet/20230329_1500                          279921  616229796
AO-20230329_15  sfc/ao/qcms_data/metar/20230329_1500                              8026   11945516
AO-20230329_15  sfc/ao/qcms_data/nepp/20230329_1500                                 58     157144
AO-20230329_15  sfc/ao/qcms_data/rwis/20230329_1500                               3350   12452368
AO-20230329_15  sfc/ao/qcms_data/urbanet/20230329_1500                            4937   13490012

NS-20230329_15  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1500                46    2357608
NS-20230329_15  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1530                34    1753480
NS-20230329_15  nonsfc/madis_data/LDAD/snow/netCDF/20230329_1500                  1282    1567484
NS-20230329_15  nonsfc/madis_data/point/acars/netcdf/20230329_1500               31411   36193200
NS-20230329_15  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_1500         228   33224996

AO-20230329_14  sfc/ao/qcms_data/buoy/20230329_1400                                845    1106164
AO-20230329_14  sfc/ao/qcms_data/crn/20230329_1400                                 245     434620
AO-20230329_14  sfc/ao/qcms_data/hcn/20230329_1400                                 102     199844
AO-20230329_14  sfc/ao/qcms_data/hfmetar/20230329_1400                           11891   18715668
AO-20230329_14  sfc/ao/qcms_data/hydro/20230329_1400                              6658    8673804
AO-20230329_14  sfc/ao/qcms_data/mesonet/20230329_1400                          249898  550299288
AO-20230329_14  sfc/ao/qcms_data/metar/20230329_1400                              8066   12004076
AO-20230329_14  sfc/ao/qcms_data/nepp/20230329_1400                                 60     160048
AO-20230329_14  sfc/ao/qcms_data/rwis/20230329_1400                               3352   12459128
AO-20230329_14  sfc/ao/qcms_data/urbanet/20230329_1400                            5487   14953012

NS-20230329_14  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1400                44    2256920
NS-20230329_14  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_1430                34    1753480
NS-20230329_14  nonsfc/madis_data/LDAD/snow/netCDF/20230329_1400                  1942    2230124
NS-20230329_14  nonsfc/madis_data/point/acars/netcdf/20230329_1400               38063   43803088
NS-20230329_14  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_1400         340   49525476
NS-20230329_14  nonsfc/madis_data/point/raob/netcdf/20230329_1400                    1     180508

DECODE          decode/data/fxa/LDAD/crn/netCDF/20230329_1500                      246     316448
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20230329_1500                      101     138828
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20230329_1500                12020   13317040
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20230329_1500                   3691    3688788
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20230329_1500               294697  364536144
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20230329_1500                      58     100348
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20230329_1500                    3382    9454036
DECODE          decode/data/fxa/LDAD/snow/netCDF/20230329_1500                    1286    1170284
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20230329_1500                 4937    4492236

IQC             sfc/iqc1/qcms_data/buoy/20230329_1500                              624     855108
IQC             sfc/iqc1/qcms_data/crn/20230329_1500                               153     301036
IQC             sfc/iqc1/qcms_data/hcn/20230329_1500                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20230329_1500                          3576    5777528
IQC             sfc/iqc1/qcms_data/hydro/20230329_1500                            3561    5106060
IQC             sfc/iqc1/qcms_data/mesonet/20230329_1500                         92005  203566260
IQC             sfc/iqc1/qcms_data/metar/20230329_1500                            3930    5948972
IQC             sfc/iqc1/qcms_data/nepp/20230329_1500                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20230329_1500                              614    3204688
IQC             sfc/iqc1/qcms_data/urbanet/20230329_1500                          1246    3671952
IQC             sfc/iqc2/qcms_data/buoy/20230329_1500                                5     151924
IQC             sfc/iqc2/qcms_data/crn/20230329_1500                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20230329_1500                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20230329_1500                          2860    4663432
IQC             sfc/iqc2/qcms_data/hydro/20230329_1500                              33    1041804
IQC             sfc/iqc2/qcms_data/mesonet/20230329_1500                         71849  159303684
IQC             sfc/iqc2/qcms_data/metar/20230329_1500                            1459    2331428
IQC             sfc/iqc2/qcms_data/nepp/20230329_1500                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20230329_1500                             2176    8484248
IQC             sfc/iqc2/qcms_data/urbanet/20230329_1500                           495    1674292
IQC             sfc/iqc3/qcms_data/buoy/20230329_1500                               51     204180
IQC             sfc/iqc3/qcms_data/crn/20230329_1500                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20230329_1500                                77     167944
IQC             sfc/iqc3/qcms_data/hfmetar/20230329_1500                          2561    4198188
IQC             sfc/iqc3/qcms_data/hydro/20230329_1500                              25    1032588
IQC             sfc/iqc3/qcms_data/mesonet/20230329_1500                         43987   98118732
IQC             sfc/iqc3/qcms_data/metar/20230329_1500                            1308    2110364
IQC             sfc/iqc3/qcms_data/nepp/20230329_1500                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20230329_1500                              820    3900968
IQC             sfc/iqc3/qcms_data/urbanet/20230329_1500                          1282    3767712
IQC             sfc/iqc4/qcms_data/buoy/20230329_1500                              148     314372
IQC             sfc/iqc4/qcms_data/crn/20230329_1500                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20230329_1500                                89     183256
IQC             sfc/iqc4/qcms_data/hfmetar/20230329_1500                          2518    4131280
IQC             sfc/iqc4/qcms_data/hydro/20230329_1500                              55    1067148
IQC             sfc/iqc4/qcms_data/mesonet/20230329_1500                         51443  114492108
IQC             sfc/iqc4/qcms_data/metar/20230329_1500                            1196    1946396
IQC             sfc/iqc4/qcms_data/nepp/20230329_1500                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20230329_1500                                0    1129368
IQC             sfc/iqc4/qcms_data/urbanet/20230329_1500                          1850    5278592
IQC             sfc/iqc5/qcms_data/buoy/20230329_1500                              130     293924
IQC             sfc/iqc5/qcms_data/crn/20230329_1500                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20230329_1500                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20230329_1500                          2290    3776512
IQC             sfc/iqc5/qcms_data/hydro/20230329_1500                               7    1011852
IQC             sfc/iqc5/qcms_data/mesonet/20230329_1500                         54812  121890432
IQC             sfc/iqc5/qcms_data/metar/20230329_1500                            1013    1678484
IQC             sfc/iqc5/qcms_data/nepp/20230329_1500                               58     157144
IQC             sfc/iqc5/qcms_data/rwis/20230329_1500                              120    1534968
IQC             sfc/iqc5/qcms_data/urbanet/20230329_1500                          1219    3600132