MADIS RT RUN-TIME STATUS for Thu Mar 30 06:32:01 GMT 2023 on vm-lnx-madis-process1a
     INFO RT VM STATUS for Node  vm-lnx-madis-process1a Time Thu Mar 30 06:32:01 GMT 2023

     PASS METAR processing tbl                     file:  128320 Mar 28 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  102  daily<1800min  102  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 17 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 727
     FAIL RSAS analysis file missing:   madis/proc/sfc/ao/sfc_netcdf/20230330_0500   
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20230330_0500:866 20230330_0400:912  #PUBLIC-DP 20230330_0500:793 20230330_0400:1367  #PUBLIC-DP-profile 20230330_0500:14 20230330_0400:24      
     PASS Load resource                 m15 < 9 cores 91% (goal > 60%)   -- % of the time M15 < 9  Samples: 392    #10=+ 0 #9 32 #8 163 #7 187
     PASS Open file resource                       check: open file < 5000 Max Files: 2835 #<1000: 94 #<10000 392
     PASS Process count                            check: processes < 1000 Max Processes: 531
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 78 #Delayed: 0
     INFO AO runtime avg sfcanl                     20  seconds   543/26
     INFO AO runtime avg sfcncdf                     1  seconds   3/3
     INFO AO runtime avg sfcver                      1  seconds   2/2
     INFO AO runtime avg sfcqcmsg                   20  seconds   531/26
     INFO AO runtime avg sfcqcout meso              38  seconds   2786/72
     INFO AO runtime avg sfcqcout_late meso         87  seconds   7425/85
     INFO AO runtime avg FSL_qcout5m_               48  seconds   3867/79
     INFO AO runtime avg FSL_analrun15m_            45  seconds   1177/26
     PASS IQC Throughput                           2  -- check: #"conflicts" <30 delayed %Delayed: 2% #Sample: 395 #Delayed: 11 Competed timely:  320
     INFO IQC runtime avg qcstg1_2                   5  seconds   33961/6791
     INFO IQC runtime avg qcstg1_2 meso             25  seconds   19637/763
     INFO IQC runtime avg qcstg1_2_late meso        14  seconds   6158/425
     INFO IQC runtime avg sfcing                    37  seconds   14359/381
     INFO IQC runtime avg sfchqc                    77  seconds   29316/379
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(9999)
     FAIL RT Throughput missing logs entries.             time < 180 sec. No hours to check:
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20230330_0500: #CPYRMA: 0 #NJWXNET: 636
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.73 % OBS FAILED  CHECK ( 7658 OF 29764) AVGTHR 1.7892 RATIO 0.100E+06 AVGEPS 0.2471E-01 FRMSE 40.200 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.00 % OBS FAILED  CHECK ( 3024 OF 75677) AVGTHR 5.2498 RATIO 0.258 AVGEPS 0.1365E-01 FRMSE 19.224 TRMSE 1.3544
     PASS QC POTENTIAL TEMPERATURE (K)             3.32 % OBS FAILED  CHECK ( 2601 OF 78292) AVGTHR 3.6296 RATIO 0.100E+06 AVGEPS 0.2507E-01 FRMSE 1526.9 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             4.34 % OBS FAILED  CHECK ( 3285 OF 75677) AVGTHR 3.9244 RATIO 0.253 AVGEPS 0.1492E-01 FRMSE 21.843 TRMSE 0.99282
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.79 % OBS FAILED  CHECK (  582 OF 74017) AVGTHR 4.4055 RATIO 0.278 AVGEPS 0.1342E-01 FRMSE 7.5864 TRMSE 1.2244
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.84 % OBS FAILED  CHECK (  619 OF 74017) AVGTHR 4.5244 RATIO 0.272 AVGEPS 0.1351E-01 FRMSE 8.0717 TRMSE 1.2311
     PASS QC 3H ALTIMETER CHANGE (MB)              1.04 % OBS FAILED  CHECK (  277 OF 26565) AVGTHR 1.2115 RATIO 0.100E+06 AVGEPS 0.1649E-01 FRMSE 41.224 TRMSE 99999.
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.36 % OBS FAILED  CHECK (   42 OF   660) AVGTHR 1.5932 RATIO 0.413 AVGEPS 0.6550E-01 FRMSE 3.6078 TRMSE 0.65790
     PASS QC ALTIMETER (MB)                       24.37 % OBS FAILED  CHECK ( 7253 OF 29764) AVGTHR 1.9232 RATIO 0.100E+06 AVGEPS 0.2424E-01 FRMSE 41.263 TRMSE 99999.
     FAIL MXST_P sizing check dailed to no information in the log file madis/proc//sfc/ao/output/sfcanl.out.230890500_3
     PASS AO Possible Seg fault                    0  #logs checked: 2267  
     PASS IQC Possible Seg fault                   0  #logs checked: 4461  
     PASS NONSFC Possible Seg fault                0  #logs checked: 781  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230330_411955.log
     INFO MADIS DB Connection cnts                 Info:486 [2023-Mar-30 06:31:46.764477] - LastUploadStats:mode:upload_sfc,pid:12271,aborted:0,fileCount:1,wholeFileErrors:0,lines:44850,inserts:30802,updates:14048,duplicates:0,errors:2,stationsAdded:0,stationsUpdated:3,publishLines:0,publishErrors:0,notes:./202303300630_receipt.sfcd2.txt,run ms:81209
     FAIL AFWS files missing               # in last hour: 0
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 23089
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890600_2:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 23088
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230881900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882000_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230882300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230890600_2:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 23089
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 23088
                                     madis/proc/sfc/ao/output/sfcanl.out.230880000_4:
                                     madis/proc/sfc/ao/output/sfcanl.out.230881300_1:

  COMPARE Mesonet data providers cnt                38     for 20230330_0400:
  COMPARE DECODE STN TBLS                           149        Newest:   214669 Mar 29 21:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Aug 21  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   214669 Mar 29 21:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Aug 21  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (05): Newest: 8 Mar 30 06:27 madis/proc/sfc/ao/qc122/qcsumry06.dat Oldest:
  COMPARE sda IO Writes 2308904                     Local Disk MB/s: 25 RS MB/s: 0 PQ MB/s: 23+ Tot secs: 3503+
  COMPARE sda IO Reads(2308904                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3503+
  COMPARE NetCDF REC Writes 2308904                 Avg DECODE rec/s: 99 Avg AO rec/s: 88 Avg NON-SFC rec/s: 11 Tot secs: 3503
     INFO Network  2308904                          eth0 15 MB/s TX    3 MB/s RX     Tot secs: 3503  
  COMPARE usr1 Disk 2308904                         25 WRITE MB/s | 1 READ MB/s | 6326 Avg WRT TPS | 16670 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P -1
AO-20230330_04  sfc/ao/qcms_data/buoy/20230330_0400                                849    1110708
AO-20230330_04  sfc/ao/qcms_data/crn/20230330_0400                                 245     434620
AO-20230330_04  sfc/ao/qcms_data/hcn/20230330_0400                                 102     199844
AO-20230330_04  sfc/ao/qcms_data/hfmetar/20230330_0400                           11914   18751456
AO-20230330_04  sfc/ao/qcms_data/hydro/20230330_0400                              2532    3920652
AO-20230330_04  sfc/ao/qcms_data/mesonet/20230330_0400                          279928  616245168
AO-20230330_04  sfc/ao/qcms_data/metar/20230330_0400                              7787   11595620
AO-20230330_04  sfc/ao/qcms_data/nepp/20230330_0400                                 36     125200
AO-20230330_04  sfc/ao/qcms_data/rwis/20230330_0400                               3361   12489548
AO-20230330_04  sfc/ao/qcms_data/urbanet/20230330_0400                            5812   15817512

NS-20230330_04  nonsfc/madis_data/LDAD/profiler/netCDF/20230330_0400                44    2256920
NS-20230330_04  nonsfc/madis_data/LDAD/profiler/netCDF/20230330_0430                34    1753480
NS-20230330_04  nonsfc/madis_data/LDAD/snow/netCDF/20230330_0400                  1272    1557444
NS-20230330_04  nonsfc/madis_data/point/acars/netcdf/20230330_0400               33640   38743176
NS-20230330_04  nonsfc/madis_data/point/acarsProfiles/netcdf/20230330_0400         237   34534856
NS-20230330_04  nonsfc/madis_data/point/raob/netcdf/20230330_0400                    3     317420

AO-20230330_03  sfc/ao/qcms_data/buoy/20230330_0300                                918    1189092
AO-20230330_03  sfc/ao/qcms_data/crn/20230330_0300                                 246     436072
AO-20230330_03  sfc/ao/qcms_data/hcn/20230330_0300                                 102     199844
AO-20230330_03  sfc/ao/qcms_data/hfmetar/20230330_0300                           12016   18910168
AO-20230330_03  sfc/ao/qcms_data/hydro/20230330_0300                              2476    3856140
AO-20230330_03  sfc/ao/qcms_data/mesonet/20230330_0300                          279927  616242972
AO-20230330_03  sfc/ao/qcms_data/metar/20230330_0300                              7774   11576588
AO-20230330_03  sfc/ao/qcms_data/nepp/20230330_0300                                 36     125200
AO-20230330_03  sfc/ao/qcms_data/rwis/20230330_0300                               3530   13060768
AO-20230330_03  sfc/ao/qcms_data/urbanet/20230330_0300                            5459   14878532

NS-20230330_03  nonsfc/madis_data/LDAD/profiler/netCDF/20230330_0300                44    2256920
NS-20230330_03  nonsfc/madis_data/LDAD/profiler/netCDF/20230330_0330                36    1854168
NS-20230330_03  nonsfc/madis_data/LDAD/snow/netCDF/20230330_0300                  1270    1555436
NS-20230330_03  nonsfc/madis_data/point/acars/netcdf/20230330_0300               41029   47196192
NS-20230330_03  nonsfc/madis_data/point/acarsProfiles/netcdf/20230330_0300         315   45886976
NS-20230330_03  nonsfc/madis_data/point/HDW/netcdf/20230330_0300                   281    1298200

DECODE          decode/data/fxa/LDAD/crn/netCDF/20230330_0400                      245     315420
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20230330_0400                      102     139672
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20230330_0400                11914   13201288
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20230330_0400                   2534    2841864
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20230330_0400               306597  379196944
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20230330_0400                      36      82308
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20230330_0400                    3389    9471368
DECODE          decode/data/fxa/LDAD/snow/netCDF/20230330_0400                    1272    1160484
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20230330_0400                 5812    5237736

IQC             sfc/iqc1/qcms_data/buoy/20230330_0400                              612     841476
IQC             sfc/iqc1/qcms_data/crn/20230330_0400                               153     301036
IQC             sfc/iqc1/qcms_data/hcn/20230330_0400                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20230330_0400                          3461    5598588
IQC             sfc/iqc1/qcms_data/hydro/20230330_0400                            2475    3854988
IQC             sfc/iqc1/qcms_data/mesonet/20230330_0400                         91497  202450692
IQC             sfc/iqc1/qcms_data/metar/20230330_0400                            3716    5635676
IQC             sfc/iqc1/qcms_data/nepp/20230330_0400                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20230330_0400                              603    3167508
IQC             sfc/iqc1/qcms_data/urbanet/20230330_0400                          1236    3645352
IQC             sfc/iqc2/qcms_data/buoy/20230330_0400                                7     154196
IQC             sfc/iqc2/qcms_data/crn/20230330_0400                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20230330_0400                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20230330_0400                          2840    4632312
IQC             sfc/iqc2/qcms_data/hydro/20230330_0400                               0    1003788
IQC             sfc/iqc2/qcms_data/mesonet/20230330_0400                         72064  159775824
IQC             sfc/iqc2/qcms_data/metar/20230330_0400                            1439    2302148
IQC             sfc/iqc2/qcms_data/nepp/20230330_0400                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20230330_0400                             2158    8423408
IQC             sfc/iqc2/qcms_data/urbanet/20230330_0400                           495    1674292
IQC             sfc/iqc3/qcms_data/buoy/20230330_0400                               49     201908
IQC             sfc/iqc3/qcms_data/crn/20230330_0400                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20230330_0400                                78     169220
IQC             sfc/iqc3/qcms_data/hfmetar/20230330_0400                          2594    4249536
IQC             sfc/iqc3/qcms_data/hydro/20230330_0400                              19    1025676
IQC             sfc/iqc3/qcms_data/mesonet/20230330_0400                         44718   99724008
IQC             sfc/iqc3/qcms_data/metar/20230330_0400                            1275    2062052
IQC             sfc/iqc3/qcms_data/nepp/20230330_0400                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20230330_0400                              741    3633948
IQC             sfc/iqc3/qcms_data/urbanet/20230330_0400                          1996    5666952
IQC             sfc/iqc4/qcms_data/buoy/20230330_0400                              146     312100
IQC             sfc/iqc4/qcms_data/crn/20230330_0400                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20230330_0400                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20230330_0400                          2518    4131280
IQC             sfc/iqc4/qcms_data/hydro/20230330_0400                              46    1056780
IQC             sfc/iqc4/qcms_data/mesonet/20230330_0400                         51895  115484700
IQC             sfc/iqc4/qcms_data/metar/20230330_0400                            1128    1846844
IQC             sfc/iqc4/qcms_data/nepp/20230330_0400                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20230330_0400                                0    1129368
IQC             sfc/iqc4/qcms_data/urbanet/20230330_0400                          2553    7148572
IQC             sfc/iqc5/qcms_data/buoy/20230330_0400                              115     276884
IQC             sfc/iqc5/qcms_data/crn/20230330_0400                                26     116632
IQC             sfc/iqc5/qcms_data/hcn/20230330_0400                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20230330_0400                          2296    3785848
IQC             sfc/iqc5/qcms_data/hydro/20230330_0400                               2    1006092
IQC             sfc/iqc5/qcms_data/mesonet/20230330_0400                         54701  121646676
IQC             sfc/iqc5/qcms_data/metar/20230330_0400                            1079    1775108
IQC             sfc/iqc5/qcms_data/nepp/20230330_0400                               36     125200
IQC             sfc/iqc5/qcms_data/rwis/20230330_0400                              207    1829028
IQC             sfc/iqc5/qcms_data/urbanet/20230330_0400                          1286    3778352