RIP - ripdp stops processing after ~ 110 simulation times

[fherreazcue]

The script calling ripdp seems to work ok with the spanish plume data, but for some unknown reason it stops processing files after the simulation time 110 (approx)...

At this point, the files generated in RIPDP use over 156G disk space, but there does not seem to be any other reason, and it is not a hard limit (some times it reaches 161G).

Same thing happened with data from Control, Half_hgt, Double_hgt, Zero_hgt and GWaves.

Errors (for each data set in the order listed above):

--------------------------------------------------------------------------------------------------------------
****  Reading model output at
       forecast time=   110.000000
        (YYMMDDHH = 15070314)
 Processing basic variables.
NetCDF error in ripdp_wrfarw.  Marker =    57.0000000
   NetCDF: HDF error                                                            
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG
./singularity_rip.sh: line 83: r_latest.sif: command not found
./singularity_rip.sh: line 84: syntax error near unexpected token `else'
./singularity_rip.sh: line 84: `else'
--------------------------------------------------------------------------------------------------------------
****  Reading model output at
       forecast time=   109.000000
        (YYMMDDHH = 15070313)
 Processing basic variables.

 Checking for hydrometeor variables.
    Did not find QNCLOUD.
    Did not find QNRAIN.
    Did not find QNICE.
    Did not find QNWFA.
    Did not find QNIFA.
 Processing other variables in output file.
NetCDF error in ripdp_wrfarw.  Marker =    119.000000
   NetCDF: HDF error                                                            
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG
./singularity_rip.sh: line 83: r_latest.sif: command not found
./singularity_rip.sh: line 84: syntax error near unexpected token `else'
./singularity_rip.sh: line 84: `else'
--------------------------------------------------------------------------------------------------------------
****  Reading model output at
       forecast time=   110.000000
        (YYMMDDHH = 15070314)
 Processing basic variables.
NetCDF error in ripdp_wrfarw.  Marker =    57.0000000
   NetCDF: HDF error                                                            
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG
./singularity_rip.sh: line 83: r_latest.sif: command not found
./singularity_rip.sh: line 84: syntax error near unexpected token `else'
./singularity_rip.sh: line 84: `else'
--------------------------------------------------------------------------------------------------------------
****  Reading model output at
       forecast time=   110.000000
        (YYMMDDHH = 15070314)
 Processing basic variables.
 NetCDF error in ripdp_wrfarw.  Marker =    57.0000000
   NetCDF: HDF error                                                            
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG
./singularity_rip.sh: line 83: r_latest.sif: command not found
./singularity_rip.sh: line 84: syntax error near unexpected token `else'
./singularity_rip.sh: line 84: `else'
--------------------------------------------------------------------------------------------------------------
****  Reading model output at
       forecast time=   102.000000
        (YYMMDDHH = 15070306)
 Processing basic variables.

 Checking for hydrometeor variables.
    Did not find QNCLOUD.
    Did not find QNRAIN.
    Did not find QNICE.
    Did not find QNWFA.
    Did not find QNIFA.
 Processing other variables in output file.
 NetCDF error in ripdp_wrfarw.  Marker =    119.000000
   NetCDF: HDF error                                                            
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG
./singularity_rip.sh: line 83: r_latest.sif: command not found
./singularity_rip.sh: line 84: syntax error near unexpected token `else'
./singularity_rip.sh: line 84: `else'
--------------------------------------------------------------------------------------------------------------

[fherreazcue]

Commit 08fe543 fixed another issue with the ripdp inputs file, which provides a workaround, by calling ripdp in "segments".
For example, using:

./singularity_rip.sh -od=Results/Control -wd=/mnt/seaes01-data01/dmg/dmg/mbessdl2/Spanish_Plume/WRF/run-zrek/ --t_0=0 --t_f=99 --dt=1 --noTraj --noPlot

and

./singularity_rip.sh -od=Results/Control -wd=/mnt/seaes01-data01/dmg/dmg/mbessdl2/Spanish_Plume/WRF/run-zrek/ --t_0=100 --t_f=168 --dt=1 --noTraj --noPlot

successfully pre-processes all the times in the simulation (0 - 168) are in two badges.

However, ripdp keeps track of the times it has pre-processed in a file called *.xtimes. For the example above it is Control.xtimes. This is important because, if the commands above are ran one after the other, the second badge overwrites the file, deleting all the info on the times 0-99.
It is just a list of the processed times, and the total number of processed times, so it can be easily modified to include all of the times, instead of just the latest badge, but it may be easier to back it up after the first command.