Check issues with pdf trajectories for 114–0 and 120–0 times

[fherreazcue]

The PDFs for some of the trajectories are empty files.

[fherreazcue]

It is actually worse than that, the trajectory calculation failed, and there is no clear reason why. There is no error, it simply stops.
It would seem that it has something to do with the length of the trajectory, as only the longest (120-0 and 114-0) failed.
This reminded me of a bug we used to have about maxtrajtime, which set the length of the trajectory, but doing the math again for that indicates it should not be an issue... And back then we did get an error message, not a sudden stop.
I am trying to find if there is a clear limit for lengths where the trajectories no longer work...
My guess, however, is that it is a separate issue, and that it might have to do with memory instead. RIP Does not appear to be very smart in that regard, and it seems to be keeping everything in memory before dumping it all... But I will investigate further.

[fherreazcue]

Simulations running for varying lengths of trajectories to try and find a cutoff...
Failure seems to have a random element, as somethimes trajectories fail even if short, e.g.:

$ cat Control.jobarray.o5009126.1
Skipping ripdp pre-processing...
Processing Trajectory 1: t_0=0 t_f=60 x:195 y:412 z:950 color=magenta
At line 249 of file rip.f (unit = 7, file = 'BTrajectories/Control_Nottingham_0-60_none_traj_1.in')
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x2b598d3c3c10 in next_record_r
	at ../../../libgfortran/io/transfer.c:3842
#1  0x2b598d3c5dd5 in finalize_transfer
	at ../../../libgfortran/io/transfer.c:4395
#2  0x55c6b3b1791c in ???
#3  0x55c6b3b191d4 in ???
#4  0x2b598dadf082 in ???
#5  0x55c6b3a91054 in ???

or:

$ cat Control.jobarray.o5009126.2
Skipping ripdp pre-processing...
Processing Trajectory 1: t_0=0 t_f=60 x:195 y:412 z:950 color=magenta
Processing Trajectory 2: t_0=0 t_f=60 x:195 y:412 z:925 color=light.magenta
Processing Trajectory 3: t_0=0 t_f=60 x:195 y:412 z:900 color=red.coral
Processing Trajectory 4: t_0=0 t_f=60 x:195 y:412 z:850 color=red
At line 249 of file rip.f (unit = 7, file = 'BTrajectories/Control_Nottingham_0-60_traj_4.in')
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x2ae3f1819c10 in next_record_r
	at ../../../libgfortran/io/transfer.c:3842
#1  0x2ae3f181bdd5 in finalize_transfer
	at ../../../libgfortran/io/transfer.c:4395
#2  0x55b8a1b3b91c in ???
#3  0x55b8a1b3d1d4 in ???
#4  0x2ae3f1f35082 in ???
#5  0x55b8a1ab5054 in ???
Processing Trajectory 5: t_0=0 t_f=60 x:195 y:412 z:800 color=orange
Processing Trajectory 6: t_0=0 t_f=60 x:195 y:412 z:750 color=mustard
Processing Trajectory 7: t_0=0 t_f=60 x:195 y:412 z:700 color=green

and some randomly succeeding:

$ cat Control.jobarray.o5009126.21
Skipping ripdp pre-processing...
Processing Trajectory 1: t_0=120 t_f=30 x:195 y:412 z:950 color=magenta
Processing Trajectory 2: t_0=120 t_f=30 x:195 y:412 z:925 color=light.magenta
Processing Trajectory 3: t_0=120 t_f=30 x:195 y:412 z:900 color=red.coral
Processing Trajectory 4: t_0=120 t_f=30 x:195 y:412 z:850 color=red
Processing Trajectory 5: t_0=120 t_f=30 x:195 y:412 z:800 color=orange
Processing Trajectory 6: t_0=120 t_f=30 x:195 y:412 z:750 color=mustard
Processing Trajectory 7: t_0=120 t_f=30 x:195 y:412 z:700 color=green
Processing Trajectory 8: t_0=120 t_f=30 x:195 y:412 z:650 color=dark.green
Processing Trajectory 9: t_0=120 t_f=30 x:195 y:412 z:600 color=blue
Processing Trajectory 10: t_0=120 t_f=30 x:195 y:412 z:500 color=light.cerulean
Processing Trajectory 11: t_0=120 t_f=30 x:195 y:412 z:400 color=lavender
Processing Trajectory 12: t_0=120 t_f=30 x:195 y:412 z:300 color=light.blue

Most errors seem to have to do with a Fortran runtime error: End of file, and the error eventually propagates to the conversion of the tabdiag file to a csv file, for example:

$ cat Control.jobarray.o5009126.4
Skipping ripdp pre-processing...
Processing Trajectory 1: t_0=0 t_f=70 x:195 y:412 z:950 color=magenta
Processing Trajectory 2: t_0=0 t_f=70 x:195 y:412 z:925 color=light.magenta
Processing Trajectory 3: t_0=0 t_f=70 x:195 y:412 z:900 color=red.coral
Processing Trajectory 4: t_0=0 t_f=70 x:195 y:412 z:850 color=red
Processing Trajectory 5: t_0=0 t_f=70 x:195 y:412 z:800 color=orange
At line 249 of file rip.f (unit = 7, file = 'BTrajectories/Control_Nottingham_0-70_traj_5.in')
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x2b9f9b450c10 in next_record_r
	at ../../../libgfortran/io/transfer.c:3842
#1  0x2b9f9b452dd5 in finalize_transfer
	at ../../../libgfortran/io/transfer.c:4395
#2  0x557ff178891c in ???
#3  0x557ff178a1d4 in ???
#4  0x2b9f9bb6c082 in ???
#5  0x557ff1702054 in ???
Processing Trajectory 6: t_0=0 t_f=70 x:195 y:412 z:750 color=mustard
At line 249 of file rip.f (unit = 7, file = 'BTrajectories/Control_Nottingham_0-70_traj_6.in')
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x2b4d032e5c10 in next_record_r
	at ../../../libgfortran/io/transfer.c:3842
#1  0x2b4d032e7dd5 in finalize_transfer
	at ../../../libgfortran/io/transfer.c:4395
#2  0x55f0a031391c in ???
#3  0x55f0a03151d4 in ???
#4  0x2b4d03a01082 in ???
#5  0x55f0a028d054 in ???
Processing Trajectory 7: t_0=0 t_f=70 x:195 y:412 z:700 color=green
Processing Trajectory 8: t_0=0 t_f=70 x:195 y:412 z:650 color=dark.green
Traceback (most recent call last):
  File "/Control/Control_Nottingham_0-70_tabdiag_to_csv.py", line 68, in <module>
    tabdiag_to_csv(cwd, sys.argv[1], sys.argv[2])
  File "/Control/Control_Nottingham_0-70_tabdiag_to_csv.py", line 37, in tabdiag_to_csv
    csv_file = re.sub(r"_traj_\d+", f"_{int(float(data[d0][4]))}hPa", csv_file)
                                                  ~~~~~~~~^^^
IndexError: list index out of range

This is to be expected though, if the tabdiag file is incorrrect, the csv file cannot be created...

Cannot really understand why the Fortran runtime error: End of file is happening though...

[fherreazcue]

(The first output like this is in /mnt/seaes01-data01/dmg/dmg/mbcxpfh2/SpanishPlume/SpanishPlumeAnalysis/Results/FailedLongTraj/job_output, cat Swarm_Diag_Trajecotries.jobarray.o4967414.49)

[fherreazcue]

Simulation results for trajectory length tests suggest that the cuttoff is at about 90-95 hours in length, and maybe the number of diagnostics does matter... for example, the simulation for 0-95 with no diagnostics generated a correct pdf, but with even the most basic diagnostics it failed.

Unless we can get more info, I'd suggest keeping trajectories under 90 hours, and try to use the minimum number of diagnostics possible.

[fherreazcue]

Swarm trajectories with lengths of 90h ran successfully in most cases (all except one), and the results are here.

SkewTs along trajectories of 90h were also successful, and the results are here.

[fherreazcue]

CSV files along trajectories of 90h ran successfully, results are in Dropbox.