Improve error handling and messages

[RussellManser-NCO]

Current behavior

• Error handling is sparse in scripts such as run_graphcast.py, gen_aigfs_ics.py, and gen_aigefs_ics.py, particularly at points where the model could fail (e.g., missing files)
• Python output does not always print in real time while the model is running
• The following python warnings are consistently printed during forecast jobs
  • FutureWarning: In a future version of xarray ...
  • WARNING:absl:Skipping gradient checkpointing for sequence length of 1

GitHub gist with more details.

Example job output which contains the described warning messages can be found on Cactus at /lfs/h1/ops/prod/output/20251006/aigfs_forecast_00.o70293747

Expected behavior

• Add descriptive error messages when errors and warnings occur (general application standards)
• Add error handling at any potential failure points (e.g., missing files) so that errors are communicated quickly and clearly (see err_chk/err_exit in production utilities)
• Fix deprecation, syntax, and other common errors/warnings (interpreted code)

Reproduction steps

1. Run the forecast for aigfs or aigefs
2. Redirect output to a log file
3. Search for the warning messages above

Additional information

In the output files there is an additional warning compath.py:115: SyntaxWarning: invalid escape sequence '\d'. This is unrelated to MLGlobal and is documented in NCO-HPC/prod_util#1.

Proposed implementation/solution

I have one quick suggestion:

• In run_graphcast.py, gen_aigfs_ics.py, and gen_aigefs_ics.py, I suggest modifying the shebang to #!/usr/bin/env -S python3 -u to allow unbuffered output