Missing ensemble members and forecast files in early NCO parallel runs

[GwenChen-NOAA]

While test running EVS aigefs verification code, I noticed that ensemble members are missing in NCO parallel runs under /lfs/h1/ops/para/com/aigefs/v1.0 for dates before 20250928. For example,

/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250928/12
/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250926/00
/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250925/00
/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250918/00

These missing members caused warnings or errors in the prep/stats job log files. The warning messages (see below) from METplus that I mentioned in the EAGLE stand-up meeting are caused by missing members too.

WARNING: get_data_plane() -> The valid time has changed, 20251014_060000 != 20251014_180000 in "/lfs/h2/emc/vpppg/noscrub/Lichuan.Chen/evs_aigefs/v2.0/prep/aigefs/atmos.20250928/aigefs/aigefs.ens20.t18z.grid3.f372.grib2"
WARNING: get_data_plane() -> The valid time has changed, 20251014_060000 != 20251014_180000 in "/lfs/h2/emc/vpppg/noscrub/Lichuan.Chen/evs_aigefs/v2.0/prep/aigefs/atmos.20250928/aigefs/aigefs.ens21.t18z.grid3.f372.grib2"
WARNING: get_data_plane() -> The valid time has changed, 20251014_060000 != 20251014_180000 in "/lfs/h2/emc/vpppg/noscrub/Lichuan.Chen/evs_aigefs/v2.0/prep/aigefs/atmos.20250928/aigefs/aigefs.ens22.t18z.grid3.f372.grib2"
...

The early parallel runs also have different directory structure and only contain ensemble products (no member forecast files). For example,

/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250923/00
/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250922/00
/lfs/h1/ops/para/com/aigefs/v1.0/aigefs.20250921/00

For 20250909-20250920, files are only available for 00Z cycle (no forecast files for 06, 12, 18 cycles).

These issues prevent EVS aigefs generating proper prep files before 20250928. As a result, the earliest valid date for stats calculation using existing NCO parallel runs is 20251015.

[junwang-noaa]

@GwenChen-NOAA Thanks for reporting the missing files.

@RussellManser-NCO It looks to me that we are missing members 20-29 for 00/06/12/18z cycle for 20250926 and 20250928, and also members 11-14 and 20-30 are missing for 00/06/12/18 20250925.

Is it possible that we regenerate the forecasts for these cycles?

[RussellManser-NCO]

@junwang-noaa Thank you for taking a closer look.

There are a few reasons for missing/incomplete files

• We started transferring member files on 20250926. For prior dates only the ensstat files were transferred
• On ~20251003 we fixed a transfer list error that left out some members

On re-runs, I have some additional details to point out.

• There have been multiple re-runs that addressed different fixes
• The most recent set of re-runs was only for the 00Z cycle on these dates:
  20250918/00
  20250919/00
  20250920/00
  20250924/00
  20250926/00
  20250927/00
  20250928/00
  20250929/00

That is, all 06/12/18 cycles prior to 2025930/00 do not include the fix in NOAA-EMC/MLGlobal#59.

With all that said, which dates are being targeted for verification? @GwenChen-NOAA

[GwenChen-NOAA]

@RussellManser-NCO, we do not have a target date for verification start. Ideally, we want consistent model forecast files with the most recent fixes. The earlier we can go back to, the more stats we can get. I have informed Jason that with missing files in the early parallel runs, we may have shorter verification stats for the briefing on 10/30.

[RussellManser-NCO]

@GwenChen-NOAA Thank you. So beginning with 20250930/00 verification should be consistent and you should have all members available to you on Cactus. You mentioned that 20251015 is the earliest date you would have for verification though. Are your verification stats not able to operate on the period from 20250930/00 and forward?

[GwenChen-NOAA]

@RussellManser-NCO, for forecasts initialized on 20250930, forecast lead 384 hours (fhr=384) is valid on 20251016. EVS code is set up to run on valid date when observations become available. Below is an example of EVS plots:

On 20251016, EVS calculates stats for fhr=24 from 20251015 fcst, fhr=48 from 20251014 fcst, fhr=72 from 20251013 fcst, ..., fhr=384 from 20250930 fcst. So, it needs 16 days of continuous prep files prior to the valid date to run stats jobs. Samira have helped me generated prep files back to 20250928.

[RussellManser-NCO]

@RussellManser-NCO, for forecasts initialized on 20250930, forecast lead 384 hours (fhr=384) is valid on 20251016. EVS code is set up to run on valid date when observations become available...

I see, thank you!

[junwang-noaa]

@GwenChen-NOAA as we discussed this morning, I am asking Russell to generate two more days (20250928/20250930) AIGEFS forecast. so we will have 32 days forecast by 20251030. Please let me know if that will enable EVS to generate AIGEFS forecast. Or if you can let us know if actually how many days EVS needs to generate 16 day forecasts. My understanding is that with 17 days forecast, we should have observations for all corresponding 16 days forecast for evaluation. Please let me know if that is not the case. Thanks

[GwenChen-NOAA]

@junwang-noaa, if you want to have stats back generated from 20251001, Russell will need to rerun AIGEFS back from 20250915 to 20250929. Also, EVS aigefs/gefs is run on 2-day lag when observations become available, so today's real-time run will generate stats for 20251018. The earlier Russell can go back to, the more days of stats we can generate.

[junwang-noaa]

@GwenChen-NOAA Thanks for the information. Since we have forecast data since 20251001, do we start to get evaluations on 20251017? As you said, you are generating stats for 20251018 today, and we have data since 20251001. So we should see all the 16 days evaluations, as we already have 2 days of data (20251001 and 20251002) for FH=384 forecast. In other words, we will get evaluation from 20251017 and forward, right?

[GwenChen-NOAA]

@GwenChen-NOAA Thanks for the information. Since we have forecast data since 20251001, do we start to get evaluations on 20251017? As you said, you are generating stats for 20251018 today, and we have data since 20251001. So we should see all the 16 days evaluations, as we already have 2 days of data (20251001 and 20251002) for FH=384 forecast. In other words, we will get evaluation from 20251017 and forward, right?

@junwang-noaa, yes, that's correct.

[junwang-noaa]

Thanks, @GwenChen-NOAA. Please let us know when the EVS evaluation page is up; we really appreciate your work!

[RussellManser-NCO]

@GwenChen-NOAA @junwang-noaa I've looked into retrieving archived data to run retrospective cases prior to 20250930. Unfortunately, 0.25 degree GEFS members are not archived on HPSS, likely due to their large volume. It looks like these files are not available on AWS either. I was not able to find any other GEFS data for the dates above in para COM or test COM. I am not aware of another location where 0.25 degree GEFS member files may be stored.

[junwang-noaa]

Thanks for letting us know. I think it might be OK as we will have day 16 forecast verification for about 2 weeks