bug in MaximumLowestMeanAbsoluteError

[amymcgovern]

When looking to make a specific plot in Australia, I discovered that there is something wrong in MaximumLowestMeanAbsoluteError in heat cases where it is NaN for both GHCN and ERA5 (though the cases differ).

If you run this:

for case in ewb_cases:
    if (case.event_type == "heat_wave"):
        case_id = case.case_id_number
        step1 = hres_heat_results[hres_heat_results.case_id_number == case_id]
        step2 = step1[step1.metric == "MaximumLowestMeanAbsoluteError"]
        step3 = step2[step2.target_source == "ERA5"]
        # does the value have a nan?
        if (step3.value.isna().any()):
            print(case)

For ERA5, you get:
IndividualCase(case_id_number=7, title='2023 Australia', start_date=datetime.datetime(2023, 11, 18, 0, 0), end_date=datetime.datetime(2023, 11, 28, 0, 0), location=BoundingBoxRegion(latitude_min=-34.5, latitude_max=-23.5, longitude_min=113.25, longitude_max=120.25), event_type='heat_wave')
IndividualCase(case_id_number=12, title='2023 China (Shanghai)', start_date=datetime.datetime(2023, 5, 24, 0, 0), end_date=datetime.datetime(2023, 6, 1, 0, 0), location=BoundingBoxRegion(latitude_min=23.75, latitude_max=33.75, longitude_min=119.0, longitude_max=124.0), event_type='heat_wave')
IndividualCase(case_id_number=13, title='2023 China (Yunnan Province)', start_date=datetime.datetime(2023, 4, 14, 0, 0), end_date=datetime.datetime(2023, 4, 23, 0, 0), location=BoundingBoxRegion(latitude_min=14.5, latitude_max=40.5, longitude_min=96.25, longitude_max=118.25), event_type='heat_wave')
IndividualCase(case_id_number=16, title='2023 Southeast Asia', start_date=datetime.datetime(2023, 4, 16, 0, 0), end_date=datetime.datetime(2023, 4, 22, 0, 0), location=BoundingBoxRegion(latitude_min=6.0, latitude_max=41.0, longitude_min=84.0, longitude_max=123.75), event_type='heat_wave')
IndividualCase(case_id_number=17, title='2023 India', start_date=datetime.datetime(2023, 2, 15, 0, 0), end_date=datetime.datetime(2023, 3, 1, 0, 0), location=BoundingBoxRegion(latitude_min=25.0, latitude_max=37.0, longitude_min=63.75, longitude_max=93.75), event_type='heat_wave')
IndividualCase(case_id_number=18, title='2021 Western Russia', start_date=datetime.datetime(2021, 6, 18, 0, 0), end_date=datetime.datetime(2021, 6, 30, 0, 0), location=BoundingBoxRegion(latitude_min=40.25, latitude_max=65.25, longitude_min=15.25, longitude_max=56.0), event_type='heat_wave')
IndividualCase(case_id_number=23, title='2022 Japan', start_date=datetime.datetime(2022, 6, 20, 0, 0), end_date=datetime.datetime(2022, 7, 5, 0, 0), location=BoundingBoxRegion(latitude_min=33.25, latitude_max=42.25, longitude_min=126.25, longitude_max=142.25), event_type='heat_wave')
IndividualCase(case_id_number=24, title='2022 India', start_date=datetime.datetime(2022, 4, 24, 0, 0), end_date=datetime.datetime(2022, 5, 4, 0, 0), location=BoundingBoxRegion(latitude_min=20.75, latitude_max=41.75, longitude_min=58.0, longitude_max=76.0), event_type='heat_wave')
IndividualCase(case_id_number=25, title='2022 East Antarctica', start_date=datetime.datetime(2022, 3, 12, 0, 0), end_date=datetime.datetime(2022, 3, 26, 0, 0), location=BoundingBoxRegion(latitude_min=-85.5, latitude_max=-64.5, longitude_min=101.0, longitude_max=145.75), event_type='heat_wave')
IndividualCase(case_id_number=26, title='2022 West Australia', start_date=datetime.datetime(2022, 1, 15, 0, 0), end_date=datetime.datetime(2022, 1, 25, 0, 0), location=BoundingBoxRegion(latitude_min=-35.5, latitude_max=-28.5, longitude_min=113.25, longitude_max=119.25), event_type='heat_wave')
IndividualCase(case_id_number=28, title='2021 New Zealand', start_date=datetime.datetime(2021, 1, 12, 18, 0), end_date=datetime.datetime(2021, 1, 18, 18, 0), location=BoundingBoxRegion(latitude_min=-48, latitude_max=-35.75, longitude_min=165.5, longitude_max=178.25), event_type='heat_wave')
IndividualCase(case_id_number=29, title='2020 Australia', start_date=datetime.datetime(2020, 11, 25, 0, 0), end_date=datetime.datetime(2020, 12, 1, 0, 0), location=BoundingBoxRegion(latitude_min=-36.25, latitude_max=-22.25, longitude_min=128.5, longitude_max=153.5), event_type='heat_wave')
IndividualCase(case_id_number=76, title='July 2024 Antarctica', start_date=datetime.datetime(2024, 7, 1, 0, 0), end_date=datetime.datetime(2024, 7, 31, 0, 0), location=BoundingBoxRegion(latitude_min=-83.5, latitude_max=-68.5, longitude_min=4.5, longitude_max=37.5), event_type='heat_wave')
IndividualCase(case_id_number=78, title='August 2024 Australia', start_date=datetime.datetime(2024, 8, 22, 0, 0), end_date=datetime.datetime(2024, 8, 30, 0, 0), location=BoundingBoxRegion(latitude_min=-27.5, latitude_max=-17.5, longitude_min=114.5, longitude_max=142.5), event_type='heat_wave')
IndividualCase(case_id_number=79, title='July/August 2024 Japan', start_date=datetime.datetime(2024, 7, 25, 0, 0), end_date=datetime.datetime(2024, 8, 5, 0, 0), location=BoundingBoxRegion(latitude_min=33.0, latitude_max=39.0, longitude_min=132.0, longitude_max=141.0), event_type='heat_wave')
IndividualCase(case_id_number=80, title='June 2024 Saudi Arabia', start_date=datetime.datetime(2024, 6, 16, 0, 0), end_date=datetime.datetime(2024, 6, 22, 0, 0), location=BoundingBoxRegion(latitude_min=18.0, latitude_max=36.0, longitude_min=22.0, longitude_max=55.0), event_type='heat_wave')
IndividualCase(case_id_number=81, title='August 2024 Europe', start_date=datetime.datetime(2024, 8, 10, 0, 0), end_date=datetime.datetime(2024, 8, 16, 0, 0), location=BoundingBoxRegion(latitude_min=34.5, latitude_max=53.5, longitude_min=3.0, longitude_max=22.0), event_type='heat_wave')
IndividualCase(case_id_number=82, title='July 2024 Ukraine', start_date=datetime.datetime(2024, 7, 12, 0, 0), end_date=datetime.datetime(2024, 7, 18, 0, 0), location=BoundingBoxRegion(latitude_min=39.0, latitude_max=55.0, longitude_min=15.0, longitude_max=39.0), event_type='heat_wave')
IndividualCase(case_id_number=83, title='June 2024 Europe', start_date=datetime.datetime(2024, 6, 20, 0, 0), end_date=datetime.datetime(2024, 6, 30, 0, 0), location=BoundingBoxRegion(latitude_min=48.25, latitude_max=55.25, longitude_min=2.0, longitude_max=9.0), event_type='heat_wave')
IndividualCase(case_id_number=85, title='May 2024 Pakistan/India', start_date=datetime.datetime(2024, 5, 23, 0, 0), end_date=datetime.datetime(2024, 5, 31, 0, 0), location=BoundingBoxRegion(latitude_min=30.0, latitude_max=43.0, longitude_min=72.0, longitude_max=86.0), event_type='heat_wave')

for GHCN, you get:
IndividualCase(case_id_number=7, title='2023 Australia', start_date=datetime.datetime(2023, 11, 18, 0, 0), end_date=datetime.datetime(2023, 11, 28, 0, 0), location=BoundingBoxRegion(latitude_min=-34.5, latitude_max=-23.5, longitude_min=113.25, longitude_max=120.25), event_type='heat_wave')
IndividualCase(case_id_number=9, title='2023 Italy', start_date=datetime.datetime(2023, 7, 7, 0, 0), end_date=datetime.datetime(2023, 7, 27, 0, 0), location=BoundingBoxRegion(latitude_min=39.5, latitude_max=52.5, longitude_min=3.0, longitude_max=17.0), event_type='heat_wave')
IndividualCase(case_id_number=12, title='2023 China (Shanghai)', start_date=datetime.datetime(2023, 5, 24, 0, 0), end_date=datetime.datetime(2023, 6, 1, 0, 0), location=BoundingBoxRegion(latitude_min=23.75, latitude_max=33.75, longitude_min=119.0, longitude_max=124.0), event_type='heat_wave')
IndividualCase(case_id_number=13, title='2023 China (Yunnan Province)', start_date=datetime.datetime(2023, 4, 14, 0, 0), end_date=datetime.datetime(2023, 4, 23, 0, 0), location=BoundingBoxRegion(latitude_min=14.5, latitude_max=40.5, longitude_min=96.25, longitude_max=118.25), event_type='heat_wave')
IndividualCase(case_id_number=16, title='2023 Southeast Asia', start_date=datetime.datetime(2023, 4, 16, 0, 0), end_date=datetime.datetime(2023, 4, 22, 0, 0), location=BoundingBoxRegion(latitude_min=6.0, latitude_max=41.0, longitude_min=84.0, longitude_max=123.75), event_type='heat_wave')
IndividualCase(case_id_number=17, title='2023 India', start_date=datetime.datetime(2023, 2, 15, 0, 0), end_date=datetime.datetime(2023, 3, 1, 0, 0), location=BoundingBoxRegion(latitude_min=25.0, latitude_max=37.0, longitude_min=63.75, longitude_max=93.75), event_type='heat_wave')
IndividualCase(case_id_number=18, title='2021 Western Russia', start_date=datetime.datetime(2021, 6, 18, 0, 0), end_date=datetime.datetime(2021, 6, 30, 0, 0), location=BoundingBoxRegion(latitude_min=40.25, latitude_max=65.25, longitude_min=15.25, longitude_max=56.0), event_type='heat_wave')
IndividualCase(case_id_number=19, title='2022 Germany', start_date=datetime.datetime(2022, 12, 23, 0, 0), end_date=datetime.datetime(2022, 12, 31, 0, 0), location=BoundingBoxRegion(latitude_min=46.0, latitude_max=55.0, longitude_min=-2.0, longitude_max=17.0), event_type='heat_wave')
IndividualCase(case_id_number=22, title='2022 France', start_date=datetime.datetime(2022, 6, 9, 0, 0), end_date=datetime.datetime(2022, 6, 21, 0, 0), location=BoundingBoxRegion(latitude_min=31.0, latitude_max=51.0, longitude_min=-9.0, longitude_max=10.0), event_type='heat_wave')
IndividualCase(case_id_number=23, title='2022 Japan', start_date=datetime.datetime(2022, 6, 20, 0, 0), end_date=datetime.datetime(2022, 7, 5, 0, 0), location=BoundingBoxRegion(latitude_min=33.25, latitude_max=42.25, longitude_min=126.25, longitude_max=142.25), event_type='heat_wave')
IndividualCase(case_id_number=24, title='2022 India', start_date=datetime.datetime(2022, 4, 24, 0, 0), end_date=datetime.datetime(2022, 5, 4, 0, 0), location=BoundingBoxRegion(latitude_min=20.75, latitude_max=41.75, longitude_min=58.0, longitude_max=76.0), event_type='heat_wave')
IndividualCase(case_id_number=25, title='2022 East Antarctica', start_date=datetime.datetime(2022, 3, 12, 0, 0), end_date=datetime.datetime(2022, 3, 26, 0, 0), location=BoundingBoxRegion(latitude_min=-85.5, latitude_max=-64.5, longitude_min=101.0, longitude_max=145.75), event_type='heat_wave')
IndividualCase(case_id_number=26, title='2022 West Australia', start_date=datetime.datetime(2022, 1, 15, 0, 0), end_date=datetime.datetime(2022, 1, 25, 0, 0), location=BoundingBoxRegion(latitude_min=-35.5, latitude_max=-28.5, longitude_min=113.25, longitude_max=119.25), event_type='heat_wave')
IndividualCase(case_id_number=28, title='2021 New Zealand', start_date=datetime.datetime(2021, 1, 12, 18, 0), end_date=datetime.datetime(2021, 1, 18, 18, 0), location=BoundingBoxRegion(latitude_min=-48, latitude_max=-35.75, longitude_min=165.5, longitude_max=178.25), event_type='heat_wave')
IndividualCase(case_id_number=29, title='2020 Australia', start_date=datetime.datetime(2020, 11, 25, 0, 0), end_date=datetime.datetime(2020, 12, 1, 0, 0), location=BoundingBoxRegion(latitude_min=-36.25, latitude_max=-22.25, longitude_min=128.5, longitude_max=153.5), event_type='heat_wave')
IndividualCase(case_id_number=76, title='July 2024 Antarctica', start_date=datetime.datetime(2024, 7, 1, 0, 0), end_date=datetime.datetime(2024, 7, 31, 0, 0), location=BoundingBoxRegion(latitude_min=-83.5, latitude_max=-68.5, longitude_min=4.5, longitude_max=37.5), event_type='heat_wave')
IndividualCase(case_id_number=78, title='August 2024 Australia', start_date=datetime.datetime(2024, 8, 22, 0, 0), end_date=datetime.datetime(2024, 8, 30, 0, 0), location=BoundingBoxRegion(latitude_min=-27.5, latitude_max=-17.5, longitude_min=114.5, longitude_max=142.5), event_type='heat_wave')
IndividualCase(case_id_number=79, title='July/August 2024 Japan', start_date=datetime.datetime(2024, 7, 25, 0, 0), end_date=datetime.datetime(2024, 8, 5, 0, 0), location=BoundingBoxRegion(latitude_min=33.0, latitude_max=39.0, longitude_min=132.0, longitude_max=141.0), event_type='heat_wave')
IndividualCase(case_id_number=80, title='June 2024 Saudi Arabia', start_date=datetime.datetime(2024, 6, 16, 0, 0), end_date=datetime.datetime(2024, 6, 22, 0, 0), location=BoundingBoxRegion(latitude_min=18.0, latitude_max=36.0, longitude_min=22.0, longitude_max=55.0), event_type='heat_wave')
IndividualCase(case_id_number=81, title='August 2024 Europe', start_date=datetime.datetime(2024, 8, 10, 0, 0), end_date=datetime.datetime(2024, 8, 16, 0, 0), location=BoundingBoxRegion(latitude_min=34.5, latitude_max=53.5, longitude_min=3.0, longitude_max=22.0), event_type='heat_wave')
IndividualCase(case_id_number=82, title='July 2024 Ukraine', start_date=datetime.datetime(2024, 7, 12, 0, 0), end_date=datetime.datetime(2024, 7, 18, 0, 0), location=BoundingBoxRegion(latitude_min=39.0, latitude_max=55.0, longitude_min=15.0, longitude_max=39.0), event_type='heat_wave')
IndividualCase(case_id_number=83, title='June 2024 Europe', start_date=datetime.datetime(2024, 6, 20, 0, 0), end_date=datetime.datetime(2024, 6, 30, 0, 0), location=BoundingBoxRegion(latitude_min=48.25, latitude_max=55.25, longitude_min=2.0, longitude_max=9.0), event_type='heat_wave')
IndividualCase(case_id_number=85, title='May 2024 Pakistan/India', start_date=datetime.datetime(2024, 5, 23, 0, 0), end_date=datetime.datetime(2024, 5, 31, 0, 0), location=BoundingBoxRegion(latitude_min=30.0, latitude_max=43.0, longitude_min=72.0, longitude_max=86.0), event_type='heat_wave')

[linear-code]

EWB-99

[amymcgovern]

While looking at this on the WP-MIP data, I asked Claude to dig Into it farther and it narrowed it down

Case 76's MaximumLowestMeanAbsoluteError is 0/1545 non-null — literally every value is NaN, across every forecast center and HRES, at every lead time. That's distinct from the babj/egrr/kwbc/rjtd pattern (partial lead-time coverage per center); here nothing at all came through for any source.

I checked whether this is unique to case 76, and it isn't — it's a clean split across all 16 heat wave cases:

Cases | MaximumLowestMeanAbsoluteError coverage | Longitude sign -- | -- | -- 70, 71, 72, 73, 74, 75, 77, 84 (all US/Canada/Mexico) | ~50–56% populated | negative (western hemisphere) 76, 78, 79, 80, 81, 82, 83, 85 (Antarctica, Australia, Japan, Saudi Arabia, Europe×2, Ukraine, Pakistan/India) | 0% — fully NaN | positive (0–180°E)

The other two metrics don't show this split at all — RootMeanSquaredError is ~99.9% populated and MaximumMeanAbsoluteError is ~55–63% populated for all 16 cases, including case 76. So it's not that the underlying forecast/target data is missing for these regions (RMSE proves the data is there and valid) — it's specific to how MaximumLowestMeanAbsoluteError gets computed, and it correlates exactly with the sign of the case's bounding-box longitude, not with hemisphere/latitude or data availability generally.

Looking at the metric itself (metrics.py:1331), it derives the target's "warmest daily-minimum" via a day-of-year groupby that only counts a day if utils.min_if_all_timesteps_present sees the exact expected timestep count (utils.py:257-274); since that target value is shared across every forecast center's MAE calculation, if it comes out NaN, every center inherits NaN identically — which is exactly the "all sources, all lead times" pattern we see. That part of the logic doesn't reference longitude, so the actual trigger is almost certainly upstream of this repo — in whatever data-windowing step feeds this specific metric during Daniel's WP-MIP CSV generation (gs://brightband-scratch/darothen/wp-mip/results/), which isn't part of this codebase.

Bottom line: it's not a "missing forecast" issue like the earlier partial-lead-time cases — it looks like a real bug specific to MaximumLowestMeanAbsoluteError's computation for any case whose bounding box uses positive (0–180°E) longitude, and it's worth flagging to Daniel since it silently zeroes out that metric for half the heat wave cases (76, 78–83, 85).

Case 76's MaximumLowestMeanAbsoluteError is 0/1545 non-null — literally every value is NaN, across every forecast center and HRES, at every lead time. That's distinct from the babj/egrr/kwbc/rjtd pattern (partial lead-time coverage per center); here nothing at all came through for any source.

I checked whether this is unique to case 76, and it isn't — it's a clean split across all 16 heat wave cases:

Cases MaximumLowestMeanAbsoluteError coverage Longitude sign
70, 71, 72, 73, 74, 75, 77, 84 (all US/Canada/Mexico) ~50–56% populated negative (western hemisphere)
76, 78, 79, 80, 81, 82, 83, 85 (Antarctica, Australia, Japan, Saudi Arabia, Europe×2, Ukraine, Pakistan/India) 0% — fully NaN positive (0–180°E)
The other two metrics don't show this split at all — RootMeanSquaredError is ~99.9% populated and MaximumMeanAbsoluteError is ~55–63% populated for all 16 cases, including case 76. So it's not that the underlying forecast/target data is missing for these regions (RMSE proves the data is there and valid) — it's specific to how MaximumLowestMeanAbsoluteError gets computed, and it correlates exactly with the sign of the case's bounding-box longitude, not with hemisphere/latitude or data availability generally.

Looking at the metric itself (metrics.py:1331), it derives the target's "warmest daily-minimum" via a day-of-year groupby that only counts a day if utils.min_if_all_timesteps_present sees the exact expected timestep count (utils.py:257-274); since that target value is shared across every forecast center's MAE calculation, if it comes out NaN, every center inherits NaN identically — which is exactly the "all sources, all lead times" pattern we see. That part of the logic doesn't reference longitude, so the actual trigger is almost certainly upstream of this repo — in whatever data-windowing step feeds this specific metric during Daniel's WP-MIP CSV generation (gs://brightband-scratch/darothen/wp-mip/results/), which isn't part of this codebase.

Bottom line: it's not a "missing forecast" issue like the earlier partial-lead-time cases — it looks like a real bug specific to MaximumLowestMeanAbsoluteError's computation for any case whose bounding box uses positive (0–180°E) longitude, and it's worth flagging to Daniel since it silently zeroes out that metric for half the heat wave cases (76, 78–83, 85).