LatLon coordinates in boundary datastore don't match grid_index

[sadamov]

When generating a boundary zarr archive, lat*lon==grid_index should hold true, where lat and lon are 1D vectors. This however is not the case, as can be reproduced following the steps below. I have attached the two config-files in a zip-folder. Make sure to update the absolute path to the DANRA datastore for cropping in the era5.datastore.yaml before proceeding.

Now follow these steps in order:

python -m mllam_data_prep ./neural-lam/test_example/danra.datastore.yaml
python -m mllam_data_prep ./neural-lam/test_example/era5.datastore.yaml

Open a Python console and inspect the created era5.datastore.zarr

import xarray as xr
ds=xr.open_zarr("era5.datastore.zarr/")
ds=ds.set_index(grid_index=["latitude","longitude"])
ds_unstacked=ds.unstack("grid_index")
try:
    assert len(ds_unstacked.longitude)*len(ds_unstacked.latitude) == len(ds.grid_index)
except AssertionError:
    print(f"Dimensions mismatch:")
    print(f"len(longitude): {len(ds_unstacked.longitude)}")
    print(f"len(latitude): {len(ds_unstacked.latitude)}")
    print(f"longitude * latitude: {len(ds_unstacked.longitude) * len(ds_unstacked.latitude)}")
    print(f"len(grid_index): {len(ds.grid_index)}")
    raise

For me this returns a significant mismatch:

Dimensions mismatch:
len(longitude): 321
len(latitude): 147
longitude * latitude: 47187
len(grid_index): 28143

yaml-files.zip

[observingClouds]

@sadamov, what branch are you running on? #45?

[sadamov]

@sadamov, what branch are you running on? #45?

I am on a research branch from @joeloskarsson: pip install git+https://github.com/joeloskarsson/mllam-data-prep.git@arcdist_fix building on top of #45. Sorry I only just realized that cropping is still in development.

[joeloskarsson]

Hmm, are we getting confused with assuming things to be rectangular here? The boundary points are not in a regular N x M grid, so should lat*lon==grid_index really hold? I am thinking more that we should have grid_index different latitudes and grid_index different longitudes.

[sadamov]

Ah I see your point, I am still confused about the boundary handling then:

1. Precropping based on domain_cropping yields already a convex-hull not a cartesian grid?
2. During model eval a boundary mask is applied to further specify the boundary range (convex hull)?
3. Plotting of the boundary cannot be done on a grid but rather pointwise latlon (mapping required)

So the issue is maybe not at all with MDP but with my understanding of when which coordinates/projections are 1D/2D vectors, and when they are supposed to form a cartesian grid and when not. I made some first attempts to plotting boundaries during the eval-step, but clearly this is not looking great. Any help is welcome!
https://github.com/sadamov/neural-lam/tree/plot_boundaries

[joeloskarsson]

I'll try to clear some things up 😄:

Precropping based on domain_cropping yields already a convex-hull not a cartesian grid?

Yes, using domain_cropping in the config will keep only the points at the given distance from the convex hull of the referenced interior data. This set of points will in general not lay on a regular grid, no.

During model eval a boundary mask is applied to further specify the boundary range (convex hull)?

No, there is no boundary mask in neural-lam right now. All points in the boundary datastore are considered boundary input points.

Plotting of the boundary cannot be done on a grid but rather pointwise latlon (mapping required)

Yes, this will have to be the case. As plots anyhow are in the crs of the interior, boundary points would still not lay on a regular grid (if they were originally in e.g. a regular lat-lon grid).

[sadamov]

Okay I understand, I will close this Issue then, as the output is actually what one would expect and open another issue with plotting boundaries in neural-lam where I link my current progress.