Selection of variables by coord values results in missing/extra output variables with nans

[ealerskans]

I am in the process of adding test that checks that e.g. tests that the chosen variables are the ones actually present in the output dataset in #60 . When doing this I wanted to try with this type of input dataset in the config file to see that it works as it should

..
inputs:
  danra_height_levels:
    #path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/height_levels.zarr
    path: /dmidata/projects/cloudphysics/danra/data/v0.5.0/height_levels.zarr
    dims: [time, x, y, altitude]
    variables:
      u:
        altitude:
          values: [100, 50]
          units: m
      v:
        altitude:
          values: [100, 75]
          units: m
    dim_mapping:
      time:
        method: rename
        dim: time
      state_feature:
        method: stack_variables_by_var_name
        dims: [altitude]
        name_format: "{var_name}{altitude}m"
      grid_index:
        method: stack
        dims: [x, y]
    target_output_variable: state
...

I am guessing that this is how we're supposed to select variables from the height levels, @leifdenby ?

However, using this type of variable selection resulted in my tests failing because there was a mismatch between the specified variables and what was actually present in the output dataset. I then tested this with mllam-data-prep v0.5.0 as well and found the same issue.

Using this type of selection I am getting the following variables in my output dataset when I inspect it

['u100m', 'u50m', 'v100m', 'v50m']

So I am not getting v75m as expected but instead v50m. Also, I had a look at the output values of the dataset and found that all the v50m values are nans.

Furthermore, adding a variable selection like this

...
inputs:
  danra_height_levels:
    #path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/height_levels.zarr
    path: /dmidata/projects/cloudphysics/danra/data/v0.5.0/height_levels.zarr
    dims: [time, x, y, altitude]
    variables:
      u:
        altitude:
          values: [100, 50, ]
          units: m
      v:
        altitude:
          values: [100, 75, ]
          units: m
      t:
        altitude:
          values: [30, ]
          units: m
    dim_mapping:
      time:
        method: rename
        dim: time
      state_feature:
        method: stack_variables_by_var_name
        dims: [altitude]
        name_format: "{var_name}{altitude}m"
      grid_index:
        method: stack
        dims: [x, y]
    target_output_variable: state
...

I am getting the following variables in the output dataset

['u100m', 'u50m', 'v100m', 'v50m', 't100m', 't50m'

So not only am I missing v75m and t30m but I am getting v50m, t100m and t50m instead, where all of them are actually nans.

[aviralgarg05]

I would like to solve the issue, please assign me

[Jayant-kernel]

Hi! I’d like to work on this if it’s unclaimed (I saw #65). Plan: reproduce with the config in the issue, then fix selection so each var is .sel()’d independently by its requested coord values before stacking (avoid union/broadcast that creates extra NaN vars). I’ll add a regression test to ensure expected outputs (v75m/t30m) appear and no extra NaN vars are produced.