Can't call open_mfdataset without creating chunked dask arrays

[TomNicholas]

What happened?

Passing chunks=None to xr.open_dataset/open_mfdataset is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed. However chunks=None doesn't currently work for xr.open_mfdataset as it gets silently coerced internally to chunks={}, which creates dask chunks aligned with the on-disk files.

Offending line of code:

xarray/xarray/backends/api.py

Line 1040 in 12123be

open_kwargs = dict(engine=engine, chunks=chunks or {}, **kwargs)

What did you expect to happen?

Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.

Minimal Complete Verifiable Example

ds = xr.tutorial.open_dataset("air_temperature")

ds1 = ds.isel(time=slice(None, 1000))
ds2 = ds.isel(time=slice(1000, None))

ds1.to_netcdf('air1.nc')
ds2.to_netcdf('air2.nc')

combined = xr.open_mfdataset(['air1.nc', 'air2.nc'], chunks=None)

print(type(combined['air'].data))

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

dask.array.core.Array

Anything else we need to know?

As the default is None, changing this without changing the default would be a breaking change. But the current behaviour is also not intended.

Environment

main

[dcherian]

Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.

Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)

We could change the default to chunks={} in anticipation though.

[TomNicholas]

Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)

😅

It's still broken at the moment though - I had a (ridiculous) case where I don't care that concat will load everything in memory, I just want to completely avoid creating dask.array objects, and right now there is no possible input option to open_mfdataset to do that.

We could change the default to chunks={} in anticipation though.

That's probably more useful, as well as actually being consistent.

[Illviljan]

See #5704 for changing chunks={} and more discussion.

[weiji14]

Passing chunks=None to xr.open_dataset/open_mfdataset is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed.

Agree with having the ability to totally avoid dask and/or any chunking manager, and have the possibility of letting the backend engine handle things on its own. Will also call out that the lazy-indexing shouldn't just return NumPy arrays, but allow for other Array API types (e.g. CuPy arrays).

To elaborate on my use-case, I'm hitting into issues at xarray-contrib/cupy-xarray#81 (comment) because xr.open_mfdataset can't avoid using dask at all, and dask defaults to returning numpy.arrays, but I want CuPy arrays 🙂 (edit: went down the 🐇🕳️ and found #8733 (comment) that mentioned cupy!). Using xr.open_dataset(..., chunks=None) works fine though, it is just that chunks=chunks or {} part that is tripping things up in xr.open_mfdataset.