Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [unreleased](https://github.com/mllam/mllam-data-prep/compare/v0.7.0...HEAD)

### Fixes
- fix bug where coordinate selection of an unshared dimension isn't applied to subsequent ouput variables when an output variable without this dimension is processed before the others [\#90](https://github.com/mllam/mllam-data-prep/pull/90) @zweihuehner & @leifdenby
- fix bug where coordinate selection of an unshared dimension isn't applied to subsequent output variables when an output variable without this dimension is processed before the others [\#90](https://github.com/mllam/mllam-data-prep/pull/90) @zweihuehner & @leifdenby

## [v0.7.0](https://github.com/mllam/mllam-data-prep/release/tag/v0.7.0)

Expand Down Expand Up @@ -90,7 +90,7 @@ interface and addresses bugs around optional dependencies for

### Added

- add access to CLI via `mllam_data_prep` and add tests for CLI with/without `dask.distributed` ![\25](https://github.com/mllam/mllam-data-prep/pull/25).
- add access to CLI via `mllam_data_prep` and add tests for CLI with/without `dask.distributed` ![\#25](https://github.com/mllam/mllam-data-prep/pull/25).
- add optional output path argument to parser. ![\#26](https://github.com/mllam/mllam-data-prep/pull/26)

### Changed
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ python -m pip install mllam-data-prep[dask-distributed]

## Developing `mllam-data-prep`

To work on developing `mllam-data-prep` it easiest to install and manage the dependencies with [pdm](https://pdm.fming.dev/). To get started clone your fork of [the main repo](https://github.com/mllam/mllam-data-prep) locally:
To work on developing `mllam-data-prep` it is easiest to install and manage the dependencies with [pdm](https://pdm.fming.dev/). To get started clone your fork of [the main repo](https://github.com/mllam/mllam-data-prep) locally:

```bash
git clone https://github.com/<your-github-username>/mllam-data-prep
Expand All @@ -41,7 +41,7 @@ pdm use --venv in-project
pdm install
```

All the linting is handelled by `pre-commit` which can be setup to automatically be run on each `git commit` by installing the git commit hook:
All the linting is handled by `pre-commit` which can be setup to automatically be run on each `git commit` by installing the git commit hook:

```bash
pdm run pre-commit install
Expand Down Expand Up @@ -93,7 +93,7 @@ When you run the above command the console will print a URL to the dask
dashboard, which you can open in a browser to monitor the progress of the
dataset creation (and see the memory usage of the workers).

![example of using mllam-data-prep with dask.distrubted for parallel processing](docs/using_dask_distributed.png)
![example of using mllam-data-prep with dask.distributed for parallel processing](docs/using_dask_distributed.png)

### Usage as a python module

Expand Down Expand Up @@ -395,7 +395,7 @@ The `inputs` section defines the source datasets to extract data from. Each sour
- `function`: the function used to derive a variable. This should be a string with the full namespace of the function, e.g. `mllam_data_prep.ops.derived_variables.physical_field.calculate_toa_radiation`.
- `kwargs`: arguments to `function`. This is a dictionary where each key is the named argument to `function` and each value is the input to the function. Here we distinguish between values to be extracted/selected from the input dataset and values supplied by the users themselves. Arguments with values to be extracted from the input dataset need to be prefixed with "ds_input." to distinguish them from other arguments. See the 'Derived Variables' section for more details.
- `coord_ranges`
- `varable` : the variable to extract the coordinate from
- `variable` : the variable to extract the coordinate from
`start`: start value of the variable
`end`: end value of the variable

Expand Down Expand Up @@ -439,4 +439,4 @@ Currently, the following derived variables are included as part of `mllam-data-p

The schema version of the configuration file is defined by the `schema_version` attribute at the top of the configuration file. This is used to keep track of changes to the configuration file format. The schema version is used to check that the configuration file is compatible with the version of `mllam-data-prep` that you are using. If the schema version of the configuration file is not compatible with the version of `mllam-data-prep` that you are using you will get an error message telling you that the schema version is not compatible.

The schema version is updated whenever the configuration format changes, with the new schema version matching the minimum version of `mllam-data-prep` that is required to use the new configuration format. As `mllam-data-prep` is still in rapid development (and hasn't reached version `v1.0.0` yet) we unfortunately make no gaurantee about backward compatibility. However, the [CHANGELOG.md](CHANGELOG.md) will always contain migration instructions when the config format changes.
The schema version is updated whenever the configuration format changes, with the new schema version matching the minimum version of `mllam-data-prep` that is required to use the new configuration format. As `mllam-data-prep` is still in rapid development (and hasn't reached version `v1.0.0` yet) we unfortunately make no guarantee about backward compatibility. However, the [CHANGELOG.md](CHANGELOG.md) will always contain migration instructions when the config format changes.
3 changes: 3 additions & 0 deletions mllam_data_prep/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ def validate_config(config_inputs):
Parameters
----------
config_inputs: Dict[str, InputDataset]
Dictionary mapping input dataset names to their configuration

Returns
-------
None
Raises InvalidConfigException if validation fails
"""

for input_dataset_name, input_dataset in config_inputs.items():
Expand Down
4 changes: 2 additions & 2 deletions mllam_data_prep/ops/chunking.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ def check_chunk_size(ds, chunks):

Returns
-------
ds: xr.Dataset
Dataset with chunking applied
None
Logs a warning if chunk size exceeds CHUNK_MAX_SIZE_WARNING
"""

for var_name, var_data in ds.data_vars.items():
Expand Down
4 changes: 1 addition & 3 deletions mllam_data_prep/ops/mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims):
dim_mapping : dict
The mapping of the input dimensions to the architecture
dimensions.
arch_dim : str
The name of the architecture dimension to map to
expected_input_var_dims : list
The list of dimensions that each variable in the input dataset
should have
Expand Down Expand Up @@ -126,7 +124,7 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims):
combined_dim_name=arch_dim,
)
else:
# TODO: this will have to involved xrarrays MultiIndex, but lets leave
# TODO: this will have to involve xarray's MultiIndex, but let's leave
# this until we need it
raise NotImplementedError(len(dims))
# set a flag we can use later to identify which coordinate the variables
Expand Down
32 changes: 30 additions & 2 deletions mllam_data_prep/ops/selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,21 @@ def select_by_kwargs(ds, **coord_ranges):

def check_point_in_dataset(coord, point, ds):
"""
check that the requested point is in the data.
Check that the requested point is in the data.

Parameters
----------
coord : str
Name of the coordinate to check
point : Any
The point value to check for in the coordinate
ds : xr.Dataset
Dataset containing the coordinate to check

Returns
-------
None
Raises ValueError if point is not found in the coordinate
"""
if point is not None and point not in ds[coord].values:
raise ValueError(
Expand All @@ -104,7 +118,21 @@ def check_point_in_dataset(coord, point, ds):

def check_step(sel_step, coord, ds):
"""
check that the step requested is exactly what the data has
Check that the step requested is exactly what the data has.

Parameters
----------
sel_step : pd.Timedelta or datetime.timedelta
The requested step size
coord : str
Name of the coordinate to check
ds : xr.Dataset
Dataset containing the coordinate to check

Returns
-------
None
Raises ValueError if step size is not constant or doesn't match requested step
"""
all_steps = ds[coord].diff(dim=coord).values
first_step = all_steps[0].astype("timedelta64[s]").astype(datetime.timedelta)
Expand Down
2 changes: 1 addition & 1 deletion mllam_data_prep/ops/subsetting.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ def extract_variable(ds, var_name, coords_to_sample=dict()):
coordinates

Returns
----------
-------
da: xr.DataArray
Extracted variable (subsetted along the specified coordinates)
"""
Expand Down
Loading