fix: parsing relative paths in workflow yml by LuukBlom · Pull Request #1365 · Deltares/hydromt

LuukBlom · 2026-02-05T14:45:28Z

Issue addressed

Fixes #1350

Explanation

Added the validator resolve_paths_or_catalogs in HydromtGlobalConfig, which is used to validate the workflow yaml.

Refactored read_workflow_yaml to now actually return HydromtModelSetup instead of 3 dicts.

Moved all of the validation of the steps from the Model class into the step classes in model_config.py
Now has:

RawStep: this class validates the structure of the yaml, no runtime function binding yet.
HydromtModelStep: class that validates the hydromt_step part, has the function as an attribute.

Cyclic import errors

While fixing this, I encountered some problematic cyclic import errors, and after some thought I have re-introduced hydromt.io again.

The problem is that some of the readers depend on other parts of the hydromt codebase, when they should only read the files and return content. This means importing any reader (even just read_toml), will import a lot of other hydromt modules.

My idea for the separation:

_io: just file io, should not depend / know about any other part of hydromt
parsers: take in raw data and return validated hydromt objects/classes
readers: combine io and parsers to read and validate files, also exposes _io functions like read_toml for backwards compatibility

This separation means we can import the low-level _io of read_yaml or read_toml pretty much everywhere in hydromt since they have no hydromt dependencies.

General Checklist

Data/Catalog checklist

data/catalogs/predefined_catalogs.yml has not been modified.
None of the old data_catalog.yml files have been changed
data/changelog.rst has been updated
new file uses LF line endings (done automatically if you used update_versions.py)
New file has been tested locally
Tests have been added using the new file in the test suite

Additional Notes (optional)

Add any additional notes or information that may be helpful.

dalmijn · 2026-02-09T08:22:10Z

I haven't reviewed it, but did notice the io.py submodule returning which we deliberately removed in #1308.
Please keep io in the readers and writers.

LuukBlom · 2026-02-09T09:13:36Z

I haven't reviewed it, but did notice the io.py submodule returning which we deliberately removed in #1308. Please keep io in the readers and writers.

Please read the PR description for my reasoning why I added it again.

sonarqubecloud · 2026-02-09T10:41:11Z

❌ The last analysis has failed.

See analysis details on SonarQube Cloud

dalmijn · 2026-02-09T10:49:27Z

I realise that, but it can cause issues as specified in PR #1308 and it's subsequent issue.

LuukBlom · 2026-02-09T11:00:08Z

I realise that, but it can cause issues as specified in PR #1308 and it's subsequent issue.

Ah okay I wasnt aware of that. Will rename to something else

sonarqubecloud · 2026-02-09T13:44:21Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot

Pull request overview

This PR refactors workflow YAML parsing/validation to properly resolve relative paths and to return a validated HydromtModelSetup object, while also moving step validation/execution logic into step model classes and introducing a low-level IO module to reduce import cycles.

Changes:

Added hydromt._io and migrated YAML/TOML + URI YAML reading into it to reduce cyclic imports.
Refactored read_workflow_yaml to return a validated HydromtModelSetup via a new parse_workflow parser.
Moved step parsing/validation/execution into RawStep/HydromtModelStep and updated CLI/tests accordingly.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
hydromt/_io.py	Introduces low-level YAML/TOML/URI helpers used across the package to avoid heavy imports.
hydromt/readers.py	Refactors `read_workflow_yaml` to return `HydromtModelSetup` and delegates parsing to `parse_workflow`.
hydromt/parsers.py	New parser module to merge defaults + validate workflow dicts into HydroMT objects.
hydromt/_validators/model_config.py	Adds `RawStep`, step binding/execution, global config path/catalog resolution, and new setup schema.
hydromt/model/model.py	Updates build/update to execute bound step objects; improves component instantiation from validated configs.
hydromt/model/steps.py	Adds a global step registry populated by `@hydromt_step`.
hydromt/cli/main.py	Updates CLI build/update/validate flow to use `HydromtModelSetup` output.
hydromt/data_catalog/data_catalog.py	Switches YAML loading helper to `hydromt._io.yml_from_uri_or_path`.
hydromt/_validators/data_catalog_v0x.py	Switches YAML loading helper to `hydromt._io.yml_from_uri_or_path`.
hydromt/_validators/data_catalog_v1x.py	Switches YAML loading helper to `hydromt._io.yml_from_uri_or_path`.
hydromt/model/components/config.py	Uses low-level IO functions from `hydromt._io`.
hydromt/model/example/example_model.py	Adjusts ExampleModel init/component wiring to updated patterns/types.
tests/test_readers.py	Adds workflow YAML parsing/path-resolution test coverage for new behavior.
tests/cli/test_cli.py	Updates assertions for new step validation error message format.
tests/model/test_model.py	Updates expectations for new step-not-found error type/message.
tests/data_catalog/test_data_catalog.py	Updates to new YAML loader function location/name.
tests/components/test_config_component.py	Adjusts imports to new `read_yaml` location.
tests/_validators/test_model_config.py	Updates to new step model constructor signature + error types/messages.
tests/_validators/test_data_catalog_validation_v0x.py	Updates to new YAML loader function location/name.
tests/_validators/test_data_catalog_validation_v1x.py	Updates to new YAML loader function location/name.
examples/working_with_models.ipynb	Updates workflow reading example to new return type.
examples/working_with_models_basics.ipynb	Notebook metadata change (kernelspec display name).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

deltamarnix

Let's have a look at the names of the config classes.
We also talked about what Model should know about. Should it know about the config classes, or should the config classes know about the model? Perhaps don't convert during the validation, but only validate and keep the data raw.

rename _deep_merge to deep_merge update _steps_contain_write update example model with properties

hboisgon

Some food for thoughts and a couple of warnings on the current implementation.

As highlighted in the ther PR, I think we need a design session about in general how do we build a model with hydromt before we can really finalise this PR. This is major change for python users for now if we do this we should do it well and avoid changing (again) in the future.

hboisgon · 2026-02-12T03:36:21Z

Depending on what we choose but there are 4 pages in the docs that need updating (search for read_workflow_yaml) + update changelog
Also should API be updated? Users will need to know what object they are getting when calling read_workflow_yaml so this should be documented.

hboisgon · 2026-02-12T03:37:21Z

 from pydantic import ValidationError

 from hydromt import __version__
+from hydromt._io import read_yaml


Better keep using readers to import read_yaml then if I understand well?

hboisgon · 2026-02-12T03:41:21Z

            mode=mode,
-            data_libs=data_libs,
-            **kwargs,
+            **workflow.globals_.model_dump(),


Not sure I am fond of model_dump here. Can we simplify for beginner python users? This is basic hydromt use building a model, I dont want to make it too complex at this stage.

hboisgon · 2026-02-12T03:42:48Z

 from pathlib import Path
 from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, Union, cast

+from hydromt._io import read_toml, read_yaml


Guess you can undo and go back to readers. Most likely plugins will copy and modify this component so we should limit calls to private functions

hboisgon · 2026-02-12T03:44:25Z

Why any change in this file? I dont think we want model components to be initialised this way. _config is private for a reason. Same for the property definition, not sure we want to envourage plugins to define their components this way.

hboisgon · 2026-02-12T05:14:20Z

+        Access the model type with `.modeltype`, the global configuration with `.globals_`,
+        and the runtime steps with `.steps`.


Not clear what modeltype, globals and steps are: str, ?, dict? needs documentation of the HydroMTModelSetup class in public API

I guess .global is not possible? Then why not .global_ to try and keep close to the workflow yaml? global goes to model init so could also be .model_init?

hboisgon · 2026-02-12T05:20:57Z

+    # Determine model type
+    resolved_modeltype = modeltype or data.get("modeltype")
+    if resolved_modeltype is None:
+        raise ValueError("Model type not specified in workflow or arguments")


Suggested change

raise ValueError("Model type not specified in workflow or arguments")

raise ValueError("Model type not specified in workflow YAML or arguments")

hboisgon · 2026-02-12T05:25:33Z

+    if (
+        abs_path
+        and root is not None
+        and (skip_abspath_sections is None or "global" not in skip_abspath_sections)


I'm having trouble following but what happens if "global" in skip_abspath_sections? Does this means that relative paths in steps will not be converted? Ie using a local data file instead of catalog entry for the setup functions?

hboisgon · 2026-02-12T05:27:14Z

+        raise ValueError("Model type not specified in workflow or arguments")
+
+    # Context for path resolution
+    context = {}


Do you expect anything else in context than just root? If not why make it a dictionnary?

hboisgon · 2026-02-12T05:32:33Z

Reviewed in the other PR. See comments there.

LuukBlom added 6 commits January 29, 2026 18:27

updated pydantic basemodels and validators to handle path resolution

1bcb954

WIP on making all tests green

1512b9e

refactor model step validation and execution

487f30f

add execute and o_model_step to step classes

9765641

move tests into test_readers.py

d705f53

revert unnecessary changes

b279691

LuukBlom and others added 2 commits February 9, 2026 11:12

update notebooks and ExampleModel class

942f251

Merge branch 'main' into fix/parsing-relative-paths-in-workflow-yml

e7b0b54

rename io to _io. Also, export _io functions in readers

aef2ee7

LuukBlom requested a review from deltamarnix February 9, 2026 12:42

deltamarnix reviewed Feb 9, 2026

View reviewed changes

Comment thread hydromt/_validators/data_catalog_v0x.py Outdated

rename '_yml_from_uri_or_path' to 'yml_from_uri_or_path'

dcf384d

add read_workflow_yaml to exported readers again

a01e0c6

deltamarnix requested a review from Copilot February 9, 2026 15:59

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Comment thread hydromt/readers.py

Comment thread hydromt/readers.py Outdated

Comment thread hydromt/_io.py Outdated

Comment thread hydromt/_io.py

Comment thread hydromt/model/steps.py Outdated

Comment thread hydromt/model/steps.py Outdated

Comment thread tests/test_readers.py

AI review comments

ff07ead

deltamarnix requested changes Feb 10, 2026

View reviewed changes

Comment thread hydromt/model/example/example_model.py Outdated

Comment thread hydromt/model/model.py Outdated

Comment thread hydromt/readers.py

Comment thread hydromt/parsers.py Outdated

review comments:

001f367

rename _deep_merge to deep_merge update _steps_contain_write update example model with properties

LuukBlom mentioned this pull request Feb 10, 2026

feat: cleanup config classes #1372

Draft

11 tasks

hboisgon reviewed Feb 12, 2026

View reviewed changes

LuukBlom mentioned this pull request Feb 24, 2026

Replace raw dicts with documented Pydantic classes for api functions #1382

Open

deltamarnix marked this pull request as draft May 5, 2026 14:16

		Access the model type with `.modeltype`, the global configuration with `.globals_`,
		and the runtime steps with `.steps`.

	raise ValueError("Model type not specified in workflow or arguments")
	raise ValueError("Model type not specified in workflow YAML or arguments")

Uh oh!

Conversation

LuukBlom commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue addressed

Explanation

Cyclic import errors

General Checklist

Data/Catalog checklist

Additional Notes (optional)

Uh oh!

dalmijn commented Feb 9, 2026

Uh oh!

LuukBlom commented Feb 9, 2026

Uh oh!

sonarqubecloud Bot commented Feb 9, 2026

Uh oh!

dalmijn commented Feb 9, 2026

Uh oh!

LuukBlom commented Feb 9, 2026

Uh oh!

Uh oh!

sonarqubecloud Bot commented Feb 9, 2026

Quality Gate passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deltamarnix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hboisgon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LuukBlom commented Feb 5, 2026 •

edited

Loading