Skip to content

feat(normalize): fail early on unsupported data type for file format#4172

Open
AshwabhB wants to merge 2 commits into
dlt-hub:develfrom
AshwabhB:feat/815-verify-data-types-in-normalize
Open

feat(normalize): fail early on unsupported data type for file format#4172
AshwabhB wants to merge 2 commits into
dlt-hub:develfrom
AshwabhB:feat/815-verify-data-types-in-normalize

Conversation

@AshwabhB

@AshwabhB AshwabhB commented Jul 5, 2026

Copy link
Copy Markdown

Description

Runs the destination data-type/file-format compatibility check (verify_supported_data_types) during normalize, so an unsupported combination fails at the normalize step instead of surfacing later at load time. Reuses the existing load-time logic unchanged, so destinations that already declare exceptions (bigquery, redshift, athena, databricks, dremio, synapse, sqlalchemy) get the earlier check for free. model and reference jobs are skipped since they carry no typed data file.

Adds a destination_type field to NormalizeConfiguration so the error message can name the destination.

This covers task 3 of the issue. The Snowflake use_decfloat/parquet exception and remaining tasks will follow separately.

Related Issues

Additional Context

Test: tests/normalize/test_normalize.py::test_normalize_fails_on_unsupported_data_type — bigquery cannot load a json column from parquet, and this now raises UnsupportedDataType during normalize. Ran make lint (mypy/ruff/flake8/bandit/docstrings) and the affected test-common paths locally; all green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant