Skip to content

fix: make Snowflake schema creation idempotent#4117

Open
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:codex/snowflake-create-schema-if-not-exists
Open

fix: make Snowflake schema creation idempotent#4117
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:codex/snowflake-create-schema-if-not-exists

Conversation

@fengjikui

Copy link
Copy Markdown

Description

Snowflake dataset creation now uses CREATE SCHEMA IF NOT EXISTS instead of plain CREATE SCHEMA. This removes the race where two concurrent workers both pass the existence check and one fails because the other created the schema first.

The change is scoped to SnowflakeSqlClient; the generic SQL client behavior remains unchanged for other destinations.

Related Issues

Additional Context

Validation run locally:

env ACTIVE_DESTINATIONS='["snowflake"]' uv run --extra snowflake pytest tests/load/snowflake/test_snowflake_table_builder.py::test_create_dataset_uses_idempotent_schema_sql -q
# 1 passed in 0.16s

uv run ruff check dlt/destinations/impl/snowflake/sql_client.py tests/load/snowflake/test_snowflake_table_builder.py
# All checks passed!

uv run ruff format --check dlt/destinations/impl/snowflake/sql_client.py tests/load/snowflake/test_snowflake_table_builder.py
# 2 files already formatted

python3 -m py_compile dlt/destinations/impl/snowflake/sql_client.py tests/load/snowflake/test_snowflake_table_builder.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(sql_client) SnowflakeSqlClient.create_dataset() missing IF NOT EXISTS causes race condition with concurrent pipeline runs

1 participant