Skip to content

feat(databricks): create tables with IF NOT EXISTS and reconcile columns for concurrent loads#4138

Open
francescomucio wants to merge 1 commit into
dlt-hub:develfrom
francescomucio:feat/4135-databricks-create-table-if-not-exists
Open

feat(databricks): create tables with IF NOT EXISTS and reconcile columns for concurrent loads#4138
francescomucio wants to merge 1 commit into
dlt-hub:develfrom
francescomucio:feat/4135-databricks-create-table-if-not-exists

Conversation

@francescomucio

Copy link
Copy Markdown
Contributor

Closes #4135

Problem

When several dlt runs load into the same Databricks table concurrently and the table doesn't exist yet, every run reads storage, sees the table missing (generate_alter = False), and emits a plain CREATE TABLE. The first commit wins; the rest fail with "table already exists". Hiding behind that crash is a second bug: a run carrying an extra column would see its create become a no-op and then fail when its load writes a column that was never added.

Change (Databricks only)

  • CREATE TABLE IF NOT EXISTS for all tables. _make_create_table is overridden to drop the "dlt system tables only" restriction (still gated by the existing supports_create_table_if_not_exists capability). Covers both the base CREATE path and the custom cluster/partition/tblproperties path.
  • Column reconciliation after the create. Databricks has no ADD COLUMN IF NOT EXISTS, so _execute_schema_update_sql re-reads the destination after the (possibly no-op) create and issues ALTER TABLE … ADD COLUMN for any columns still missing, tolerating the FIELDS_ALREADY_EXISTS error raised when a concurrent run adds the same column first.

Net effect: identical-schema parallel loads stop racing, and divergent-schema loads converge.

Notes

  • The reconciliation read runs only inside update_stored_schema (i.e. when the schema hash actually changes), not on every load.
  • Reconciled columns may not appear in the per-load-package applied_update diff; this is informational only — the data write maps columns by name against the full stored schema, so the load is unaffected.

Tests

Added to tests/load/databricks/test_databricks_table_builder.py (no live connection needed):

  • CREATE TABLE IF NOT EXISTS emitted on both create paths.
  • Reconciliation adds a column a concurrent create missed.
  • FIELDS_ALREADY_EXISTS is tolerated; other errors re-raise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(databricks): create tables with IF NOT EXISTS and reconcile columns under concurrent loads

1 participant