Describe the problem
When multiple workers initialize the same schema concurrently against Databricks, the load fails with DELTA_METADATA_CHANGED on the _dlt_version table.
Every time a worker initializes the schema, dlt's _execute_schema_update_sql runs all the DDL for that schema — including COMMENT ON TABLE _dlt_version IS 'Created by DLT. Tracks schema updates'. In Delta, COMMENT ON TABLE is implemented as SET TBLPROPERTIES, which is a metadata write transaction.
The race: all 5 workers query get_stored_schema_by_hash simultaneously, all find the hash not stored yet, all proceed to run the full DDL including the COMMENT ON TABLE. Delta's optimistic concurrency detects two concurrent metadata writes on the same table and rejects all but the first with DELTA_METADATA_CHANGED.
With task_max_retries: 1, the retry finds the schema hash already stored (written by the first worker that succeeded), get_stored_schema_by_hash returns non-null, and the worker skips DDL entirely — no COMMENT ON TABLE, no conflict. So the failure is only avoided by a retry that happens to land after the winning worker committed.
Expected behavior
Concurrent workers initializing the same schema should not fail. dlt should not require task_max_retries to be set just to survive a metadata-write race on its own internal _dlt_version (and other dlt) tables.
Related
This is a distinct race from the CREATE TABLE / FIELDS_ALREADY_EXISTS one in #4135 / #4138: here the conflict is on a SET TBLPROPERTIES metadata write under Delta optimistic concurrency, surfacing as DELTA_METADATA_CHANGED.
Describe the problem
When multiple workers initialize the same schema concurrently against Databricks, the load fails with
DELTA_METADATA_CHANGEDon the_dlt_versiontable.Every time a worker initializes the schema, dlt's
_execute_schema_update_sqlruns all the DDL for that schema — includingCOMMENT ON TABLE _dlt_version IS 'Created by DLT. Tracks schema updates'. In Delta,COMMENT ON TABLEis implemented asSET TBLPROPERTIES, which is a metadata write transaction.The race: all 5 workers query
get_stored_schema_by_hashsimultaneously, all find the hash not stored yet, all proceed to run the full DDL including theCOMMENT ON TABLE. Delta's optimistic concurrency detects two concurrent metadata writes on the same table and rejects all but the first withDELTA_METADATA_CHANGED.With
task_max_retries: 1, the retry finds the schema hash already stored (written by the first worker that succeeded),get_stored_schema_by_hashreturns non-null, and the worker skips DDL entirely — noCOMMENT ON TABLE, no conflict. So the failure is only avoided by a retry that happens to land after the winning worker committed.Expected behavior
Concurrent workers initializing the same schema should not fail. dlt should not require
task_max_retriesto be set just to survive a metadata-write race on its own internal_dlt_version(and other dlt) tables.Related
This is a distinct race from the
CREATE TABLE/FIELDS_ALREADY_EXISTSone in #4135 / #4138: here the conflict is on aSET TBLPROPERTIESmetadata write under Delta optimistic concurrency, surfacing asDELTA_METADATA_CHANGED.