`truncate_staging_dataset=True` fails with DuckLake destination due to concurrent `ATTACH` file lock conflict

[mathisdrn]

dlt version

1.22

Describe the problem

When loading data using dlt with the new ducklake destination and the configuration truncate_staging_dataset = True enabled (to clean up the staging tables at the end of the ingestion run), the cleanup step fails with a warning.

This is because:

1. The main load job process keeps the primary database connection open.
2. In dlt/load/load.py, the _maybe_truncate_staging_dataset method requests a new database client instance via self.get_destination_client(schema) to execute the TRUNCATE statements.
3. This second client tries to open a new connection and execute the ATTACH 'ducklake:...' statement.
4. DuckDB/DuckLake throws a Unique file handle conflict error because the database file is already attached by the active connection in the same process:

   Binder Error: Failed to attach DuckLake MetaData "__ducklake_metadata_bug_db" at path ... Unique file handle conflict: Cannot attach "__ducklake_metadata_bug_db" - the database file "/path/to/bug.ducklake" is already attached by database "__ducklake_metadata_bug_db"

5. dlt catches the exception, logs a warning, and swallows it. The pipeline finishes successfully but the staging tables are left untruncated.

Expected behavior

No response

Steps to reproduce

Save the following script as reproduce_bug.py and run it:

import os
import shutil
import logging
import dlt
from dlt.destinations.impl.ducklake.configuration import DuckLakeCredentials

# Configure logging to see the dlt warning
logging.basicConfig(level=logging.INFO)

# 1. Clean up previous runs
db_dir = os.path.abspath("bug_storage")
if os.path.exists(db_dir):
    shutil.rmtree(db_dir)
os.makedirs(db_dir, exist_ok=True)

# 2. Configure dlt programmatically
os.environ["LOAD__TRUNCATE_STAGING_DATASET"] = "True"

# 3. Create ducklake credentials
credentials = DuckLakeCredentials(
    ducklake_name="bug_db",
    catalog=f"duckdb:///{db_dir}/bug.ducklake",
    storage=f"file:///{db_dir}/bug.ducklake.files",
)

# 4. Initialize pipeline
pipeline = dlt.pipeline(
    pipeline_name="bug_pipeline",
    destination=dlt.destinations.ducklake(
        credentials=credentials,
        override_data_path=True,
    ),
    dataset_name="raw",
)

# 5. Simple source with merge disposition
@dlt.resource(name="items", write_disposition="merge", primary_key="id")
def get_items():
    yield {"id": 1, "value": "A"}
    yield {"id": 2, "value": "B"}

# 6. Run pipeline
load_info = pipeline.run(get_items())

# 7. Check if staging table contains data
import duckdb
conn = duckdb.connect()
conn.execute(f"ATTACH 'ducklake:{db_dir}/bug.ducklake' AS bug_db (DATA_PATH '{db_dir}/bug.ducklake.files', OVERRIDE_DATA_PATH true)")
cnt = conn.execute("SELECT count(*) FROM bug_db.raw_staging.items").fetchone()[0]
print(f"Staging table row count: {cnt} (Expected: 0 if truncate worked)")

---

Output / Error Log

2026-07-05 21:42:33,676|[WARNING]|dlt|load.py|_maybe_truncate_staging_dataset:860|Staging dataset truncate failed due to the following error: Connection with `client_type=DuckLakeSqlClient` to `dataset_name=raw` failed. Please check if you configured the credentials at all and provided the right credentials values. You can be also denied access or your internet connection may be down. The actual reason given is: Binder Error: Failed to attach DuckLake MetaData "__ducklake_metadata_bug_db" at path + "/Users/mathisderenne/GitHub/orca/bug/storage/bug.ducklake"Unique file handle conflict: Cannot attach "__ducklake_metadata_bug_db" - the database file "/Users/mathisderenne/GitHub/orca/bug/storage/bug.ducklake" is already attached by database "__ducklake_metadata_bug_db" However, it didn't affect the data integrity.

Staging table row count: 2 (Expected: 0 if truncate worked)

Operating system

macOS

Runtime environment

Local

Python version

3.13

dlt data source

No response

dlt destination

DuckDB using DuckLake

Other deployment details

No response

Additional information

Suggested Solutions / Workarounds

1. Reuse Connection: If _maybe_truncate_staging_dataset could reuse the active connection in the existing job_client (which is passed to the method as an argument!) instead of instantiating a new client via self.get_destination_client(schema), this conflict would be bypassed.
2. Release / Detach prior to Truncation: Ensure the main loading client releases or detaches the catalog database before the staging truncation client is instantiated.