Support relative storage paths in `ducklake` destination

[mathisdrn]

Feature description

Feature Request: Support relative storage paths in ducklake destination for database portability and remote querying (frozen DuckLake)

Problem Description

Currently, when initializing or writing to a dlt.destinations.ducklake destination with a local filesystem storage path, dlt automatically normalizes and resolves the storage path to a local absolute URL.

This absolute path gets written directly into the catalog database's ducklake_metadata table under the data_path key.

As a result, the generated DuckLake catalog is not portable:

1. Broken Local Portability: If another developer pulls the repository or if the directory is relocated, the database queries fail because the metadata is pointing to the absolute path of the original creator (e.g., /Users/original_user/project/storage/...).
2. Broken Remote Querying (Frozen DuckLake): If the catalog is hosted remotely (e.g., in a public GitHub repository or HTTP/S3 bucket) to be queried serverless-style using DuckDB/DuckLake extension via ATTACH 'ducklake:https://raw.githubusercontent.com/.../orca.ducklake', the queries fail because the remote catalog still tries to locate files on the absolute path of the machine that compiled it.

---

Root Cause

Under the hood, dlt.destinations.impl.ducklake.configuration relies on FilesystemConfigurationWithLocalFiles to configure the storage bucket URL.

During resolving, the FilesystemConfiguration (and FilesystemConfigurationWithLocalFiles) class enforces absolute paths:

1. In normalize_bucket_url():

   def normalize_bucket_url(self) -> None:
       if self.is_local_path(self.bucket_url):
           self.bucket_url = self.make_file_url(self.bucket_url)

2. And make_file_url() forces relative local paths to resolve to absolute paths via the running environment's CWD:

   @staticmethod
   def make_file_url(local_path: str) -> str:
       if not os.path.isabs(local_path):
           local_path = os.path.join(active().local_dir, local_path)
       return make_fsspec_url("file", local_path, None)

Because of this, even if a relative path like storage/orca.ducklake.files is supplied to DuckLakeCredentials, it is written to the database catalog metadata as /Users/.../storage/orca.ducklake.files/.

---

Proposed Solution

Provide a way to preserve relative storage paths during DuckLake catalog initialization so they are stored as-is in the metadata (e.g., orca.ducklake.files/).

Since DuckLake natively supports relative paths (resolving them relative to the location of the .ducklake file itself when attached in DuckDB/DuckLake), preserving the relative path in dlt would make the output catalog fully portable out-of-the-box for both local developers and remote query clients.

Potential Approaches:

• Introduce an option in DuckLakeCredentials to skip path normalization or keep paths relative.
• Implement a relative_storage boolean configuration that tells dlt to store a relative folder path in ducklake_metadata while executing locally with resolved paths.

---

Temporary Workaround

I am currently working around this by executing a post-processing script right after pipeline completion to manually execute SQL on the metadata table:

UPDATE ducklake_metadata SET value = 'orca.ducklake.files/' WHERE key = 'data_path';

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

No response

Proposed solution

No response

Related issues

No response