feat(mssql): replace pyODBC + ODBC installation dependency with Microsoft's mssql-python#4141
Open
sdebruyn wants to merge 4 commits into
Open
feat(mssql): replace pyODBC + ODBC installation dependency with Microsoft's mssql-python#4141sdebruyn wants to merge 4 commits into
sdebruyn wants to merge 4 commits into
Conversation
sdebruyn
changed the title
sdebruyn
changed the title
sdebruyn
force-pushed
the
feat/mssql-python-driver
branch
3 times, most recently
from
5248136 to
799d1d0
Compare
…SQL destinations Add an `authentication` credential option to MsSqlCredentials so the mssql, synapse and fabric destinations can authenticate to Azure-hosted SQL with Microsoft Entra ID instead of a SQL login: - azure-identity token methods (cli, default/auto, environment, interactive, devicecode, msi/managedidentity): dlt acquires an access token and injects it via pyodbc attrs_before (SQL_COPT_SS_ACCESS_TOKEN), so they work cross-platform (including macOS, where the ODBC driver's built-in Entra ID modes are unreliable). - driver-native methods (ActiveDirectoryServicePrincipal, ActiveDirectoryPassword, ActiveDirectoryIntegrated, ActiveDirectoryInteractive) are passed as Authentication=. - empty authentication keeps the existing SQL login (backwards compatible); the default Service Principal method without a secret falls back to DefaultAzureCredential. The shared auth helpers live in the mssql destination; synapse inherits them and fabric reuses them, dropping its previous fabric-only token override. azure-identity is imported lazily (only for token methods). Adds unit tests for every method on mssql and fabric, a Docker-based integration test for the SQL-login connection path, and documentation. Closes dlt-hub#2059 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sdebruyn
force-pushed
the
feat/mssql-python-driver
branch
from
799d1d0 to
d7835e4
Compare
sdebruyn
added a commit
to sdebruyn/dlt
that referenced
this pull request
Jun 30, 2026
This branch tracks dlt and bundles the Fabric-related pull requests (dlt-hub#4140, dlt-hub#4141, dlt-hub#4142) for combined testing. Point everything else to the upstream dlt project. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sdebruyn
force-pushed
the
feat/mssql-python-driver
branch
3 times, most recently
from
a88f5d3 to
454d4a7
Compare
This was referenced Jul 1, 2026
Open
…n to two methods Redesign the `authentication` credential option to be driver-aware for the pyodbc driver used at this point in the mssql/synapse/fabric destinations: - Driver-native (passed as `Authentication=` in the DSN, poolable): the ODBC driver itself performs the sign-in. Adds `ActiveDirectoryMsi` alongside the existing `ActiveDirectoryServicePrincipal`, `ActiveDirectoryPassword`, `ActiveDirectoryIntegrated`, `ActiveDirectoryInteractive`. - azure-identity token injection (via pyodbc `attrs_before`, not poolable): narrowed to the two methods pyodbc/msodbcsql cannot perform natively, renamed to match ODBC's own naming convention: `ActiveDirectoryDefault` (DefaultAzureCredential) and `ActiveDirectoryDeviceCode` (DeviceCodeCredential). - The thin `default` alias for `ActiveDirectoryDefault` is kept; `auto` is not carried over since it never existed on `devel` and has no backwards-compatibility reason to exist here (PR1 would have been the one introducing it). - Removes the other dlt-custom lowercase names (`cli`, `environment`, `interactive`, `devicecode`, `msi`, `managedidentity`): the native `ActiveDirectory*` names now cover Interactive/DeviceCode/Msi directly, and `cli`/`environment` are covered by `ActiveDirectoryDefault`, whose `DefaultAzureCredential` chain already includes the Azure CLI and environment credentials. Updates the mssql/fabric/synapse destination docs and the offline mssql/fabric configuration unit tests to match.
Replace pyodbc with Microsoft's mssql-python driver for the mssql, synapse and fabric destinations. mssql-python bundles the SQL Server client libraries, so no separate ODBC driver installation is required, and it works natively across platforms (including arm64 macOS). - MsSqlClient (formerly PyOdbcMsSqlClient, kept as an alias) connects via mssql_python.connect with attrs_before token injection and autocommit. - datetimeoffset is returned natively as a tz-aware datetime, so the pyodbc output-converter workaround is removed. - Database exceptions are mapped from mssql-python's typed exception hierarchy. - The ODBC driver-selection machinery (pyodbc.drivers(), SUPPORTED_DRIVERS, driver validation) is removed and no DRIVER is emitted in the connection string. The `driver` credential option is kept but deprecated and ignored for backwards compatibility. - The mssql/synapse/fabric extras now depend on mssql-python instead of pyodbc. Adds a Docker-based integration test that loads data end-to-end through mssql-python, and updates the docs and lockfile.
sdebruyn
force-pushed
the
feat/mssql-python-driver
branch
from
454d4a7 to
a51b7e7
Compare
…h to mssql-python mssql-python signs in for every supported Entra ID authentication method itself, so dlt no longer needs to acquire or inject any token for named `authentication` methods: - Removed the azure-identity injection machinery for authentication methods entirely: `AZURE_IDENTITY_INJECTION_AUTHENTICATION`, `create_token_credential`, `get_token_credential`, `uses_token_authentication`, `setup_token_credential`, and the Service Principal-without-secret fallback to `DefaultAzureCredential`. `attrs_before` is never built for a named method anymore. - `DRIVER_NATIVE_AUTHENTICATION` becomes a single `SUPPORTED_AUTHENTICATION` set: every one of the seven `ActiveDirectory*` methods is validated the same way and written straight to `Authentication=` in the DSN. The `default` alias still normalizes to `ActiveDirectoryDefault` before being written, since mssql-python only recognizes the canonical name. - A Service Principal without a secret is no longer silently redirected to `DefaultAzureCredential` — it's passed through like any other method, and mssql-python is left to accept or reject it. - `build_token_attrs_before`/`to_odbc_attrs_before` are kept (always returning `None` today) rather than removed, as the injection point for a future explicit access-token feature. - Comments and docs no longer describe a native-vs-injected split or reference pyodbc; there is one supported-methods list, and mssql-python owns the sign-in for all of it. Updates the mssql/fabric/synapse destination docs and the offline mssql/fabric configuration unit tests to match.
sdebruyn
force-pushed
the
feat/mssql-python-driver
branch
from
a51b7e7 to
2d216f3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PyODBC and the ODBC Driver installation is often very bothersome for users. Microsoft now has mssql-python, which comes with a built-in ODBC driver. This PR replaces the dependency.
Because mssql-python performs the Microsoft Entra ID sign-in itself for every
Authentication=method — includingActiveDirectoryDefaultandActiveDirectoryDeviceCode— this PR also simplifies the authentication added in #4140. dlt no longer acquires or injects an access token for any named method: it just validates the method, normalizes thedefaultalias toActiveDirectoryDefault, and writesAuthentication=<method>into the DSN. mssql-python does the rest. The azure-identity token-injection code paths and their unit tests are removed accordingly.This PR builds upon #4140.
Related Issues
Additional Context
I also made dbt-fabric and did the same change there. Work is happening to get that merged upstream in Microsoft's version of dbt-fabric (which I also am the author of).