Skip to content

bug: [pg_replication] requirements.txt and pg_replication dependency-group are missing sql_database / SQLAlchemy #685

Description

@duc-accelerate-data

dlt version: 1.26.0
Source name: pg_replication

Describe the problem

After running dlt init pg_replication <destination> and installing only what the generated requirements.txt declares, the starter pipeline fails at import time with MissingDependencyException for dlt.sources.sql_database.

sources/pg_replication/helpers.py imports dlt.sources.sql_database (line 43), which is guarded by dlt's sql_database extras (SQLAlchemy + dialects). However the connector's dependency manifests are incomplete:

  • sources/pg_replication/requirements.txt declares only dlt>=1.0.0 and psycopg2-binary>=2.9.9.
  • [dependency-groups].pg_replication in pyproject.toml declares only psycopg2-binary>=2.9.9.

Neither pulls SQLAlchemy in. The starter pipeline can never get past from pg_replication import replication_resource until the user manually runs pip install "dlt[sql_database]" — which the docs do not mention.

The bug is invisible during normal contributor work on this repo because [tool.uv].default-groups includes both sql_database and pg_replication, so a plain uv sync drags SQLAlchemy in via the sibling group. As soon as anything installs the pg_replication group on its own — dlt init, downstream tools that consume requirements.txt, or CI matrices that pick one group at a time — the connector breaks.

Expected behavior

After running the documented onboarding flow (dlt init pg_replication <destination> + pip install -r requirements.txt), the user should be able to run the generated pipeline script without a MissingDependencyException from the verified source itself. The only remaining failure should be the expected "credentials are placeholder values" error, which prompts the user to fill in .dlt/secrets.toml.

Steps to reproduce

Clean Python 3.12 environment, following dlt's documented onboarding:

mkdir /tmp/dlt-pg-repro && cd /tmp/dlt-pg-repro
uv venv .venv --python python3.12
uv pip install --python .venv/bin/python "dlt>=1.0.0"

# documented onboarding
.venv/bin/dlt init pg_replication duckdb

# install per dlt init's instructions
uv pip install --python .venv/bin/python -r requirements.txt

# run the starter pipeline dlt init generated
.venv/bin/python pg_replication_pipeline.py

Result:

File ".../pg_replication/helpers.py", line 43, in <module>
  from dlt.sources.sql_database import (
File ".../dlt/common/libs/sql_alchemy.py", line 15, in <module>
  raise MissingDependencyException(
dlt.common.exceptions.MissingDependencyException:
You must install additional dependencies to run `dlt sql_database helpers`.
If you use pip you may do the following:
    pip install "dlt[sql_database]"

Same failure reproduces via the contributor uv path:

uv sync --python python3.12 --group dltpure --group pg_replication \
        --no-default-groups --inexact
PYTHONPATH=$PWD .venv/bin/python -c "import sources.pg_replication"
# → same MissingDependencyException

dlt init does rewrite dlt>=1.0.0 to dlt[<destination>]>=1.0.0 in the generated requirements.txt (so the destination's extras land), but it merges destination extras only — the source's extras still need to be declared in the source's own requirements.txt.

How you are using the source?

I'm considering using this source in my work, but bug is preventing this.

Operating system

macOS (also reproducible on Linux in CI)

Runtime environment

Local

Python version

3.12.7

dlt destination

duckdb (the same import-time failure occurs regardless of destination — destination only changes which destination extras dlt init writes into requirements.txt)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Planned

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions