Skip to content

fix: suppress pyarrow nullability warning for primary keys#4132

Open
Sam-Razavi wants to merge 1 commit into
dlt-hub:develfrom
Sam-Razavi:fix/3581-pyarrow-primary-key-nullability-warning
Open

fix: suppress pyarrow nullability warning for primary keys#4132
Sam-Razavi wants to merge 1 commit into
dlt-hub:develfrom
Sam-Razavi:fix/3581-pyarrow-primary-key-nullability-warning

Conversation

@Sam-Razavi

Copy link
Copy Markdown

Description

Fixes #3581.

When remove_nullability_adapter is used with the PyArrow backend, dlt correctly keeps reflected primary-key columns as nullable=False. PyArrow does not retain primary-key metadata and reports those fields as nullable, which causes a misleading Arrow-schema conflict warning.

This PR suppresses only that expected mismatch when:

  • the computed dlt column is a primary key
  • dlt has nullable=False
  • Arrow has nullable=True

Schema merging remains unchanged, and primary-key columns remain non-nullable.

Related Issues

Additional Context

Added a SQL + PyArrow regression test using remove_nullability_adapter that verifies the reflected id column remains a non-nullable primary key and that no Arrow-schema merge message is emitted.

Local checks completed:

  • Focused SQL + PyArrow regression test passed
  • Existing test_remove_nullability passed
  • flake8 for changed files passed
  • git diff --check passed

Targeted mypy did not complete locally within five minutes and produced no diagnostics, so CI should verify that check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Using remove_nullability_adapter with PyArrow triggers warning on tables with primary key

1 participant