Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions INSTALL-AND-REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,15 @@ filters. Replace `pc.field("age") >= 18` with `table.column("age") >= 18`, and r
`[a, b]` with `a & b`. Code using PyArrow for Arrow-side filtering can continue to use
`pyarrow.compute.field()` outside the Tower table API.

Arrow schemas passed to `create()` or `create_if_not_exists()` are validated directly by
PyIceberg, which assigns field IDs and preserves nested nullability and `b"doc"` field
metadata. Timestamp units from seconds through microseconds, UTC-zoned microsecond
timestamps, `time64[us]`, `date32`, and Decimal128 values up to precision 38 are supported.
Nanosecond timestamps are rejected by default instead of being silently downcast, as are
`time32`, `time64[ns]`, `date64`, Float16, Decimal256, and non-UTC zoned timestamps. Convert
those fields explicitly before creating the table when the loss is acceptable. PyIceberg's
native validation exceptions propagate unchanged.

### dbt Core support

```bash
Expand Down
15 changes: 9 additions & 6 deletions src/tower/_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
)
from .exceptions import PyArrowFilterMigrationError
from .tower_api_client.models import CatalogCredentials
from .utils.pyarrow import convert_pyarrow_schema
from .utils.tables import (
make_table_name,
namespace_or_default,
Expand Down Expand Up @@ -650,7 +649,9 @@ def create(self, schema: pa.Schema) -> Table:

Args:
schema (pa.Schema): The PyArrow schema defining the structure of the table.
This will be converted to an Iceberg schema internally.
PyIceberg validates it and assigns Iceberg field IDs. Lossy or
unsupported types, including nanosecond timestamps by default, are
rejected.

Returns:
Table: A new Table instance wrapping the created Iceberg table.
Expand Down Expand Up @@ -685,7 +686,7 @@ def create(self, schema: pa.Schema) -> Table:
# along the way.
table = catalog.create_table(
identifier=table_name,
schema=convert_pyarrow_schema(schema),
schema=schema,
)

return Table(
Expand All @@ -712,8 +713,10 @@ def create_if_not_exists(self, schema: pa.Schema) -> Table:

Args:
schema (pa.Schema): The PyArrow schema defining the structure of the table.
This will be converted to an Iceberg schema internally. Note that this
schema is only used if the table needs to be created.
PyIceberg validates it and assigns Iceberg field IDs. Lossy or
unsupported types, including nanosecond timestamps by default, are
rejected. This schema is only used if the
table needs to be created.

Returns:
Table: A Table instance wrapping either the newly created or existing Iceberg table.
Expand Down Expand Up @@ -747,7 +750,7 @@ def create_if_not_exists(self, schema: pa.Schema) -> Table:
# exists.
table = catalog.create_table_if_not_exists(
identifier=table_name,
schema=convert_pyarrow_schema(schema),
schema=schema,
)

return Table(
Expand Down
178 changes: 0 additions & 178 deletions src/tower/utils/pyarrow.py

This file was deleted.

Loading
Loading