deltactl

Use-case

deltactl is just a CLI wrapped around common Deltalake operations exposed by the Rust deltalake library. The goal is to provide a single binary capable of inspecting and maintaining Deltalake tables.

• Create simple maintenance jobs for self-managed tables in object store; no need for a Spark runtime or Python environment.
• Observe and debug table states in an interrogative fashion.
• Develop and test table operations and patterns on the local filesystem.

Installation

Compile the binary from source with cargo.

• From the Git repository:

  cargo install --locked --git https://github.com/lsjoeberg/deltactl

• From local source:

  git clone https://github.com/lsjoeberg/deltactl && cd deltactl
  cargo install --path .

Object store support

The deltalake library uses the object_store crate to interact with tables stored in remote object storage services. The deltactl CLI supports Amazon S3 and Azure Blob Storage via features flags; these are not enabled by default.

To enable support for e.g. Azure storage, install with:

cargo install -F azure --locked --git https://github.com/lsjoeberg/deltactl

The object_store crate exposes a wide configuration surface via environment variables. These parameters can also be specified as --storage-options via the CLI. See the supported options for each object storage service in the object_store crate documentation:

• Azure
• S3

Note that other storage providers are also supported if they emulate e.g. AWS S3, such as Scaleway. See examples below for how to configure emulated object store services.

Example usage

Local filesystem

Deltalake tables can live entirely on a regular filesystem.

# Inspect the last five entries in the commit log.
deltactl history -n5 --oneline --uri './data/my_table'

Azure Blob Storage

Note

Requires feature flag azure.

Authenticate with Azure CLI:

deltactl details \
  -o'use_azure_cli=true' \
  --uri 'abfss://<container>@<account>.dfs.core.windows.net/<table_name>'

Amazon S3

Note

Requires feature flag s3.

Connect to a Scaleway S3 bucket:

1. Set environment variables:

   # .env
   export AWS_ACCESS_KEY_ID="***"
   export AWS_SECRET_ACCESS_KEY="***"
   export AWS_REGION="nl-ams"
   export AWS_ENDPOINT_URL="https://s3.nl-ams.scw.cloud"

2. Load storage options and interact with a table:

   source ./.env
   deltactl details --uri 's3://<bucket>/<table_name>'