Skip to content

README.md

Latest commit

 

History

History
68 lines (52 loc) · 1.54 KB

README.md

File metadata and controls

68 lines (52 loc) · 1.54 KB
Raw

Native HDFS Python bindings based on hdfs-native Rust package.

Installation

pip install hdfs-native

Example

from hdfs_native import Client
client = Client("hdfs://localhost:9000")

status = client.get_file_info("/file.txt")

CLI

There is a built-in CLI hdfsn that implements most of the behavior of hdfs dfs but with a more bash-like syntax. The easiest way to use the CLI is with UV.

Install CLI with UV

uv tool install hdfs-ntaive

Auto-complete support

The CLI supports auto-complete for HDFS paths using argcomplete. There are two ways to enable this support

To permanently enable support for all Python modules using argcomplete:

uv tool install argcomplete
activate-global-python-argcomplete

To enable support just for hdfsn in your active shell:

uv tool install argcomplete
eval "$(register-python-argcomplete hdfsn)"

Kerberos support

Kerberos (SASL GSSAPI) is supported through a runtime dynamic link to libgssapi_krb5. This must be installed separately, but is likely already installed on your system. If not you can install it by:

Debian-based systems

apt-get install libgssapi-krb5-2

RHEL-based systems

yum install krb5-libs

MacOS

brew install krb5

Running tests

The same requirements apply as the Rust tests, requiring Java, Maven, Hadoop, and Kerberos tools to be on your path. Then you can:

python3 -m venv .venv
source .venv/bin/activate
pip3 install maturin
maturin develop -E devel
pytest