Skip to content

Commit 4532da8

Browse files
authored
feat(encryption): support AES-256 keys for reading Parquet data files (#2878)
## Which issue does this PR close? - Closes #. Depends on the arrow-rs 58.4.0 release tracked in apache/arrow-rs#10349. ## What changes are included in this PR? - Bump `arrow`/`parquet` deps from `58` to `58.4`. arrow-rs 58.4.0 adds AES-256 for Parquet modular encryption (58.3.0 was AES-128 only), landed via apache/arrow-rs#9203. - Add an AES-256 encrypted-Parquet read test. Refactored the existing 128-bit test into a shared `assert_encrypted_parquet_roundtrip(key)` helper with `_aes_128` and `_aes_256` callers. Note: 192-bit is not supported for data files. arrow-rs uses `ring`, which has no AES-192-GCM. Can revisit later. ## Are these changes tested? Yes. Unit test `test_read_encrypted_parquet_aes_256` writes and reads back a 256-bit encrypted Parquet file. It fails on arrow-rs 58.3.0 (`ring` rejects the 32-byte key) and passes on 58.4.0.
1 parent 6960920 commit 4532da8

3 files changed

Lines changed: 61 additions & 50 deletions

File tree

Cargo.lock

Lines changed: 37 additions & 37 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -49,15 +49,15 @@ aes-gcm = "0.10"
4949
anyhow = "1.0.72"
5050
apache-avro = { version = "0.21", features = ["snappy", "zstandard"] }
5151
array-init = "2"
52-
arrow = "58"
53-
arrow-arith = "58"
54-
arrow-array = "58"
55-
arrow-buffer = "58"
56-
arrow-cast = "58"
57-
arrow-ord = "58"
58-
arrow-schema = "58"
59-
arrow-select = "58"
60-
arrow-string = "58"
52+
arrow = "58.4"
53+
arrow-arith = "58.4"
54+
arrow-array = "58.4"
55+
arrow-buffer = "58.4"
56+
arrow-cast = "58.4"
57+
arrow-ord = "58.4"
58+
arrow-schema = "58.4"
59+
arrow-select = "58.4"
60+
arrow-string = "58.4"
6161
as-any = "0.3.2"
6262
async-trait = "0.1.89"
6363
aws-config = "1.8.7"
@@ -121,7 +121,7 @@ murmur3 = "0.5.2"
121121
once_cell = "1.20"
122122
opendal = "0.57"
123123
ordered-float = "4"
124-
parquet = "58"
124+
parquet = "58.4"
125125
pilota = "0.11.10"
126126
pretty_assertions = "1.4"
127127
pyo3 = "0.28"

crates/iceberg/src/arrow/reader/pipeline.rs

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -737,9 +737,10 @@ mod tests {
737737
}
738738
}
739739

740-
#[tokio::test]
741-
async fn test_read_encrypted_parquet() {
742-
let encryption_key = b"0123456789abcdef";
740+
/// Writes a single-column Parquet file encrypted with `encryption_key`, then reads it
741+
/// back through `ArrowReader` and asserts the round-tripped values. The key length
742+
/// selects the AES-GCM variant in arrow-rs (16 -> AES-128, 32 -> AES-256).
743+
async fn assert_encrypted_parquet_roundtrip(encryption_key: &[u8]) {
743744
let aad_prefix = b"aad_prefix";
744745

745746
let schema = Arc::new(
@@ -807,6 +808,16 @@ mod tests {
807808
assert_eq!(ids.values(), &[10, 20, 30]);
808809
}
809810

811+
#[tokio::test]
812+
async fn test_read_encrypted_parquet_aes_128() {
813+
assert_encrypted_parquet_roundtrip(b"0123456789abcdef").await;
814+
}
815+
816+
#[tokio::test]
817+
async fn test_read_encrypted_parquet_aes_256() {
818+
assert_encrypted_parquet_roundtrip(b"0123456789abcdef0123456789abcdef").await;
819+
}
820+
810821
#[tokio::test]
811822
async fn test_read_encrypted_parquet_without_key_metadata_fails() {
812823
let encryption_key = b"0123456789abcdef";

0 commit comments

Comments
 (0)