Encryption Support

[rambleraptor]

Feature Request / Improvement

iceberg-go will read the encryption-keys metadata, but it doesn't support encrypting / decrypting data using that metadata.

I think the biggest part of this will be figuring out the proper interfaces. There will many different implementations of these interfaces over time.

I'd love to hear everyone's idea on the basic design here:

Interfaces

• Encryption interfaces + Plaintext manager — EncryptionManager, EncryptedInputFile/EncryptedOutputFile, EncryptionKeyMetadata, NativeEncryption*, and the no-op PlaintextEncryptionManager.
• KMS client interface — KeyManagementClient (wrapKey/unwrapKey/supportsKeyGeneration/generateKey), plus a test/in-memory impl and encryption.kms-type/encryption.kms-impl catalog-property wiring.
• StandardEncryptionManager — envelope KEK/DEK logic, local DEK generation, key cache, rotation/timestamp tracking, encryption.key-id + encryption.data-key-length table properties. Consumes the existing encryption-keys metadata array.

Integration

• EncryptingFileIO wrapper — slot encryption into the io.IO layer: decrypt on Open of Data/Manifest/ManifestList/Statistics files based on key_metadata, encrypt on write.
• Parquet native encryption for data/delete files — wire FileEncryptionProperties/FileDecryptionProperties from NativeEncryptionOutputFile into table/internal/parquet_files.go (write) and the scan read path.
• Manifest + manifest-list encryption — whole-file AES-GCM using the existing key_metadata fields, including the manifest-list double-encryption (KEK + timestamp AAD) on the write/read planners.
• Puffin blob / statistics-file encryption — encrypt DV and stats blobs; populate StatisticsFile.KeyMetadata.

Testing

• Format-version compatibility + end-to-end + cross-client tests — reject encryption props on v1, gate behavior, and round-trip a Java-produced encrypted table end-to-end (the equivalent of the DV cross-client compliance PR test(table): add cross-client DV format compliance tests #1131).

[laskoviymishka]

That's a cool feature! We totally should do this!
I would like to dive a bit deeper here with a design (how this would be setted from user perspective).

[hectar-glitches]

@laskoviymishka I did some scavenging around the repo. I saw that encryption.go has the EncryptionKey struct (v3-gated), manifest.go has KeyMetadata() []byte on both ManifestFile and DataFile, statistics.go has KeyMetadata *string on StatisticsFile, and the Avro schemas already wire key_metadata fields (IDs 519 and 131).
The core design question is how key_metadata is delivered to the IO layer. We can extend with an optional interface mirroring how ReadFileIO/WriteFileIO/ListableIO extend today, with an EncryptedIO that includes OpenEncrypted and CreateEncrypted methods that accept keyMetadata []byte. Existing io.IO users are unaffected; the scan and write paths already have DataFile.KeyMetadata() and pass it through when the underlying IO implements EncryptedIO.
Parquet native encryption is already supported. arrow-go/v18 already exposes FileEncryptionProperties/FileDecryptionProperties publicly. We can then just wire them up. I don't think pqarrow changes are needed.

[rambleraptor]

I should've sketched out my plan in a bit more detail! I'm OOO until Tuesday and I'll respond then

[hectar-glitches]

@rambleraptor any new sketches

[rambleraptor]

Here's what I've got in mind. Let me know if this is the level of detail that you're thinking about.

• Table object will store a KeyManagementClient. We can use WithKMSClient() to set it. There would be a default, but vendors could have their own that hook up to their KMS solutions.
• Tables currently have a FS method that return an iceberg.IO for reading / writing. My new idea here is to create an interface called EncryptingFileIO that has access to both the underlying IO and an EncryptionManager. The KeyManagementClient would be responsible for instantiating this.
• EncryptingFileIO has the same methods as standard IO, but will do encryption in the background.
• The EncryptionManager would be responsible for holding the encryption keys + doing encryption/decryption.
• We'd have a PlainTextEncryptionManager that just no-ops everything and does the same thing that normally happens.

How does that sound?

[hectar-glitches]

Agree with the overall shape. How does EncryptingFileIO receive key_metadata when Open(name) is called? The scan planner has DataFile.KeyMetadata() []byte in hand at open time, and the manifest reader has ManifestFile.KeyMetadata() []byte, but i dont think the file path alone is enough to pick the right DEK.

[rambleraptor]

Yeah, that's a great point. I think we've got two choices:

• Have Open() called as part of the EncryptingFileIO (or another encrypting class)
• Open() will receive encrypted bytes and then always pass it off elsewhere

Is that a decision you're happy to wait until we reach code review to figure out?

[hectar-glitches]

Yup, happy to wait

[hectar-glitches]

happy to start on the interfaces while the key_metadata routing question is deferred. My plan is to:

Create a new encryption package with EncryptionManager, KeyManagementClient, and PlaintextEncryptionManager no-op
In-memory KeyManagementClient for tests
EncryptingFileIO wrapper once interfaces are merged

Does that order work?

[hectar-glitches]

#1447

[hectar-glitches]

@rambleraptor following up now that the first slice (EncryptionManager/KeyManagementClient interfaces + PlaintextEncryptionManager + in-memory KMS) is merged in #1447

Everything downstream needs "StandardEncryptionManager" to exist before it can do real encryption

Proposed split:

"StandardEncryptionManager" + KMS catalog-property wiring ("encryption.kms-type"/"encryption.kms-impl", "encryption.key-id"/"encryption.data-key-length") to envelope KEK/DEK logic, key cache, rotation/timestamp AAD. This gates most of the rest, so whoever picks it up first unblocks everyone else.

"EncryptingFileIO" wrapper can also be built and tested against the existing interfaces (e.g. "PlaintextEncryptionManager" or a small fake) without waiting on "StandardEncryptionManager", then swapped over to "StandardEncryptionManager" once it lands.

Once "StandardEncryptionManager" lands, we can split further (mostly independent files, can run in parallel):
Parquet native encryption ("table/internal/parquet_files.go" + read path)
Manifest + manifest-list encryption ("manifest.go" + list planners)
Puffin blob/statistics-file encryption ("puffin/")

Format-version gating, end-to-end, and cross-client compatibility tests.

Happy to take a piece of this!