Skip to content

feat(encryption): add StandardEncryptionManager - #1620

Open
hectar-glitches wants to merge 1 commit into
apache:mainfrom
hectar-glitches:feat/standard-encryption-manager
Open

feat(encryption): add StandardEncryptionManager#1620
hectar-glitches wants to merge 1 commit into
apache:mainfrom
hectar-glitches:feat/standard-encryption-manager

Conversation

@hectar-glitches

Copy link
Copy Markdown
Contributor

Summary

Adds 'StandardEncryptionManager', a KMS-backed 'EncryptionManager' implementation for generic (format-agnostic) file envelope encryption, e.g. manifests, manifest lists, and Puffin statistics files. This follows on from #1447, which merged the 'EncryptionManager'/'KeyManagementClient' interfaces, 'PlaintextEncryptionManager', and the in-memory KMS, and #1493, which added the KMS catalog-property registry ('kms-type'). As discussed in #1289, this is the piece everything downstream ('EncryptingFileIO', Parquet native encryption, manifest/manifest-list encryption, Puffin blob encryption) needs before it can do real encryption, so it's the next slice in the proposed split.

It uses a 'KeyManagementClient' to generate/wrap a fresh AES-256 data encryption key (DEK) per file, splits each file into fixed-size plaintext blocks (default 64KB), and seals each block independently with AES-GCM using a unique nonce (per-file random prefix combined with the block index). Since blocks are sealed independently, decrypted files support random access ('Seek'/'ReadAt') without decrypting the whole file, which is required since 'icebergio.File' mandates 'io.ReaderAt'/'io.Seeker'. Everything needed to decrypt (wrapped DEK, key ID, nonce prefix, block size, plaintext length) is persisted as a small JSON blob in the opaque per-file 'key_metadata' already carried by manifest entries. It fails closed, consistent with 'PlaintextEncryptionManager': it requires a non-empty 'keyID' on write ('ErrKeyIDRequired') and non-empty key metadata on read ('ErrKeyMetadataRequired'), rather than silently no-op'ing.

Testing covers round trips (sub-block, multi-block, exact block-size multiple, empty file), random access ('ReadAt' spanning a block boundary, 'Seek' plus sequential read), tamper detection (a flipped ciphertext byte fails with 'ErrAuthenticationFailed'), and error paths (empty keyID, empty key metadata, unknown KMS key ID, malformed key metadata). Verified with 'go vet', 'go test -race -count=2', and 'golangci-lint' (0 issues).

This PR is the standalone encryption engine only, it does not yet wire 'StandardEncryptionManager' into 'EncryptingFileIO' or any table/catalog code path, that follow-up work is tracked in #1289.

…lope encryption.

Signed-off-by: hectar-glitches <hectar@uni.minerva.edu>
@hectar-glitches
hectar-glitches marked this pull request as ready for review July 30, 2026 21:26

@tanmayrauth tanmayrauth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Engine works. One heads-up: this isn't the AGS1 stream format Java's StandardEncryptionManager produces, so encrypted files won't interop with Java — worth a quick call on whether that matters here. Plus two small code nits. Not
blocking. Details inline.

func (f *standardOutputFile) Write(p []byte) (int, error) {
total := len(p)
for len(p) > 0 {
space := f.blockSize - len(f.buf)

@tanmayrauth tanmayrauth Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: NewStandardEncryptionManager doesn't validate blockSize, so a bad WithBlockSize only surfaces at write time — WithBlockSize(-1) panics slice bounds out of range [:-1] on the first Write, and WithBlockSize(0) spins forever here (len(f.buf) == f.blockSize is 0 == 0 every iteration, so p is never consumed). A blockSize > 0 guard in NewEncryptedOutputFile (which already returns an error) rejects it early.

underlying: file,
aead: aead,
noncePrefix: meta.NoncePrefix,
blockSize: meta.BlockSize,

@tanmayrauth tanmayrauth Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor robustness nit: NewDecryptedInputFile checks the version but doesn't validate the rest of the decoded metadata. A well-formed-JSON key_metadata with "block-size":0 and non-zero "plaintext-length" builds fine here, then the first Read/ReadAt panics with integer divide by zero (numBlocks at :379, curOff / int64(f.blockSize) at :423). Not reachable from this manager's own output today, but it's untrusted input on a crypto read path — better to fail closed than panic. Validate meta.BlockSize > 0, meta.PlaintextLength >= 0, and len(meta.NoncePrefix) == 4 right after the version check (:198) and return a wrapped error. The existing "malformed key metadata" test only covers invalid JSON, so add a well-formed-but-block-size-0 case.

// closed, returning [ErrKeyIDRequired] or [ErrKeyMetadataRequired] rather
// than silently falling back to plaintext. Use [PlaintextEncryptionManager]
// for tables or files that are not encrypted.
type StandardEncryptionManager struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads-up on the format, not a blocker — a call worth making before this gets wired in, since the format becomes a breaking change once reads/writes depend on it.

Java's StandardEncryptionManager (same name) produces the AGS1 stream via AesGcmOutputStream: "AGS1" header + block size, 1 MiB blocks, an inline random 12-byte nonce per block, AAD = fileAadPrefix ‖ block ordinal, and Avro StandardKeyMetadata {encryption_key, aad_prefix, file_length}. This produces a different format — no header, 64 KB blocks, derived nonce, nil AAD, JSON key_metadata — so a Java reader can't read a file written here and vice versa.

Is cross-engine interop of encrypted metadata a goal? For a table encrypted by one engine and read by another it has to be, but if the intent is an iceberg-go-only format for now, this is fine as-is. One thing to note either way: there's no fileAadPrefix hook here, which is what Java uses to bind an AAD (e.g. the manifest-list case); adding that later is easier before the format is locked in. Flagging for the design discussion in #1289, not asking for changes in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants