feat(encryption): add StandardEncryptionManager - #1620
Conversation
…lope encryption. Signed-off-by: hectar-glitches <hectar@uni.minerva.edu>
tanmayrauth
left a comment
There was a problem hiding this comment.
Engine works. One heads-up: this isn't the AGS1 stream format Java's StandardEncryptionManager produces, so encrypted files won't interop with Java — worth a quick call on whether that matters here. Plus two small code nits. Not
blocking. Details inline.
| func (f *standardOutputFile) Write(p []byte) (int, error) { | ||
| total := len(p) | ||
| for len(p) > 0 { | ||
| space := f.blockSize - len(f.buf) |
There was a problem hiding this comment.
Nit: NewStandardEncryptionManager doesn't validate blockSize, so a bad WithBlockSize only surfaces at write time — WithBlockSize(-1) panics slice bounds out of range [:-1] on the first Write, and WithBlockSize(0) spins forever here (len(f.buf) == f.blockSize is 0 == 0 every iteration, so p is never consumed). A blockSize > 0 guard in NewEncryptedOutputFile (which already returns an error) rejects it early.
| underlying: file, | ||
| aead: aead, | ||
| noncePrefix: meta.NoncePrefix, | ||
| blockSize: meta.BlockSize, |
There was a problem hiding this comment.
Minor robustness nit: NewDecryptedInputFile checks the version but doesn't validate the rest of the decoded metadata. A well-formed-JSON key_metadata with "block-size":0 and non-zero "plaintext-length" builds fine here, then the first Read/ReadAt panics with integer divide by zero (numBlocks at :379, curOff / int64(f.blockSize) at :423). Not reachable from this manager's own output today, but it's untrusted input on a crypto read path — better to fail closed than panic. Validate meta.BlockSize > 0, meta.PlaintextLength >= 0, and len(meta.NoncePrefix) == 4 right after the version check (:198) and return a wrapped error. The existing "malformed key metadata" test only covers invalid JSON, so add a well-formed-but-block-size-0 case.
| // closed, returning [ErrKeyIDRequired] or [ErrKeyMetadataRequired] rather | ||
| // than silently falling back to plaintext. Use [PlaintextEncryptionManager] | ||
| // for tables or files that are not encrypted. | ||
| type StandardEncryptionManager struct { |
There was a problem hiding this comment.
Heads-up on the format, not a blocker — a call worth making before this gets wired in, since the format becomes a breaking change once reads/writes depend on it.
Java's StandardEncryptionManager (same name) produces the AGS1 stream via AesGcmOutputStream: "AGS1" header + block size, 1 MiB blocks, an inline random 12-byte nonce per block, AAD = fileAadPrefix ‖ block ordinal, and Avro StandardKeyMetadata {encryption_key, aad_prefix, file_length}. This produces a different format — no header, 64 KB blocks, derived nonce, nil AAD, JSON key_metadata — so a Java reader can't read a file written here and vice versa.
Is cross-engine interop of encrypted metadata a goal? For a table encrypted by one engine and read by another it has to be, but if the intent is an iceberg-go-only format for now, this is fine as-is. One thing to note either way: there's no fileAadPrefix hook here, which is what Java uses to bind an AAD (e.g. the manifest-list case); adding that later is easier before the format is locked in. Flagging for the design discussion in #1289, not asking for changes in this PR.
Summary
Adds 'StandardEncryptionManager', a KMS-backed 'EncryptionManager' implementation for generic (format-agnostic) file envelope encryption, e.g. manifests, manifest lists, and Puffin statistics files. This follows on from #1447, which merged the 'EncryptionManager'/'KeyManagementClient' interfaces, 'PlaintextEncryptionManager', and the in-memory KMS, and #1493, which added the KMS catalog-property registry ('kms-type'). As discussed in #1289, this is the piece everything downstream ('EncryptingFileIO', Parquet native encryption, manifest/manifest-list encryption, Puffin blob encryption) needs before it can do real encryption, so it's the next slice in the proposed split.
It uses a 'KeyManagementClient' to generate/wrap a fresh AES-256 data encryption key (DEK) per file, splits each file into fixed-size plaintext blocks (default 64KB), and seals each block independently with AES-GCM using a unique nonce (per-file random prefix combined with the block index). Since blocks are sealed independently, decrypted files support random access ('Seek'/'ReadAt') without decrypting the whole file, which is required since 'icebergio.File' mandates 'io.ReaderAt'/'io.Seeker'. Everything needed to decrypt (wrapped DEK, key ID, nonce prefix, block size, plaintext length) is persisted as a small JSON blob in the opaque per-file 'key_metadata' already carried by manifest entries. It fails closed, consistent with 'PlaintextEncryptionManager': it requires a non-empty 'keyID' on write ('ErrKeyIDRequired') and non-empty key metadata on read ('ErrKeyMetadataRequired'), rather than silently no-op'ing.
Testing covers round trips (sub-block, multi-block, exact block-size multiple, empty file), random access ('ReadAt' spanning a block boundary, 'Seek' plus sequential read), tamper detection (a flipped ciphertext byte fails with 'ErrAuthenticationFailed'), and error paths (empty keyID, empty key metadata, unknown KMS key ID, malformed key metadata). Verified with 'go vet', 'go test -race -count=2', and 'golangci-lint' (0 issues).
This PR is the standalone encryption engine only, it does not yet wire 'StandardEncryptionManager' into 'EncryptingFileIO' or any table/catalog code path, that follow-up work is tracked in #1289.