feat(storage): replace Azure Blob with on-prem MinIO/S3 (cutover)#4197
Draft
TaprootFreak wants to merge 7 commits into
Draft
feat(storage): replace Azure Blob with on-prem MinIO/S3 (cutover)#4197TaprootFreak wants to merge 7 commits into
TaprootFreak wants to merge 7 commits into
Conversation
Replace @azure/storage-blob with a provider-agnostic StorageService backed by S3StorageService (MinIO via @aws-sdk/client-s3), a MockStorageService for local dev, and a createStorageService() factory. blobUrl/blobName live on the base class so the URL shape stays reversible and host-independent: blobName() decodes both the legacy Azure-host and the new MinIO-host URLs to the same container-relative key, which keeps DB-persisted URLs valid across the migration. Add the GeBüV anchoring pipeline (Merkle + OpenTimestamps + WORM bucket provisioning) under storage/anchoring with the AddArchiveAnchoring migration, and record content hashes of KYC documents and EP2 settlement reports on upload as a best-effort side-booking (failures are logged, never roll back the upload). Swap the config azure.storage block for an s3 block, delete azure-storage.service.ts, and migrate all consumers (kyc-document, support-document, fiat-output-job) plus the BlobContent/Blob importers grown on develop to the new module. Restore KycDocumentService.toHostStableUrl on the storage-backed service via the shared StorageService.encodePath so served URLs stay byte-identical apart from the host. Also make the GwG compliance download (Handelsregisterauszug, Vollmacht) select KYC documents by their host-independent container-relative key instead of full-URL string equality, so pre-cutover documents whose stored Azure-host URL no longer equals the live MinIO-host URL are no longer silently dropped from the compliance ZIP.
…cutover Regression guard for the silent-data-loss blocker: when kyc_step.result holds a legacy Azure-host blob URL and the live KycFileBlob.url is regenerated from the MinIO backend, the fileDownloadConfig entries 11 (Handelsregisterauszug) and 12 (Vollmacht) must still select the document via the host-independent object key. Covers COMMERCIAL_REGISTER, LEGAL_ENTITY and AUTHORITY steps, percent-encoded file names, and the fail-closed behaviour when a URL is missing.
Two fail-closed hardenings for the Azure->MinIO WORM cutover, so a misconfiguration surfaces as a loud error instead of silently under-protecting GeBüV-retention-relevant compliance records. - provision-bucket: reject RETENTION_YEARS below the GeBüV 10-year floor. COMPLIANCE-mode Object Lock retention is extend-only and irreversible, so a value of 1..9 would permanently under-retain WORM objects with no way to correct it. Keep the 11y default; a value >= 10 is required. - EP2 settlement sink: add StorageService.uploadWormBlob and route the per-merchant EP2 report write through it. The S3 implementation verifies the target bucket has Object Lock enabled (GetObjectLockConfiguration) before the first PUT and throws otherwise, since Object Lock cannot be retro-fitted onto an existing bucket. Verified once per container to keep the probe off the per-PUT hot path; the kyc path is unchanged. Add proving tests: the retention floor rejects 9 and accepts 10/11, and a WORM-sink write fails closed (no PUT) into a non-locked / unverifiable bucket while succeeding into a locked one.
…ersisted proofs, TOCTOU-safe recordHash - Merkle tree rebuilt per RFC 6962: 0x00 leaf / 0x01 node domain separation and the largest-power-of-two split, replacing the Bitcoin-style last-node duplication that made [a,b,c] and [a,b,c,c] hash to the same root (CVE-2012-2459). No production anchor exists yet (archive_batch is empty in every environment), so the format change is free now and would be irreversible after the first anchor. - verifyDocument derives the proof leaf from the supplied bytes, not the stored hash, so a tampered document no longer reports proofValid=true. Added a non-optional `verified` field (found && hashMatches && anchored && proofValid); `pending` stays separate. - Inclusion proofs are persisted (archive_file.merkleProof) and used verbatim; verifyDocument fails closed when a batch is set but the proof is missing, instead of live-reconstructing from sibling rows that may be pruned over the 11-year retention window. - recordHash uses a conditional IsNull() update and reloads fail-closed on race loss, closing the TOCTOU window against the nightly anchorPending run. - archive_file.bucket/name are `text`, not varchar(256): storage keys embed an unbounded, user-controlled file name; a long name made the recordHash insert throw and silently drop the document from anchoring. - merkle proof codec validates each sibling as a 32-byte hex digest before decoding.
…econcile safety net - KYC uploads go through uploadWormBlob (was uploadBlob), so a kyc bucket that does not enforce Object Lock is rejected fail-closed instead of silently accepting mutable compliance records. copyBlobs asserts Object Lock before copying, since an account-merge copy into the kyc bucket is a fresh WORM write. copyBlobs returns the destination keys so merge-copied documents get anchored the same way uploads are. - reconcileHashes cron backfills anchor gaps left by a failed best-effort recordHash. It lists keys only (no per-object HeadObject fan-out over 1.3M objects) and caps the backfill per run, logging loudly when capped, to avoid OOM and the cron timeout. Bucket enumeration is per-row fault-tolerant: one unparsable paymentLinksConfig no longer aborts the whole run, and the kyc bucket is always reconciled regardless of the EP2 lookup. - redundant per-method try/catch removed from the DfxCron jobs; the framework already logs.
…nt compliance download - kycBlobKey throws on a blob URL without the container marker instead of silently returning the full URL, which would reinstate the exact host-equality bug the cutover filter fixes. To keep that fail-loud without taking down the whole GwG compliance export, the per-target filter in downloadUserData is wrapped: a single dirty step result is recorded in error_log.csv and skipped, the rest of the ZIP is still produced. - provision-bucket uses dedicated S3_ADMIN_* credentials (no fallback to the app keys, which cannot create buckets or set Object Lock), distinguishes 403 from 404 instead of guessing, and reads the Object Lock config back after applying it, failing closed on any mismatch. - .env.example documents the seven new S3_* / S3_ADMIN_* vars and drops the dead AZURE_STORAGE_CONNECTION_STRING; the still-used AZURE_* client vars stay.
TaprootFreak
force-pushed
the
feat/storage-minio-cutover
branch
from
b1cac3c to
124fae9
Compare
Collaborator
Author
|
Review: 2 passes to zero findings.
CI green (8/8, incl. Kept as draft on purpose — per the PR body, releasing this is the cutover; it becomes ready-to-merge only in the maintenance window, not before. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace Azure Blob with on-prem MinIO/S3 for document storage
Supersedes the draft sketch #3921, rebuilt on current
develop(the sketch had drifted ~150 commits behind and no longer compiled against develop).Introduces a provider-agnostic
StorageServiceabstraction with anS3StorageService(MinIO via@aws-sdk/client-s3), aMockStorageServicefor local dev, and acreateStorageService()factory. All KYC / support / EP2 document paths move off Azure Blob. Also brings in the GeBüV compliance-archive pipeline (WORM bucket provisioning, per-object hashing, daily Merkle root + OpenTimestamps/Bitcoin anchoring, verification).Per the operational runbook, releasing this is the cutover — merging before the document data is copied and verified would point production at an empty store. Keep as draft until the cutover window.
Fixed vs. the original sketch
developconsumer of the removedazure-storage.service(moved theBlob*types intostorage.service) and restoredKycDocumentService.toHostStableUrlon the storage-backed service.Verification
tsc+ full jest suite (2471 tests) + eslint green. New tests prove the compliance-download host-independence, theblobName()migration invariant (resolves both legacy and new URLs), the retention floor, and the WORM fail-closed guard.