From 81a7c2922df3dc2e6f0e3d175712fe850416ff82 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 21 Jun 2026 08:15:03 +0000 Subject: [PATCH] docs: thicken non-AWS provider depth + add cross-provider migration case (v0.6.5) Continues v0.6.4's multi-provider rebalancing by deepening the content a non-AWS diagnosis actually loads. The provider-quirks files were already solid; the gap was the cross-provider/compat references (~440-490 byte stubs). Docs + one golden case; no new scripts. - Deepened 3 cross-provider stub references to gold-standard depth (modeled on provider-differences.md, with honest verify-before-asserting hedges): migration-sync/cross-provider-compatibility.md (ETag/checksum, metadata header prefixes, ACL/ownership, storage class + provider-safe verification), event-notification/provider-compatibility.md (events are provider-native; AWS validators are model:aws only), bigdata-pipeline/provider-compatibility.md (concrete S3A config for non-AWS endpoints). - Added non-AWS migration golden case cross-provider-migration-cos-etag (AWS->COS multipart ETag differs, single-part verifies clean, --size-only 0 diffs -> not corruption -> migration-sync) with a baseline. Raises the under-covered migration-sync skill and exercises the deepened reference. Validation: all gates green; 211 pytest + 21 extension tests; 31/31 baselines PASS; routing-corpus recalls the new case. Version 0.6.4 -> 0.6.5. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01QhkyBLeDtBTmvTdVBqgCAV --- CHANGELOG.md | 28 ++++++++ docs/ARCHITECTURE.md | 2 +- docs/cli-reference.md | 2 +- pyproject.toml | 2 +- skill-registry.yaml | 2 +- .../references/provider-compatibility.md | 43 ++++++++++-- .../cross-provider-migration-cos-etag.md | 38 +++++++++++ .../description.md | 12 ++++ .../expected.json | 11 +++ .../input/report.txt | 18 +++++ .../references/provider-compatibility.md | 41 +++++++++-- .../cross-provider-compatibility.md | 68 ++++++++++++++++--- 12 files changed, 242 insertions(+), 25 deletions(-) create mode 100644 skills/storageops-eval-golden-cases/baseline-outputs/cross-provider-migration-cos-etag.md create mode 100644 skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/description.md create mode 100644 skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/expected.json create mode 100644 skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/input/report.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 5201f27..0876b4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## 2026-06-21 — v0.6.5: Thicken non-AWS provider depth + +Continues the multi-provider rebalancing (v0.6.4) by deepening the actual content a +non-AWS diagnosis loads. The `provider-quirks/` files were already solid; the gap was +the cross-provider/compat references, which were ~440–490 byte stubs. Docs + one +golden case; no new scripts. + +- **Deepened 3 cross-provider stub references to gold-standard depth** (modeled on + `security-iam-policy/references/provider-differences.md`, with honest "verify + before asserting" hedges): + - migration-sync `cross-provider-compatibility.md`: the four things that differ + (ETag/checksum, metadata header prefixes `x-amz`/`x-bce`/`x-oss`/`x-cos`, + ACL/ownership, storage class) and a provider-safe verification method; cross-links + the ETag matrix + `multipart_etag_calculator.py`. + - event-notification `provider-compatibility.md`: events are a provider-native + subsystem, not a wire feature; the AWS validators are `"model":"aws"` only; + split rule-match vs target-permission on the provider's own model. + - bigdata-pipeline `provider-compatibility.md`: concrete S3A config for non-AWS + endpoints (`fs.s3a.endpoint`, `fs.s3a.path.style.access`, signing region, + credentials provider) and what breaks committers/listings. +- **Added a non-AWS migration golden case** `cross-provider-migration-cos-etag` + (AWS→Tencent COS: multipart ETag differs, single-part objects verify clean, + `--size-only` shows 0 diffs → not corruption → migration-sync), with a baseline. + Raises the under-covered migration-sync skill and exercises the deepened reference. + +Validation: all gates green; 211 pytest + 21 extension tests; 31/31 baselines 100% +PASS; routing-corpus recalls the new case. Version 0.6.4 → 0.6.5. + ## 2026-06-21 — v0.6.4: Multi-provider rebalancing StorageOps serves *all* S3-compatible object storage, not only AWS. A self-review diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2d170e8..f6abe5f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,6 @@ # Architecture -StorageOps v0.6.4 is a Pi Coding Agent extension and skill pack. +StorageOps v0.6.5 is a Pi Coding Agent extension and skill pack. ## Components diff --git a/docs/cli-reference.md b/docs/cli-reference.md index c142296..3693f83 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -41,7 +41,7 @@ from the older local package. Deployment provenance is written to ## `storageops --version` ```text -StorageOps v0.6.4 (pi: 0.78.0) +StorageOps v0.6.5 (pi: 0.78.0) httpmon : /root/.storageops/bin/httpmon api key : api-key file independent install : yes (~/.storageops/agent) diff --git a/pyproject.toml b/pyproject.toml index 1d10482..e56ad6b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "storageops" -version = "0.6.4" +version = "0.6.5" description = "StorageOps — S3-compatible object storage diagnostic toolkit. A Pi Coding Agent extension + skill pack." requires-python = ">=3.11" readme = "README.md" diff --git a/skill-registry.yaml b/skill-registry.yaml index 2aa816e..12a6ea1 100644 --- a/skill-registry.yaml +++ b/skill-registry.yaml @@ -1,4 +1,4 @@ -# StorageOps Skill Registry v0.6.4 +# StorageOps Skill Registry v0.6.5 # Machine-readable metadata for all StorageOps skills. # Skills are auto-discovered by Pi from the skills/ directory. diff --git a/skills/storageops-bigdata-pipeline/references/provider-compatibility.md b/skills/storageops-bigdata-pipeline/references/provider-compatibility.md index 2732bc5..a76319b 100644 --- a/skills/storageops-bigdata-pipeline/references/provider-compatibility.md +++ b/skills/storageops-bigdata-pipeline/references/provider-compatibility.md @@ -1,13 +1,42 @@ # Big Data Provider Compatibility ## When to read -Use for BOS, OSS, COS, OBS, MinIO, or other non-AWS S3-compatible endpoints. +Use when Spark / Hive / Flink / Hadoop (via the S3A connector) runs against a +non-AWS S3-compatible endpoint (Baidu BOS, Alibaba OSS, Tencent COS, Huawei OBS, +MinIO) and jobs fail at connect/sign/list/commit. S3A defaults are tuned for AWS; +non-AWS endpoints usually need explicit endpoint, path-style, and signing config. -## Checks -- Endpoint style: virtual-hosted vs path-style. -- Multipart behavior and ETag semantics. -- LIST delimiter/pagination quirks. -- SDK/S3A version compatibility with provider signing requirements. +> **Verify against your Hadoop/connector version.** The `fs.s3a.*` keys below are +> stable Hadoop-AWS options, but provider-specific values and version support vary — +> confirm against your cluster's Hadoop version and the provider's docs. + +## S3A configuration for a non-AWS endpoint +- **Endpoint:** `fs.s3a.endpoint` must point at the provider endpoint (e.g. the OSS + `oss-.aliyuncs.com` / COS / BOS host), not the AWS default. +- **Path-style access:** many non-AWS endpoints need + `fs.s3a.path.style.access=true` (virtual-hosted style requires bucket DNS the + provider may not serve). A `Bucket does not exist` / DNS error with a correct + bucket is the classic symptom of the wrong addressing style. +- **Signing region:** SigV4 signs with a region in the credential scope; a non-AWS + endpoint rejects a signature scoped to the wrong region. Set the connector's + region/endpoint consistently (see the OSS signing-region case in the protocol + skill). +- **Credentials provider:** point `fs.s3a.aws.credentials.provider` at a simple + access-key provider for static provider keys rather than AWS instance/role + providers. + +## What breaks committers and listings +- **Multipart / ETag:** the S3A committers and any checksum verification assume the + AWS multipart ETag shape; non-AWS ETags differ (see + `../../storageops-s3-protocol-compatibility/references/checksum-etag.md`). Do not + rely on cross-provider ETag equality for commit validation. +- **LIST pagination / delimiter:** continuation-token and delimiter handling can + differ subtly; very large directory listings are where quirks surface. +- **Committer choice:** the magic committer depends on S3-guarantees the provider + may implement differently — verify the committer works on a small job before a + large run, and prefer a staging committer if the magic committer misbehaves. ## Routing -If errors are signature, XML, checksum, or endpoint style related, involve `storageops-s3-protocol-compatibility`. +Signature / XML / checksum / endpoint-style errors → involve +`storageops-s3-protocol-compatibility`. Throughput/throttling on listing storms → +`storageops-performance-diagnosis`. diff --git a/skills/storageops-eval-golden-cases/baseline-outputs/cross-provider-migration-cos-etag.md b/skills/storageops-eval-golden-cases/baseline-outputs/cross-provider-migration-cos-etag.md new file mode 100644 index 0000000..e9cb9f6 --- /dev/null +++ b/skills/storageops-eval-golden-cases/baseline-outputs/cross-provider-migration-cos-etag.md @@ -0,0 +1,38 @@ +# Summary + +Category: migration_sync +Route: storageops-migration-sync +Confidence: 0.87 +Root Cause Type: cross_provider_etag_mismatch_not_corruption +Evidence Quality: sufficient +Primary Diagnosis: root_cause_type=cross_provider_etag_mismatch_not_corruption, affected_layer=object-store + +The migration is not corrupting data. Tencent COS does not compute multipart ETags +with the AWS algorithm, so for every multipart object the AWS source ETag and the +COS destination ETag differ even though the bytes are identical. The evidence is +decisive: `rclone check --size-only` reports 0 differences, and only single-part +objects (whose ETag is a plain MD5) verify clean — exactly the signature of a +cross-provider multipart ETag-format difference, not data loss. + +# Key Evidence + +- `--size-only` shows 0 differences, so every object is present at the correct byte + size; the 412 "differences" come only from the hash/ETag comparison. +- Only multipart objects differ; single-part objects (plain-MD5 ETag) verify clean — + the mismatch tracks the multipart ETag computation, not the data. +- The COS multipart ETag carries the same part count (`-14`) but a different hex + digest, consistent with a different multipart algorithm rather than altered bytes. +- A multipart ETag is never the whole-object MD5, and COS's multipart computation is + not AWS's, so cross-provider ETag equality cannot be expected. + +# Remediation + +- Verify this migration with an explicit content checksum (e.g. an enabled + `x-amz-checksum-sha256` or a streamed full-object hash on a sample), not by + comparing raw multipart ETags across providers; `rclone check --size-only` plus a + sampled explicit-hash check is sufficient here. +- Make the verifier provider-aware: skip the ETag comparison for multipart objects on + COS (see the cross-provider-compatibility reference) and compare on size + explicit + checksum instead. +- Do not re-run the whole migration or delete the destination; the data is intact and + only the multipart ETag format differs between AWS and COS. diff --git a/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/description.md b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/description.md new file mode 100644 index 0000000..60e3ba7 --- /dev/null +++ b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/description.md @@ -0,0 +1,12 @@ +# Cross-provider migration: COS multipart ETag differs from AWS + +A non-AWS migration case. Copying multipart objects from AWS S3 to Tencent COS, the +post-migration hash check flags every multipart object because COS does not compute +multipart ETags with the AWS algorithm — only single-part objects (whose ETag is a +plain MD5) verify clean. The bytes are identical (`--size-only` shows 0 diffs). + +Expected diagnosis (migration-sync): not corruption — a cross-provider multipart +ETag-format difference. Verify a cross-provider migration with an explicit content +checksum (or size + a provider-native integrity signal), never by comparing raw +multipart ETags across providers. This case keeps the migration corpus covering a +non-AWS destination, and exercises the cross-provider-compatibility reference. diff --git a/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/expected.json b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/expected.json new file mode 100644 index 0000000..abcf2ab --- /dev/null +++ b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/expected.json @@ -0,0 +1,11 @@ +{ + "case_type": "diagnosis", + "expected_category": "migration_sync", + "expected_root_cause_types": ["cross_provider_etag_mismatch_not_corruption", "multipart_etag_provider_difference"], + "expected_min_confidence": 0.8, + "must_include_evidence_keywords": ["migration", "COS", "multipart", "ETag"], + "should_include_evidence_keywords": ["size-only", "single-part", "rclone"], + "must_include_recommendation_keywords": ["checksum", "verify", "ETag"], + "must_not_include": ["delete destination", "re-migrate everything", "make the bucket public", "corrupted, re-copy"], + "required_report_sections": ["Summary", "Key Evidence", "Remediation"] +} diff --git a/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/input/report.txt b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/input/report.txt new file mode 100644 index 0000000..06d62a3 --- /dev/null +++ b/skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/input/report.txt @@ -0,0 +1,18 @@ +Migration verification fails after copying from AWS S3 to Tencent COS. + +We are migrating a bucket from AWS S3 to Tencent Cloud Object Storage (COS) with +rclone sync. The transfer reports success, but our post-migration verification step +flags hundreds of large objects as mismatched: + + $ rclone check s3:prod-archive cos:prod-archive --size-only -> 0 differences + $ rclone check s3:prod-archive cos:prod-archive -> 412 differences + (hash/ETag comparison) + +Spot check of one flagged object (datasets/shard-0007.parquet, 880 MiB, multipart): + AWS S3 ETag: "7e3a1c0b9d2f48a6b5c4d3e2f1a09b8c-14" (14 parts) + COS ETag: "2f9b8c7d6e5a4b3c2d1e0f9a8b7c6d5e-14" (different hex, same -14) + object size: identical on both sides + Only the small/single-part objects verify clean; every multipart object differs. + +Is the data corrupted in transit, or is this a provider ETag difference? How do we +verify this cross-provider migration correctly? diff --git a/skills/storageops-event-notification/references/provider-compatibility.md b/skills/storageops-event-notification/references/provider-compatibility.md index 134bb23..6d3bfbc 100644 --- a/skills/storageops-event-notification/references/provider-compatibility.md +++ b/skills/storageops-event-notification/references/provider-compatibility.md @@ -1,12 +1,41 @@ # Event Notification Provider Compatibility ## When to read -Use for BOS, OSS, COS, MinIO, or other S3-compatible event systems. +Use when the bucket is on a non-AWS provider (Baidu BOS, Alibaba OSS, Tencent COS, +Huawei OBS, MinIO) and events are not delivered. The S3 *API* is compatible, but +event notification is **not** a wire-protocol feature — it is a provider-native +subsystem, so AWS assumptions (the `s3.amazonaws.com` principal, Lambda/SQS/SNS +targets) do not carry over. -## Notes -- Event names and payload schemas vary by provider. -- Some providers route through cloud-native event buses rather than AWS SNS/SQS/Lambda. -- Prefix/suffix filter semantics and retry guarantees can differ. +> **AWS-specific tooling.** `notification_config_analyzer.py` and +> `notification_target_policy_validator.py` model AWS S3→Lambda/SQS/SNS and emit +> `"model": "aws"`. They do **not** validate the providers below — use them only for +> AWS, and verify non-AWS specifics against the provider's docs. + +## What differs by provider (orient, then verify) +- **Targets are provider-native, not SNS/SQS/Lambda.** Typically the event goes to + the provider's own function-compute and message-queue services rather than AWS + ones (e.g. a serverless-function target and a managed message queue). The *target + permission model* is the provider's, not an AWS resource policy with + `s3.amazonaws.com`. +- **Event taxonomies differ.** Object-created / object-removed style events exist on + most providers, but the exact event-type strings, the multipart-completion event, + and delete-marker semantics are provider-specific — confirm the names against the + provider's docs rather than assuming `s3:ObjectCreated:*`. +- **Filter & retry semantics differ.** Prefix/suffix filter rules, overlap handling, + and delivery-retry / dead-letter guarantees are not guaranteed to match AWS. + +## Diagnostic approach on a non-AWS provider +1. Identify the provider first (endpoint host / console naming). +2. Split the two legs as always: does the rule *match* the object (event type + + filter), and does the *target* accept delivery (the provider's own permission + model)? +3. Check the target's native permission/binding in the provider console/API — not an + AWS-style resource policy. +4. Confirm the event-type string and filter semantics against the provider's docs. ## Routing -If the failure is target permissions, use this skill; if it is endpoint/signature related, route to protocol or network skills. +- Target permission / rule-match → stay in this skill (but use the provider's model, + not the AWS validators). +- Endpoint / signature / region errors → `storageops-s3-protocol-compatibility` or + `storageops-network-endpoint-access`. diff --git a/skills/storageops-migration-sync/references/cross-provider-compatibility.md b/skills/storageops-migration-sync/references/cross-provider-compatibility.md index 7af3af1..b157c23 100644 --- a/skills/storageops-migration-sync/references/cross-provider-compatibility.md +++ b/skills/storageops-migration-sync/references/cross-provider-compatibility.md @@ -1,13 +1,65 @@ # Cross-Provider Compatibility ## When to read -Use when migrations cross AWS/BOS/OSS/COS/OBS/MinIO or metadata/ACL/checksum mismatches occur. +Use when a migration or sync crosses providers (AWS S3 ↔ Baidu BOS / Alibaba OSS / +Tencent COS / Huawei OBS / MinIO / GCS), or when post-migration verification reports +metadata, ACL, or checksum mismatches. The wire protocol is S3-compatible, but the +semantics underneath are not identical — most "corruption after migration" reports +are a semantic difference, not data loss. -## Risks -- ETag is not always a full-object MD5. -- Metadata keys may be normalized or dropped. -- ACL and ownership models differ. -- Storage class and lifecycle semantics differ. +> **Verify before asserting.** Provider behaviours and header names change by +> region/edition over time. Treat the notes below as orientation and confirm +> against the provider's current docs before concluding. -## Validation -Compare object count, total bytes, sampled metadata, and checksums where provider semantics allow. +## The four things that differ (and how to handle each) + +### 1. ETag / checksum semantics — the #1 false alarm +A multipart ETag is **not** a whole-object MD5, and the multipart computation is +provider-specific: +- AWS S3 / MinIO: `MD5(concat(part MD5s))-N` (trailing part count). +- Baidu BOS: same underlying hash but a **leading dash** `-<32 hex>`, no count. +- Alibaba OSS / Tencent COS: **not** the AWS computation; treat the exact algorithm + as unverified. + +So comparing multipart ETags **across providers is meaningless**. Verify integrity +with an explicit content checksum instead (e.g. `x-amz-checksum-sha256` where +supported, or a streamed full-object hash). See +`../../storageops-s3-protocol-compatibility/references/checksum-etag.md` for the +canonical matrix, and use +`../../storageops-data-consistency/scripts/multipart_etag_calculator.py` to confirm +a re-chunk vs corruption. + +### 2. Metadata / header normalization +User metadata and system headers may be renamed or dropped across the boundary: +- AWS uses `x-amz-meta-*`; BOS uses `x-bce-meta-*`, OSS `x-oss-meta-*`, COS + `x-cos-meta-*`. A naive copy that forwards `x-amz-*` headers verbatim may lose + metadata at a non-AWS destination. +- Header **case** and ordering can be normalized; some providers lowercase keys. +- Content-Type / Content-Encoding usually carry over; custom headers may not. +Verify by sampling `HEAD` on both sides and diffing the user-metadata set. + +### 3. ACL / ownership model +Canned ACLs (`private`, `public-read`) are broadly portable, but bucket-owner +enforcement, object ownership, and public-access toggles differ by provider. A +migrated object may land with a different effective owner/ACL — re-apply the intended +ACL explicitly at the destination rather than assuming it transferred. + +### 4. Storage class & lifecycle +Class names and minimum-duration / minimum-billable rules differ (e.g. AWS IA/Glacier +vs OSS IA/Archive vs COS STANDARD_IA/ARCHIVE). A migration that preserves the source +class name may not map to an equivalent destination class — map classes explicitly. + +## Verification method (provider-safe) +1. **Completeness:** compare object count and total bytes per prefix on each side + (provider inventory reports where available). +2. **Integrity:** compare an **explicit checksum** (not the ETag) for sampled + objects; for multipart objects never compare raw ETags across providers. +3. **Metadata/ACL:** `HEAD`-sample both sides and diff user metadata and effective + ACL. +4. Treat a multipart-ETag-only mismatch as **expected** until an explicit checksum + disagrees. See `integrity-verification.md` for the full procedure. + +## Do not +- Conclude "corruption" from a cross-provider ETag mismatch alone. +- Assume `x-amz-*` metadata survives a copy to a non-AWS provider. +- Assume storage-class names or lifecycle minimums map one-to-one.