Thicken non-AWS provider depth: deepen cross-provider references + COS migration case (v0.6.5)

CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## 2026-06-21 — v0.6.5: Thicken non-AWS provider depth
+
+Continues the multi-provider rebalancing (v0.6.4) by deepening the actual content a
+non-AWS diagnosis loads. The `provider-quirks/` files were already solid; the gap was
+the cross-provider/compat references, which were ~440–490 byte stubs. Docs + one
+golden case; no new scripts.
+
+- **Deepened 3 cross-provider stub references to gold-standard depth** (modeled on
+  `security-iam-policy/references/provider-differences.md`, with honest "verify
+  before asserting" hedges):
+  - migration-sync `cross-provider-compatibility.md`: the four things that differ
+    (ETag/checksum, metadata header prefixes `x-amz`/`x-bce`/`x-oss`/`x-cos`,
+    ACL/ownership, storage class) and a provider-safe verification method; cross-links
+    the ETag matrix + `multipart_etag_calculator.py`.
+  - event-notification `provider-compatibility.md`: events are a provider-native
+    subsystem, not a wire feature; the AWS validators are `"model":"aws"` only;
+    split rule-match vs target-permission on the provider's own model.
+  - bigdata-pipeline `provider-compatibility.md`: concrete S3A config for non-AWS
+    endpoints (`fs.s3a.endpoint`, `fs.s3a.path.style.access`, signing region,
+    credentials provider) and what breaks committers/listings.
+- **Added a non-AWS migration golden case** `cross-provider-migration-cos-etag`
+  (AWS→Tencent COS: multipart ETag differs, single-part objects verify clean,
+  `--size-only` shows 0 diffs → not corruption → migration-sync), with a baseline.
+  Raises the under-covered migration-sync skill and exercises the deepened reference.
+
+Validation: all gates green; 211 pytest + 21 extension tests; 31/31 baselines 100%
+PASS; routing-corpus recalls the new case. Version 0.6.4 → 0.6.5.
+
 ## 2026-06-21 — v0.6.4: Multi-provider rebalancing
 
 StorageOps serves *all* S3-compatible object storage, not only AWS. A self-review

docs/ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # Architecture
 
-StorageOps v0.6.4 is a Pi Coding Agent extension and skill pack.
+StorageOps v0.6.5 is a Pi Coding Agent extension and skill pack.
 
 ## Components
 

docs/cli-reference.md

@@ -41,7 +41,7 @@ from the older local package. Deployment provenance is written to
 ## `storageops --version`
 
 ```text
-StorageOps v0.6.4  (pi: 0.78.0)
+StorageOps v0.6.5  (pi: 0.78.0)
   httpmon             : /root/.storageops/bin/httpmon
   api key             : api-key file
   independent install : yes  (~/.storageops/agent)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "storageops"
-version = "0.6.4"
+version = "0.6.5"
 description = "StorageOps — S3-compatible object storage diagnostic toolkit. A Pi Coding Agent extension + skill pack."
 requires-python = ">=3.11"
 readme = "README.md"

skill-registry.yaml

@@ -1,4 +1,4 @@
-# StorageOps Skill Registry v0.6.4
+# StorageOps Skill Registry v0.6.5
 
 # Machine-readable metadata for all StorageOps skills.
 # Skills are auto-discovered by Pi from the skills/ directory.

skills/storageops-bigdata-pipeline/references/provider-compatibility.md

@@ -1,13 +1,42 @@
 # Big Data Provider Compatibility
 
 ## When to read
-Use for BOS, OSS, COS, OBS, MinIO, or other non-AWS S3-compatible endpoints.
+Use when Spark / Hive / Flink / Hadoop (via the S3A connector) runs against a
+non-AWS S3-compatible endpoint (Baidu BOS, Alibaba OSS, Tencent COS, Huawei OBS,
+MinIO) and jobs fail at connect/sign/list/commit. S3A defaults are tuned for AWS;
+non-AWS endpoints usually need explicit endpoint, path-style, and signing config.
 
-## Checks
-- Endpoint style: virtual-hosted vs path-style.
-- Multipart behavior and ETag semantics.
-- LIST delimiter/pagination quirks.
-- SDK/S3A version compatibility with provider signing requirements.
+> **Verify against your Hadoop/connector version.** The `fs.s3a.*` keys below are
+> stable Hadoop-AWS options, but provider-specific values and version support vary —
+> confirm against your cluster's Hadoop version and the provider's docs.
+
+## S3A configuration for a non-AWS endpoint
+- **Endpoint:** `fs.s3a.endpoint` must point at the provider endpoint (e.g. the OSS
+  `oss-<region>.aliyuncs.com` / COS / BOS host), not the AWS default.
+- **Path-style access:** many non-AWS endpoints need
+  `fs.s3a.path.style.access=true` (virtual-hosted style requires bucket DNS the
+  provider may not serve). A `Bucket does not exist` / DNS error with a correct
+  bucket is the classic symptom of the wrong addressing style.
+- **Signing region:** SigV4 signs with a region in the credential scope; a non-AWS
+  endpoint rejects a signature scoped to the wrong region. Set the connector's
+  region/endpoint consistently (see the OSS signing-region case in the protocol
+  skill).
+- **Credentials provider:** point `fs.s3a.aws.credentials.provider` at a simple
+  access-key provider for static provider keys rather than AWS instance/role
+  providers.
+
+## What breaks committers and listings
+- **Multipart / ETag:** the S3A committers and any checksum verification assume the
+  AWS multipart ETag shape; non-AWS ETags differ (see
+  `../../storageops-s3-protocol-compatibility/references/checksum-etag.md`). Do not
+  rely on cross-provider ETag equality for commit validation.
+- **LIST pagination / delimiter:** continuation-token and delimiter handling can
+  differ subtly; very large directory listings are where quirks surface.
+- **Committer choice:** the magic committer depends on S3-guarantees the provider
+  may implement differently — verify the committer works on a small job before a
+  large run, and prefer a staging committer if the magic committer misbehaves.
 
 ## Routing
-If errors are signature, XML, checksum, or endpoint style related, involve `storageops-s3-protocol-compatibility`.
+Signature / XML / checksum / endpoint-style errors → involve
+`storageops-s3-protocol-compatibility`. Throughput/throttling on listing storms →
+`storageops-performance-diagnosis`.

skills/storageops-eval-golden-cases/baseline-outputs/cross-provider-migration-cos-etag.md

@@ -0,0 +1,38 @@
+# Summary
+
+Category: migration_sync
+Route: storageops-migration-sync
+Confidence: 0.87
+Root Cause Type: cross_provider_etag_mismatch_not_corruption
+Evidence Quality: sufficient
+Primary Diagnosis: root_cause_type=cross_provider_etag_mismatch_not_corruption, affected_layer=object-store
+
+The migration is not corrupting data. Tencent COS does not compute multipart ETags
+with the AWS algorithm, so for every multipart object the AWS source ETag and the
+COS destination ETag differ even though the bytes are identical. The evidence is
+decisive: `rclone check --size-only` reports 0 differences, and only single-part
+objects (whose ETag is a plain MD5) verify clean — exactly the signature of a
+cross-provider multipart ETag-format difference, not data loss.
+
+# Key Evidence
+
+- `--size-only` shows 0 differences, so every object is present at the correct byte
+  size; the 412 "differences" come only from the hash/ETag comparison.
+- Only multipart objects differ; single-part objects (plain-MD5 ETag) verify clean —
+  the mismatch tracks the multipart ETag computation, not the data.
+- The COS multipart ETag carries the same part count (`-14`) but a different hex
+  digest, consistent with a different multipart algorithm rather than altered bytes.
+- A multipart ETag is never the whole-object MD5, and COS's multipart computation is
+  not AWS's, so cross-provider ETag equality cannot be expected.
+
+# Remediation
+
+- Verify this migration with an explicit content checksum (e.g. an enabled
+  `x-amz-checksum-sha256` or a streamed full-object hash on a sample), not by
+  comparing raw multipart ETags across providers; `rclone check --size-only` plus a
+  sampled explicit-hash check is sufficient here.
+- Make the verifier provider-aware: skip the ETag comparison for multipart objects on
+  COS (see the cross-provider-compatibility reference) and compare on size + explicit
+  checksum instead.
+- Do not re-run the whole migration or delete the destination; the data is intact and
+  only the multipart ETag format differs between AWS and COS.

skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/description.md

@@ -0,0 +1,12 @@
+# Cross-provider migration: COS multipart ETag differs from AWS
+
+A non-AWS migration case. Copying multipart objects from AWS S3 to Tencent COS, the
+post-migration hash check flags every multipart object because COS does not compute
+multipart ETags with the AWS algorithm — only single-part objects (whose ETag is a
+plain MD5) verify clean. The bytes are identical (`--size-only` shows 0 diffs).
+
+Expected diagnosis (migration-sync): not corruption — a cross-provider multipart
+ETag-format difference. Verify a cross-provider migration with an explicit content
+checksum (or size + a provider-native integrity signal), never by comparing raw
+multipart ETags across providers. This case keeps the migration corpus covering a
+non-AWS destination, and exercises the cross-provider-compatibility reference.

skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/expected.json

@@ -0,0 +1,11 @@
+{
+  "case_type": "diagnosis",
+  "expected_category": "migration_sync",
+  "expected_root_cause_types": ["cross_provider_etag_mismatch_not_corruption", "multipart_etag_provider_difference"],
+  "expected_min_confidence": 0.8,
+  "must_include_evidence_keywords": ["migration", "COS", "multipart", "ETag"],
+  "should_include_evidence_keywords": ["size-only", "single-part", "rclone"],
+  "must_include_recommendation_keywords": ["checksum", "verify", "ETag"],
+  "must_not_include": ["delete destination", "re-migrate everything", "make the bucket public", "corrupted, re-copy"],
+  "required_report_sections": ["Summary", "Key Evidence", "Remediation"]
+}

skills/storageops-eval-golden-cases/cases/cross-provider-migration-cos-etag/input/report.txt

@@ -0,0 +1,18 @@
+Migration verification fails after copying from AWS S3 to Tencent COS.
+
+We are migrating a bucket from AWS S3 to Tencent Cloud Object Storage (COS) with
+rclone sync. The transfer reports success, but our post-migration verification step
+flags hundreds of large objects as mismatched:
+
+  $ rclone check s3:prod-archive cos:prod-archive --size-only   -> 0 differences
+  $ rclone check s3:prod-archive cos:prod-archive               -> 412 differences
+       (hash/ETag comparison)
+
+Spot check of one flagged object (datasets/shard-0007.parquet, 880 MiB, multipart):
+  AWS S3 ETag:  "7e3a1c0b9d2f48a6b5c4d3e2f1a09b8c-14"   (14 parts)
+  COS ETag:     "2f9b8c7d6e5a4b3c2d1e0f9a8b7c6d5e-14"   (different hex, same -14)
+  object size:  identical on both sides
+  Only the small/single-part objects verify clean; every multipart object differs.
+
+Is the data corrupted in transit, or is this a provider ETag difference? How do we
+verify this cross-provider migration correctly?

skills/storageops-event-notification/references/provider-compatibility.md

@@ -1,12 +1,41 @@
 # Event Notification Provider Compatibility
 
 ## When to read
-Use for BOS, OSS, COS, MinIO, or other S3-compatible event systems.
+Use when the bucket is on a non-AWS provider (Baidu BOS, Alibaba OSS, Tencent COS,
+Huawei OBS, MinIO) and events are not delivered. The S3 *API* is compatible, but
+event notification is **not** a wire-protocol feature — it is a provider-native
+subsystem, so AWS assumptions (the `s3.amazonaws.com` principal, Lambda/SQS/SNS
+targets) do not carry over.
 
-## Notes
-- Event names and payload schemas vary by provider.
-- Some providers route through cloud-native event buses rather than AWS SNS/SQS/Lambda.
-- Prefix/suffix filter semantics and retry guarantees can differ.
+> **AWS-specific tooling.** `notification_config_analyzer.py` and
+> `notification_target_policy_validator.py` model AWS S3→Lambda/SQS/SNS and emit
+> `"model": "aws"`. They do **not** validate the providers below — use them only for
+> AWS, and verify non-AWS specifics against the provider's docs.
+
+## What differs by provider (orient, then verify)
+- **Targets are provider-native, not SNS/SQS/Lambda.** Typically the event goes to
+  the provider's own function-compute and message-queue services rather than AWS
+  ones (e.g. a serverless-function target and a managed message queue). The *target
+  permission model* is the provider's, not an AWS resource policy with
+  `s3.amazonaws.com`.
+- **Event taxonomies differ.** Object-created / object-removed style events exist on
+  most providers, but the exact event-type strings, the multipart-completion event,
+  and delete-marker semantics are provider-specific — confirm the names against the
+  provider's docs rather than assuming `s3:ObjectCreated:*`.
+- **Filter & retry semantics differ.** Prefix/suffix filter rules, overlap handling,
+  and delivery-retry / dead-letter guarantees are not guaranteed to match AWS.
+
+## Diagnostic approach on a non-AWS provider
+1. Identify the provider first (endpoint host / console naming).
+2. Split the two legs as always: does the rule *match* the object (event type +
+   filter), and does the *target* accept delivery (the provider's own permission
+   model)?
+3. Check the target's native permission/binding in the provider console/API — not an
+   AWS-style resource policy.
+4. Confirm the event-type string and filter semantics against the provider's docs.
 
 ## Routing
-If the failure is target permissions, use this skill; if it is endpoint/signature related, route to protocol or network skills.
+- Target permission / rule-match → stay in this skill (but use the provider's model,
+  not the AWS validators).
+- Endpoint / signature / region errors → `storageops-s3-protocol-compatibility` or
+  `storageops-network-endpoint-access`.

skills/storageops-migration-sync/references/cross-provider-compatibility.md

@@ -1,13 +1,65 @@
 # Cross-Provider Compatibility
 
 ## When to read
-Use when migrations cross AWS/BOS/OSS/COS/OBS/MinIO or metadata/ACL/checksum mismatches occur.
+Use when a migration or sync crosses providers (AWS S3 ↔ Baidu BOS / Alibaba OSS /
+Tencent COS / Huawei OBS / MinIO / GCS), or when post-migration verification reports
+metadata, ACL, or checksum mismatches. The wire protocol is S3-compatible, but the
+semantics underneath are not identical — most "corruption after migration" reports
+are a semantic difference, not data loss.
 
-## Risks
-- ETag is not always a full-object MD5.
-- Metadata keys may be normalized or dropped.
-- ACL and ownership models differ.
-- Storage class and lifecycle semantics differ.
+> **Verify before asserting.** Provider behaviours and header names change by
+> region/edition over time. Treat the notes below as orientation and confirm
+> against the provider's current docs before concluding.
 
-## Validation
-Compare object count, total bytes, sampled metadata, and checksums where provider semantics allow.
+## The four things that differ (and how to handle each)
+
+### 1. ETag / checksum semantics — the #1 false alarm
+A multipart ETag is **not** a whole-object MD5, and the multipart computation is
+provider-specific:
+- AWS S3 / MinIO: `MD5(concat(part MD5s))-N` (trailing part count).
+- Baidu BOS: same underlying hash but a **leading dash** `-<32 hex>`, no count.
+- Alibaba OSS / Tencent COS: **not** the AWS computation; treat the exact algorithm
+  as unverified.
+
+So comparing multipart ETags **across providers is meaningless**. Verify integrity
+with an explicit content checksum instead (e.g. `x-amz-checksum-sha256` where
+supported, or a streamed full-object hash). See
+`../../storageops-s3-protocol-compatibility/references/checksum-etag.md` for the
+canonical matrix, and use
+`../../storageops-data-consistency/scripts/multipart_etag_calculator.py` to confirm
+a re-chunk vs corruption.
+
+### 2. Metadata / header normalization
+User metadata and system headers may be renamed or dropped across the boundary:
+- AWS uses `x-amz-meta-*`; BOS uses `x-bce-meta-*`, OSS `x-oss-meta-*`, COS
+  `x-cos-meta-*`. A naive copy that forwards `x-amz-*` headers verbatim may lose
+  metadata at a non-AWS destination.
+- Header **case** and ordering can be normalized; some providers lowercase keys.
+- Content-Type / Content-Encoding usually carry over; custom headers may not.
+Verify by sampling `HEAD` on both sides and diffing the user-metadata set.
+
+### 3. ACL / ownership model
+Canned ACLs (`private`, `public-read`) are broadly portable, but bucket-owner
+enforcement, object ownership, and public-access toggles differ by provider. A
+migrated object may land with a different effective owner/ACL — re-apply the intended
+ACL explicitly at the destination rather than assuming it transferred.
+
+### 4. Storage class & lifecycle
+Class names and minimum-duration / minimum-billable rules differ (e.g. AWS IA/Glacier
+vs OSS IA/Archive vs COS STANDARD_IA/ARCHIVE). A migration that preserves the source
+class name may not map to an equivalent destination class — map classes explicitly.
+
+## Verification method (provider-safe)
+1. **Completeness:** compare object count and total bytes per prefix on each side
+   (provider inventory reports where available).
+2. **Integrity:** compare an **explicit checksum** (not the ETag) for sampled
+   objects; for multipart objects never compare raw ETags across providers.
+3. **Metadata/ACL:** `HEAD`-sample both sides and diff user metadata and effective
+   ACL.
+4. Treat a multipart-ETag-only mismatch as **expected** until an explicit checksum
+   disagrees. See `integrity-verification.md` for the full procedure.
+
+## Do not
+- Conclude "corruption" from a cross-provider ETag mismatch alone.
+- Assume `x-amz-*` metadata survives a copy to a non-AWS provider.
+- Assume storage-class names or lifecycle minimums map one-to-one.