Add model registry continuity fixtures

skills/ai-security/model-supply-chain/SKILL.md

@@ -14,7 +14,7 @@ phase: [build, review, operate]
 frameworks: [OWASP-LLM03-2025, SLSA-v1.0, MITRE-ATLAS]
 difficulty: advanced
 time_estimate: "45-90min"
-version: "1.0.0"
+version: "1.0.1"
 author: unitoneai
 license: MIT
 allowed-tools: Read, Grep, Glob
@@ -82,6 +82,8 @@ Before beginning the assessment, gather the following. If any item is unavailabl
 | Model signing or attestation | CI/CD configs, SLSA provenance files, Sigstore artifacts | Confirms cryptographic supply chain verification |
 | Access controls on model storage | Cloud storage IAM, artifact registry permissions | Determines who can replace or modify model weights |
 | Adapter/plugin sources | LoRA configs, adapter download code | Third-party adapters inherit the same supply chain risks |
+| Internal mirror and retention policy | Model registry settings, object-lock policies, artifact lifecycle rules | Confirms pinned artifacts remain rebuildable after upstream deletion, gating, or license changes |
+| Rollback and rebuild evidence | Release records, restore tests, incident drills, CI rebuild logs | Shows whether production can recover when a public model, adapter, tokenizer, or config becomes unavailable |
 
 ---
 
@@ -133,7 +135,75 @@ Glob: **/config.json
 
 ---
 
-### Step 2 -- Training Data Lineage
+### Step 2 -- Registry Availability and Rollback Continuity
+
+Assess whether every production model artifact can still be rebuilt, audited, and rolled back if an upstream registry removes the model, gates access, renames the repository, changes license terms, or evicts cached files. Integrity controls prove what was downloaded; continuity controls prove the organization can still operate when the upstream source disappears.
+
+**What to look for in code and configuration:**
+
+- Production models loaded only from public registry URLs, even when revisions and checksums are pinned.
+- No immutable internal mirror for weights, adapters, tokenizer files, config files, generation templates, evaluation fixtures, or model card snapshots.
+- Registry retention described as `best_effort`, `latest`, or cache-based rather than a defined duration with deletion controls.
+- Rollback artifacts referenced as `latest` instead of a specific mirrored version, digest, and release identifier.
+- LoRA or PEFT adapters pinned by checksum while the base model, tokenizer, or config is referenced only by name.
+- License or access terms captured only by linking to the live model card, with no dated snapshot retained for the approved production release.
+- No rebuild or rollback drill showing that a clean environment can reproduce or restore the deployed model without reaching the original public registry.
+
+**Detection methods using allowed tools:**
+
+```
+# Find public registry dependencies and cache-only release paths
+Grep: "huggingface.co|hf_hub_download|snapshot_download|from_pretrained|modelscope|civitai|torch.hub" in **/*.{py,yaml,yml,json,md,toml}
+Grep: "cache_dir|TRANSFORMERS_CACHE|HF_HOME|latest|best_effort|rollback|retention|mirror" in **/*.{py,yaml,yml,json,md,toml}
+
+# Check adapter, tokenizer, and config identity binding
+Grep: "lora|qlora|peft|adapter|base_model|tokenizer|config.json|generation_config" in **/*.{py,yaml,yml,json,md,toml}
+
+# Check for immutable retention and restore evidence
+Grep: "object.lock|retention|immutab|versioning|legal.hold|restore|rebuild|disaster.recovery|drill" in **/*.{yaml,yml,json,md,toml,tf}
+```
+
+**Continuity evidence gates:**
+
+| Gate | Required Evidence | Finding if Missing |
+|---|---|---|
+| MODEL-RET-01 | Internal immutable mirror path for each production model file | Rebuild or rollback depends on public-registry availability |
+| MODEL-RET-02 | Retention duration and deletion protection for mirrored artifacts | Artifacts may be evicted before audit, rollback, or regulatory needs end |
+| MODEL-RET-03 | Complete artifact set: weights, adapter, base model, tokenizer, config, generation template, and evaluation fixtures | Partial mirror cannot reproduce deployed behavior |
+| MODEL-RET-04 | Adapter is bound to exact base model, tokenizer, config, runtime, and digest identities | Pinned adapter can attach to mutable or unavailable base components |
+| MODEL-RET-05 | Dated license, model card, acceptable-use terms, and approval snapshot | Future upstream term changes erase release-time evidence |
+| MODEL-RET-06 | Rollback copy identifies previous known-good artifact set by immutable digest and release ID | Rollback relies on `latest` or mutable public URLs |
+| MODEL-RET-07 | Clean-environment rebuild or rollback drill succeeds without contacting the original public registry | Continuity is assumed but not tested |
+| MODEL-RET-08 | Upstream deletion, gating, rename, and cache-eviction risk owner is documented | Registry availability risk has no owner or response path |
+
+**Registry Continuity Evidence output:**
+
+| Field | Description |
+|---|---|
+| Model release | Production release, model ID, version, environment, owner |
+| Upstream source | Public registry URL, repo, revision, and access terms at approval time |
+| Mirrored artifact set | Weight, adapter, base, tokenizer, config, template, eval fixture, model-card snapshot |
+| Digest binding | Digest or signature for each mirrored component |
+| Retention control | Object lock, lifecycle policy, legal hold, versioning, or deletion guard |
+| License/access snapshot | Dated license/model-card/acceptable-use snapshot and approval record |
+| Rollback target | Previous known-good release ID and immutable internal artifact path |
+| Drill evidence | Clean-environment rebuild/rollback result, date, and failure mode if any |
+| Assessment | `Compliant`, `Finding`, or `Not Evaluable` with reason |
+
+**What constitutes a finding:**
+
+| Condition | Severity |
+|---|---|
+| Production rollback depends on an upstream public model that is not internally mirrored | High |
+| Adapter is pinned but its base model, tokenizer, or config identity is mutable or unavailable | High |
+| No retention duration or deletion protection for production model artifacts | High |
+| License/access terms are not snapshotted at release approval time | Medium |
+| No clean-environment rebuild or rollback drill for critical models | Medium |
+| Registry cache may evict artifacts while metadata remains available | Medium |
+
+---
+
+### Step 3 -- Training Data Lineage
 
 Assess the provenance, integrity, and governance of data used to train or fine-tune models.
 
@@ -174,7 +244,7 @@ Grep: "s3://|gs://|az://|https://" in **/*data*.{py,yaml,yml,json,toml}
 
 ---
 
-### Step 3 -- Fine-Tuning Pipeline Security
+### Step 4 -- Fine-Tuning Pipeline Security
 
 Assess the integrity and access controls of the fine-tuning pipeline from data ingestion through weight production.
 
@@ -229,7 +299,7 @@ Glob: **/Jenkinsfile
 
 ---
 
-### Step 4 -- Inference Dependency Review
+### Step 5 -- Inference Dependency Review
 
 Assess the security of libraries, frameworks, and runtime dependencies used in the model serving path.
 
@@ -278,7 +348,7 @@ Grep: "langchain|llamaindex|llama.index|vllm|ray|transformers|onnxruntime" in **
 
 ---
 
-### Step 5 -- Model Card Evaluation
+### Step 6 -- Model Card Evaluation
 
 Assess the completeness and accuracy of model documentation as a supply chain trust signal.
 
@@ -319,7 +389,7 @@ Grep: "model.card|intended.use|training.data|evaluation|limitations|ethical" in
 
 ---
 
-### Step 6 -- Backdoor Detection Patterns
+### Step 7 -- Backdoor Detection Patterns
 
 Assess whether architectural and procedural controls exist to detect model backdoors -- targeted modifications that cause specific misbehavior on trigger inputs while maintaining normal performance on standard benchmarks.
 
@@ -378,14 +448,14 @@ Assess whether architectural and procedural controls exist to detect model backd
 
 ## Model Inventory
 
-| Model | Source | Format | Checksum Verified | Pinned Version | Model Card |
-|---|---|---|---|---|---|
-| [name] | [source] | [format] | [Yes/No] | [Yes/No] | [Complete/Partial/Missing] |
+| Model | Source | Format | Checksum Verified | Pinned Version | Internal Mirror | Retention | Rollback Copy | Model Card |
+|---|---|---|---|---|---|---|---|---|
+| [name] | [source] | [format] | [Yes/No] | [Yes/No] | [Yes/No] | [duration/policy] | [Yes/No] | [Complete/Partial/Missing] |
 
 ## Findings
 
 ### Finding [N]: [Title]
-- **Category:** [Provenance | Training Data | Fine-Tuning Pipeline | Inference Dependency | Model Card | Backdoor Detection]
+- **Category:** [Provenance | Registry Availability | Training Data | Fine-Tuning Pipeline | Inference Dependency | Model Card | Backdoor Detection]
 - **Severity:** [Critical | High | Medium | Low | Informational]
 - **OWASP LLM Category:** LLM03:2025 -- Supply Chain Vulnerabilities
 - **MITRE ATLAS Technique:** [technique ID and name]
@@ -401,6 +471,7 @@ Assess whether architectural and procedural controls exist to detect model backd
 | Domain | Current State | Target State | Gap Severity |
 |---|---|---|---|
 | Model provenance | [description] | [recommendation] | [severity] |
+| Registry availability and rollback | [description] | [recommendation] | [severity] |
 | Training data lineage | [description] | [recommendation] | [severity] |
 | Fine-tuning pipeline | [description] | [recommendation] | [severity] |
 | Inference dependencies | [description] | [recommendation] | [severity] |
@@ -433,13 +504,15 @@ Assess whether architectural and procedural controls exist to detect model backd
 
 1. **Verifying checksums against attacker-controlled sources.** Downloading a model from a public registry and verifying its checksum against a value published on the same registry provides no security. If the attacker compromised the model, they also control the published checksum. Checksums must be verified against an independently trusted source -- the model publisher's signed release, a separate attestation service, or an internal model registry that independently computed the hash on first ingestion.
 
-2. **Treating `safetensors` as a complete solution.** The `safetensors` format eliminates arbitrary code execution during deserialization, which is a critical improvement over pickle-based formats. However, it does not protect against model weight manipulation (backdoors), training data poisoning, or any other supply chain attack that operates on the model's learned parameters rather than its serialization format. `safetensors` addresses one attack vector; the other five steps in this assessment remain necessary.
+2. **Treating revision pins as availability guarantees.** Pinning a Hugging Face commit and verifying a digest protects integrity, but it does not guarantee that the model, tokenizer, adapter base, or license terms will remain accessible later. Production releases need an immutable internal mirror, retention policy, and tested rollback copy for the full artifact set.
+
+3. **Treating `safetensors` as a complete solution.** The `safetensors` format eliminates arbitrary code execution during deserialization, which is a critical improvement over pickle-based formats. However, it does not protect against model weight manipulation (backdoors), training data poisoning, unavailable upstream artifacts, or any other supply chain attack that operates on the model's learned parameters rather than its serialization format. `safetensors` addresses one attack vector; the other steps in this assessment remain necessary.
 
-3. **Auditing application dependencies but ignoring ML framework dependencies.** Standard SCA tooling often covers `requests`, `flask`, or `django` but misses ML-specific libraries (transformers, vLLM, Ray, LangChain) that have had critical CVEs. Ensure vulnerability scanning covers the full dependency tree including ML frameworks.
+4. **Auditing application dependencies but ignoring ML framework dependencies.** Standard SCA tooling often covers `requests`, `flask`, or `django` but misses ML-specific libraries (transformers, vLLM, Ray, LangChain) that have had critical CVEs. Ensure vulnerability scanning covers the full dependency tree including ML frameworks.
 
-4. **Assuming Hugging Face models are vetted.** Hugging Face Hub is a hosting platform, not a curation service. Any user can upload any model. While Hugging Face has introduced malware scanning and model signing capabilities, the majority of hosted models have no cryptographic provenance. Treat Hugging Face models as untrusted artifacts requiring verification, the same way you treat npm packages.
+5. **Assuming Hugging Face models are vetted.** Hugging Face Hub is a hosting platform, not a curation service. Any user can upload any model. While Hugging Face has introduced malware scanning and model signing capabilities, the majority of hosted models have no cryptographic provenance. Treat Hugging Face models as untrusted artifacts requiring verification, the same way you treat npm packages.
 
-5. **Evaluating models only on benchmarks.** Standard benchmarks measure general capability, not supply chain integrity. A backdoored model will perform normally on benchmarks by design. Behavioral differential testing with curated, domain-specific test sets that probe for targeted manipulation is required to surface backdoors.
+6. **Evaluating models only on benchmarks.** Standard benchmarks measure general capability, not supply chain integrity. A backdoored model will perform normally on benchmarks by design. Behavioral differential testing with curated, domain-specific test sets that probe for targeted manipulation is required to surface backdoors.
 
 ---
 

skills/ai-security/model-supply-chain/tests/benign/mirrored-model-rollback-continuity.yaml

@@ -0,0 +1,46 @@
+case_id: MODEL-RET-BENIGN-001
+description: Production model release remains rebuildable after upstream removal because the full artifact set is mirrored and rollback-tested.
+model_release:
+  name: support-safety-classifier
+  release_id: msc-2026-06-09.3
+  environment: production
+upstream_source:
+  registry: huggingface
+  repository: org-example/safety-classifier
+  revision: 8f2b4c1d9a0e
+  access_terms_snapshot: licenses/safety-classifier-2026-06-01.md
+mirrored_artifacts:
+  internal_registry: s3://ml-artifacts-prod/models/support-safety-classifier/msc-2026-06-09.3
+  retention_years: 7
+  object_lock: compliance
+  versioning: true
+  files:
+    weights: model.safetensors
+    adapter: domain-lora.safetensors
+    base_model: base-model.safetensors
+    tokenizer: tokenizer.json
+    config: config.json
+    generation_template: generation_config.json
+    model_card_snapshot: MODEL_CARD.md
+    eval_fixture: rollback-smoke-eval.jsonl
+continuity_controls:
+  adapter_bound_to_base_digest: true
+  tokenizer_digest_bound: true
+  rollback_target_release_id: msc-2026-05-30.1
+  rollback_uses_internal_mirror_only: true
+  clean_environment_drill:
+    date: "2026-06-07"
+    upstream_network_disabled: true
+    result: passed
+expected_assessment:
+  result: no_finding
+  satisfied_gates:
+    - MODEL-RET-01
+    - MODEL-RET-02
+    - MODEL-RET-03
+    - MODEL-RET-04
+    - MODEL-RET-05
+    - MODEL-RET-06
+    - MODEL-RET-07
+    - MODEL-RET-08
+  reason: The release is mirrored with retention, complete artifact identity, license snapshot, rollback target, and clean-environment drill evidence.

skills/ai-security/model-supply-chain/tests/vulnerable/public-registry-yanked-model-no-mirror.yaml

@@ -0,0 +1,46 @@
+case_id: MODEL-RET-VULN-001
+description: Pinned public model and adapter cannot be rebuilt or rolled back after upstream removal because continuity evidence is missing.
+model_release:
+  name: vendor-domain-lora-classifier
+  release_id: latest
+  environment: production
+upstream_source:
+  registry: huggingface
+  repository: vendor/domain-lora
+  revision: 8f2b4c1d9a0e
+  sha256_recorded: true
+  access_terms_snapshot: null
+artifact_identity:
+  adapter_digest_pinned: true
+  base_model_reference: example/safety-classifier
+  base_model_revision: null
+  tokenizer_digest: null
+  config_digest: null
+mirror_and_retention:
+  internal_mirror: none
+  registry_retention: best_effort_cache
+  object_lock: false
+  rollback_artifact: latest
+  model_card_snapshot: live_url_only
+rebuild_drill:
+  clean_environment_tested: false
+  upstream_removed_or_gated_scenario: not_tested
+expected_findings:
+  - gate: MODEL-RET-01
+    severity: High
+    reason: Production rebuild and rollback depend on a public registry with no immutable internal mirror.
+  - gate: MODEL-RET-03
+    severity: High
+    reason: Runtime-critical tokenizer, config, base model, and model-card snapshots are not retained.
+  - gate: MODEL-RET-04
+    severity: High
+    reason: The adapter is pinned but its base model and tokenizer identity are mutable or unavailable.
+  - gate: MODEL-RET-05
+    severity: Medium
+    reason: License and access terms are referenced by live URL only.
+  - gate: MODEL-RET-06
+    severity: High
+    reason: Rollback uses latest instead of an immutable internal release ID and digest.
+  - gate: MODEL-RET-07
+    severity: Medium
+    reason: No clean-environment rebuild or rollback drill proves continuity without the upstream registry.