RHEcosystemAppEng · dmartinol · Jun 5, 2026 · Jun 5, 2026 · Jun 5, 2026
@@ -41,18 +41,45 @@ Use this when reviewing a PR that adds or modifies a federated module in `market
 2. **Extract:**
    - Module name
    - Repository URL
-   - Ref (if provided)
+   - Ref (required 40-character commit SHA)
    - Path within the repo
-   - License (if declared)
-3. **Output to user:** Summary of the proposed federated module
+3. **Validate ref format** before continuing:
+   - Must be exactly 40 hexadecimal characters
+   - Reject branch names, tags, or short SHAs
+4. **Output to user:** Summary of the proposed federated module
+
+### Phase 1.5: Validate catalog against external repo
+
+Cross-check `federation/modules/<name>/.catalog/` against the external pack at the pinned `ref`, and verify `docs/plugins.json` / marketplace alignment.
+
+1. **Action:** Run the catalog cross-check script (skip if the PR does not add a federation catalog yet):
+
+```bash
+uv run python scripts/validate_federation_catalog.py \
+  --module-name <module-name> \
+  --repo-url <repo-url> \
+  --ref <commit-sha> \
+  --pack-path <path> \
+  --module-json '<module-json-from-marketplace-yaml>'
+```
+
+2. **Output to user:** Pass/fail for each check:
+   - Catalog present under `federation/modules/<name>/.catalog/`
+   - Collection compliance (schema, fragments, JSON mirror)
+   - **External skill roster** — catalog `contents.skills` matches `skills/` in the linked repo at `ref`
+   - `docs/plugins.json` title matches catalog `name:`; catalog `id:` matches module name
+   - Marketplace `version`, `description`, and `repository` align with catalog
+
+3. **If the catalog is missing** from the PR, report that cross-check was skipped and request the contributor add `federation/modules/<name>/.catalog/` before merge.
 
 ### Phase 2: License check
 
-1. **Action:** Check the repository for a LICENSE file:
+1. **Action:** Clone the repository at the pinned commit and check for a LICENSE file:
 
 ```bash
-git clone --quiet --depth 1 <repo-url> /tmp/federation-review
-cat /tmp/federation-review/LICENSE
+git clone --quiet --no-checkout <repo-url> /tmp/federation-review
+cd /tmp/federation-review && git checkout --quiet <commit-sha>
+cat LICENSE
 ```
 
 2. **Evaluate** license compatibility with Apache 2.0:
@@ -62,17 +89,15 @@ cat /tmp/federation-review/LICENSE
 
 ### Phase 3: Run automated validation
 
-1. **Action:** Run the validation script:
+1. **Action:** Run the validation script (``--ref`` is required and must be a commit SHA):
 
 ```bash
-uv run python scripts/validate_federation.py <repo-url> --pack-path <path> --module-json '<module-json>'
-
-# With a specific ref
-uv run python scripts/validate_federation.py <repo-url> --ref <ref> --pack-path <path> --module-json '<module-json>'
+uv run python scripts/validate_federation.py <repo-url> --ref <commit-sha> --pack-path <path> --module-json '<module-json>'
 ```
 
 2. **Output to user:** The full validation report with pass/fail for each check:
-   - Clone and access
+   - Federation ref (40-character commit SHA)
+   - Clone and access at pinned commit
    - Lola module schema (name, description, version, repository)
    - Tier 1 (agentskills.io spec)
    - Tier 2 (design principles)
@@ -84,9 +109,10 @@ uv run python scripts/validate_federation.py <repo-url> --ref <ref> --pack-path
 1. **Action:** Verify the module is loadable by Lola using the PR branch:
 
 ```bash
-lola market add test-federation https://raw.githubusercontent.com/<owner>/<repo>/<branch>/marketplace/rh-agentic-collection.yml
-lola market ls test-federation
-lola market rm test-federation
+MARKET="federation-review-$(openssl rand -hex 4)"
+lola market add "$MARKET" https://raw.githubusercontent.com/<owner>/<repo>/<branch>/marketplace/rh-agentic-collection.yml
+lola market ls "$MARKET"
+lola market rm "$MARKET"
 ```
 
 2. **Check:** The federated module appears in the module list with correct name, version, and description.
@@ -98,6 +124,7 @@ Present a combined summary to the user:
 
 | Check | Result |
 |-------|--------|
+| Catalog cross-check | Passed / Failed / Skipped (no catalog in PR) |
 | License | Compatible / Incompatible / Not found |
 | Automated validation | All passed / N checks failed |
 | Lola verification | Module visible / Not visible |
@@ -121,12 +148,13 @@ None — no MCP tools are invoked.
 ### Reference Documentation
 
 **Internal:**
-- [Federation Review Guide](../../../docs/FEDERATION_REVIEW_GUIDE.md) — full evaluation criteria
+- [Federation Review Guide](../../../FEDERATION_REVIEW_GUIDE.md) — full evaluation criteria
 - [CONTRIBUTING.md](../../../CONTRIBUTING.md) — contribution paths overview
 - [SKILL_DESIGN_PRINCIPLES.md](../../../SKILL_DESIGN_PRINCIPLES.md) — Tier 2 design principles
 
 **Scripts:**
-- `scripts/validate_federation.py` — automated validation checks
+- `scripts/validate_federation.py` — external pack validation at pinned ref
+- `scripts/validate_federation_catalog.py` — federation catalog cross-check vs external repo
 
 ## Human-in-the-Loop
 
@@ -141,9 +169,14 @@ Skill: Reading PR diff...
        Found federated module: cursor-sdk
        Repository: https://github.com/cursor/plugins
        Path: cursor-sdk
-       Ref: (default branch)
+       Ref: a1b2c3d4e5f6789012345678901234567890abcd
+
+       Checking catalog vs external repo...
+       uv run python scripts/validate_federation_catalog.py ...
+       ✅ external_skill_roster matches linked repo at ref
+       ✅ plugins.json title matches catalog
 
-       Checking license...
+       Checking license at pinned commit...
        LICENSE file: MIT
        MIT is compatible with Apache 2.0. Confirm? [Y/n]
 User: Y
@@ -158,6 +191,7 @@ Skill: Running automated validation...
        ✅ Module "cursor-sdk" visible in marketplace
 
        Summary:
+       - Catalog cross-check: ✅ roster and metadata aligned
        - License: ✅ MIT (compatible)
        - Validation: ❌ Tier 1 failures
        - Lola: ✅ Module loadable

@@ -55,8 +55,8 @@
 
 - **Repository URL:** <!-- https://github.com/org/repo -->
 - **Pack path:** <!-- Subdirectory within the repo, or "." for root -->
-- **Ref:** <!-- Commit SHA or tag, or "default branch" -->
-- **License:** <!-- Must be compatible with Apache 2.0 (Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause) -->
+- **Ref:** <!-- Required 40-character commit SHA (not a branch or tag) -->
+- **License:** <!-- Verified from repo LICENSE during review; must be compatible with Apache 2.0 -->
 - **Contact:** <!-- @github-handle or email of the pack owner -->
 
 ## Validation

@@ -160,6 +160,8 @@ The workflow will:
 - ✅ Required files presence (README.md, mcps.json, etc.)
 - ✅ Plugin metadata completeness
 - ✅ MCP server configurations
+- ✅ Collection catalog compliance (`.catalog/` schema, fragments, JSON mirror)
+- ✅ Federated catalog cross-check (clone external repos at pinned `ref`; roster and marketplace metadata)
 
 **Changed skills validation (`./scripts/ci-validate-changed-skills.sh`):**
 - ✅ Detects which skills were modified in the PR/push

@@ -103,10 +103,14 @@ jobs:
               name = mod.get('name', 'unknown')
 
               print(f'\n--- Validating: {name} ---')
-              cmd = [sys.executable, 'scripts/validate_federation.py', repo, '--pack-path', pack_path, '--json',
+              if not ref:
+                  print(f'Module {name} missing required ref (40-character commit SHA)')
+                  failed = True
+                  all_results.append({'name': name, 'report': None, 'catalog_report': None})
+                  continue
+
+              cmd = [sys.executable, 'scripts/validate_federation.py', repo, '--ref', ref, '--pack-path', pack_path, '--json',
                      '--module-json', json.dumps(mod)]
-              if ref:
-                  cmd.extend(['--ref', ref])
               if skills:
                   cmd.extend(['--skills'] + skills)
 
@@ -117,13 +121,38 @@ jobs:
 
               try:
                   report = json.loads(result.stdout)
-                  all_results.append({'name': name, 'report': report})
               except json.JSONDecodeError:
-                  all_results.append({'name': name, 'report': None})
+                  report = None
 
               if result.returncode != 0:
                   failed = True
 
+              cat_cmd = [
+                  sys.executable, 'scripts/validate_federation_catalog.py',
+                  '--module-name', name,
+                  '--repo-url', repo,
+                  '--ref', ref,
+                  '--pack-path', pack_path,
+                  '--module-json', json.dumps(mod),
+                  '--json',
+              ]
+              cat_result = subprocess.run(cat_cmd, capture_output=True, text=True)
+              print(cat_result.stdout)
+              if cat_result.stderr:
+                  print(cat_result.stderr, file=sys.stderr)
+              try:
+                  cat_report = json.loads(cat_result.stdout)
+              except json.JSONDecodeError:
+                  cat_report = None
+              if cat_result.returncode != 0:
+                  failed = True
+
+              all_results.append({
+                  'name': name,
+                  'report': report,
+                  'catalog_report': cat_report,
+              })
+
           with open('/tmp/federation-results.json', 'w') as f:
               json.dump(all_results, f)
 
@@ -142,12 +171,18 @@ jobs:
             const timestamp = new Date().toISOString().replace('T', ' ').substring(0, 19) + ' UTC';
 
             const checkLabels = {
-              clone: { name: 'Clone & access', desc: 'Repository cloned (pinned ref or default branch)' },
+              clone: { name: 'Clone & access', desc: 'Repository cloned at pinned commit SHA' },
+              federation_ref: { name: 'Commit SHA', desc: 'ref is a required 40-character commit SHA' },
               lola_structure: { name: 'Lola module schema', desc: 'Required fields: name, description, version, repository' },
               tier1_agentskills: { name: 'Tier 1', desc: 'agentskills.io spec compliance' },
               tier2_design_principles: { name: 'Tier 2', desc: 'Design principles compliance (warning only)' },
               mcp_version_pinning: { name: 'MCP pinning', desc: 'No `:latest` tags, no hardcoded credentials' },
               gitleaks: { name: 'Credential scan', desc: 'gitleaks secret detection' },
+              catalog_present: { name: 'Catalog present', desc: 'federation/modules/<name>/.catalog/collection.yaml exists' },
+              catalog_compliance: { name: 'Catalog compliance', desc: 'Schema, fragments, JSON mirror' },
+              external_skill_roster: { name: 'Catalog roster', desc: 'Catalog skills match external pack at ref' },
+              plugins_json_title: { name: 'plugins.json title', desc: 'Display title matches catalog name/id' },
+              marketplace_catalog_metadata: { name: 'Marketplace alignment', desc: 'Version, description, repository match catalog' },
             };
             // Checks that warn instead of blocking the merge
             const warnOnlyChecks = new Set(['tier2_design_principles']);
@@ -165,12 +200,23 @@ jobs:
                 }
                 moduleSections += '| Check | Description | Status |\n';
                 moduleSections += '|-------|-------------|--------|\n';
-                for (const check of report.checks) {
+                for check of report.checks) {
                   const label = checkLabels[check.name] || { name: check.name, desc: '' };
                   const isWarnOnly = warnOnlyChecks.has(check.name);
                   const statusIcon = check.skipped ? '⚠️' : (check.passed ? '✅' : (isWarnOnly ? '⚠️' : '❌'));
                   moduleSections += `| ${label.name} | ${label.desc} | ${statusIcon} |\n`;
                 }
+                const catReport = mod.catalog_report;
+                if (catReport && catReport.checks) {
+                  moduleSections += '\n**Catalog cross-check**\n\n';
+                  moduleSections += '| Check | Description | Status |\n';
+                  moduleSections += '|-------|-------------|--------|\n';
+                  for (const check of catReport.checks) {
+                    const label = checkLabels[check.name] || { name: check.name, desc: '' };
+                    const statusIcon = check.skipped ? '⚠️' : (check.passed ? '✅' : '❌');
+                    moduleSections += `| ${label.name} | ${label.desc} | ${statusIcon} |\n`;
+                  }
+                }
                 moduleSections += '\n';
               }
             } catch (e) {

@@ -45,7 +45,7 @@ There are two ways to add skills to this project:
 
 **Direct Contribution** — Skills are added directly to this repository, inside an existing pack. The contributor opens a PR, skills are reviewed and merged, and maintainers own them from that point. Use `/agentic-contribution-skill` in Claude Code or follow [CONTRIBUTING.md](CONTRIBUTING.md).
 
-**Federation** — An external repository containing a complete, independent Lola pack is referenced in our catalog. The code stays in the external repo; users install it directly via Lola. The external owner maintains their pack. To request federation, open a PR adding the module to `marketplace/rh-agentic-collection.yml` with the `federation` label. Maintainers evaluate the pack using the [Federation Review Guide](docs/FEDERATION_REVIEW_GUIDE.md) and the `/federation-review` skill.
+**Federation** — An external repository containing a complete, independent Lola pack is referenced in our catalog. The code stays in the external repo; users install it directly via Lola. The external owner maintains their pack. To request federation, open a PR adding the module to `marketplace/rh-agentic-collection.yml` with the `federation` label. Maintainers evaluate the pack using the [Federation Review Guide](FEDERATION_REVIEW_GUIDE.md) and the `/federation-review` skill.
 
 ## Working with Agentic Collections
 

@@ -73,7 +73,7 @@ Both tiers must pass before submitting a PR.
 ## Resources
 
 - [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) -- Design principles and templates
-- [Federation Review Guide](docs/FEDERATION_REVIEW_GUIDE.md) -- How external packs are evaluated for federation
+- [Federation Review Guide](FEDERATION_REVIEW_GUIDE.md) -- How external packs are evaluated for federation
 - [agentskills.io specification](https://agentskills.io/specification) -- Base skill standard
 - [Documentation site](https://rhecosystemappeng.github.io/agentic-collections) -- Browse all collections
 - [GitHub Issues](https://github.com/RHEcosystemAppEng/agentic-collections/issues) -- Report bugs or ask questions

@@ -20,23 +20,37 @@ CI also runs automated validation on any PR with the `federation` label (see [CI
 Steps 1–6 can be run with a single command from the agentic-collections repo root:
 
 ```bash
-# Full pack at repo root (default branch)
-uv run python scripts/validate_federation.py <repo-url>
-
-# At a specific ref
-uv run python scripts/validate_federation.py <repo-url> --ref <ref>
+# Full pack at repo root, pinned to a commit SHA
+uv run python scripts/validate_federation.py <repo-url> --ref <40-character-commit-sha>
 
 # Pack in a subdirectory
-uv run python scripts/validate_federation.py <repo-url> --pack-path <path>
+uv run python scripts/validate_federation.py <repo-url> --ref <commit-sha> --pack-path <path>
 
 # Only specific skills
-uv run python scripts/validate_federation.py <repo-url> --skills <skill1> <skill2>
+uv run python scripts/validate_federation.py <repo-url> --ref <commit-sha> --skills <skill1> <skill2>
 
 # JSON output (for CI)
-uv run python scripts/validate_federation.py <repo-url> --json
+uv run python scripts/validate_federation.py <repo-url> --ref <commit-sha> --json
 ```
 
-The script checks: clone access, Lola module schema, Tier 1, Tier 2, MCP version pinning, and gitleaks. If all pass, the pack is ready for manual review (steps 7–8).
+The script checks: commit SHA format, clone access at the pinned commit, Lola module schema, Tier 1, Tier 2, MCP version pinning, and gitleaks.
+
+### Catalog cross-check (in-repo federation artifacts)
+
+When the PR includes `federation/modules/<name>/.catalog/`, verify the catalog matches the external pack at `ref`:
+
+```bash
+uv run python scripts/validate_federation_catalog.py \
+  --module-name <name> \
+  --repo-url <repo-url> \
+  --ref <commit-sha> \
+  --pack-path <path> \
+  --module-json '<marketplace-module-json>'
+```
+
+This checks collection compliance, **skill roster parity** with the linked repo at `ref`, `docs/plugins.json` title alignment, and marketplace version/description/repository consistency.
+
+If all pass, the pack is ready for manual review (steps 7–8).
 
 ---
 
@@ -45,14 +59,14 @@ The script checks: clone access, Lola module schema, Tier 1, Tier 2, MCP version
 ### Step 1: Verify access and basic info
 
 - [ ] Repository URL is reachable and public
-- [ ] If a specific ref (SHA or tag) is declared, it exists: `git ls-remote <repo-url> <ref>`
+- [ ] `ref` is a 40-character commit SHA (not a branch or tag) and exists: `git ls-remote <repo-url> <commit-sha>`
 - [ ] Owner/contact information is provided
 - [ ] License file exists in the repo and is compatible with Apache 2.0 (e.g., Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause)
 
 ```bash
 git clone --no-checkout <repo-url> /tmp/federation-review
 cd /tmp/federation-review
-git checkout <ref>
+git checkout <commit-sha>
 ```
 
 ### Step 2: Verify Lola module schema
@@ -65,6 +79,7 @@ The module entry in `marketplace/rh-agentic-collection.yml` must have the requir
 | `description` | Yes | Brief description |
 | `version` | Yes | Module version |
 | `repository` | Yes | Git URL to the external repo |
+| `ref` | Yes (project extension) | 40-character commit SHA pinning the pack (not a Lola field; required by this repo) |
 
 If the request is for a **subset of skills** (not the full pack), verify only the listed skill paths exist.
 
@@ -129,7 +144,7 @@ Verify the declared AI agent compatibility from the issue:
 | Minor issues | Comment on issue with specific fixes, label `federation/changes-requested` |
 | Major issues | Reject with explanation, label `federation/rejected`, close issue |
 
-When approving, add the pack as a new entry in `modules` in `marketplace/rh-agentic-collection.yml` (include the `license` field from the issue, and use the external repository URL) and link back to the issue.
+When approving, add the pack as a new entry in `modules` in `marketplace/rh-agentic-collection.yml` (use the external repository URL; license compatibility is verified from the repo's LICENSE file during review) and link back to the issue.
 
 ---
 
@@ -150,14 +165,17 @@ The label-based trigger ensures validation only runs on federation PRs, not on e
 After merging a federation PR, verify the module is visible in the marketplace:
 
 ```bash
+# Use a unique ephemeral market name for each verification run
+MARKET="federation-review-$(openssl rand -hex 4)"
+
 # Add the marketplace (use the raw YAML URL for a specific branch or main)
-lola market add test-federation https://raw.githubusercontent.com/RHEcosystemAppEng/agentic-collections/main/marketplace/rh-agentic-collection.yml
+lola market add "$MARKET" https://raw.githubusercontent.com/RHEcosystemAppEng/agentic-collections/main/marketplace/rh-agentic-collection.yml
 
 # List modules — the federated pack should appear alongside internal packs
-lola market ls test-federation
+lola market ls "$MARKET"
 
 # Clean up when done
-lola market rm test-federation
+lola market rm "$MARKET"
 ```
 
 To test a PR branch before merging, replace `main` with the branch name in the URL.
@@ -177,6 +195,7 @@ rm -rf /tmp/federation-review
 | Public access | Yes | Yes | `git ls-remote` |
 | License compatibility | Yes | No | Manual review |
 | Lola module schema | Yes | Yes | `scripts/validate_federation.py` |
+| Federation catalog cross-check | Yes | Yes | `scripts/validate_federation_catalog.py` |
 | Tier 1 (agentskills.io) | Yes | Yes | `scripts/validate_federation.py` |
 | Tier 2 (design principles) | Yes | Yes | `scripts/validate_federation.py` |
 | MCP version pinning | Yes | Yes | `scripts/validate_federation.py` |