Add action to generate and upload MCP-Eval badges from JSON report

[StreetLamb]

Summary

• Add a badge generation action and update workflow to use it.

Checklist

• Tests added/updated
• Docs updated (README, docs/*.mdx)
• Lint passes locally
• Linked issue (if any)

Screenshots / Logs

If applicable, add screenshots or logs.

Breaking Changes

• Yes
• No

If yes, describe the migration path.

[saqadri]

Nice! How do we get the badge after running the action? Is there a way to have some link in a README that points to the badge, which would automatically be updated by the GitHub action?

[StreetLamb]

@saqadri In our workflow we can do something like:

...
- name: Generate badges
  uses: ./.github/actions/mcp-eval/badges
  with:
    report-path: ${{ steps.mcpeval.outputs.results-json-path }}
    output-dir: badges
    upload-artifacts: 'false'
# Update README badges to point to the correct paths
- name: Update README badges
  run: |
    # Ensure badges exist in README
    if ! grep -q "!\[mcp-tests\]" README.md; then
      # Add badges at the top of README if they don't exist
      sed -i '1a\\n![mcp-tests](badges/tests.svg)\n![mcp-cov](badges/coverage.svg)\n' README.md
    fi
# Commit badge files
- name: Commit changes
  run: |
    git config user.name "github-actions[bot]"
    git config user.email "github-actions[bot]@users.noreply.github.com"
    git add badges/*.svg README.md || true
    # Only commit if there are changes
    if ! git diff --cached --quiet; then
      git commit -m "chore(mcpeval): update badges"
      git push
    fi

1. Badge generation action creates the badges in the output directory.
2. Ensure readme reference the badges.
3. Commits the updated badge svg files to the repo

[saqadri]

@saqadri In our workflow we can do something like:

...
- name: Generate badges
  uses: ./.github/actions/mcp-eval/badges
  with:
    report-path: ${{ steps.mcpeval.outputs.results-json-path }}
    output-dir: badges
    upload-artifacts: 'false'
# Update README badges to point to the correct paths
- name: Update README badges
  run: |
    # Ensure badges exist in README
    if ! grep -q "!\[mcp-tests\]" README.md; then
      # Add badges at the top of README if they don't exist
      sed -i '1a\\n![mcp-tests](badges/tests.svg)\n![mcp-cov](badges/coverage.svg)\n' README.md
    fi
# Commit badge files
- name: Commit changes
  run: |
    git config user.name "github-actions[bot]"
    git config user.email "github-actions[bot]@users.noreply.github.com"
    git add badges/*.svg README.md || true
    # Only commit if there are changes
    if ! git diff --cached --quiet; then
      git commit -m "chore(mcpeval): update badges"
      git push
    fi

1. Badge generation action creates the badges in the output directory.
2. Ensure readme reference the badges.
3. Commits the updated badge svg files to the repo

@StreetLamb messaged async as well but in interest of building in the open -- we should follow the standard practice for badge generation to make them dynamic:
https://chatgpt.com/share/68b889f1-8df8-8011-8288-0539eca0fdb7

That would make it much simpler to have developers use them.
If we upload things to GH pages it makes it easier to see reports as well.

[saqadri]

Looks great! Can you share what this would look like used by someone? i.e. what URL they should set in their badge to have it show up in a readme or some other place?

[StreetLamb]

@saqadri Updated the ci/cd page to include info on how to create the workflow and use the badges. The mcpeval.yml workflow will trigger once this PR is merged which should start the test and update our github pages.

[saqadri]

@StreetLamb awesome! Let's merge it and give it a spin

[StreetLamb]

@saqadri Workflow working as expected!

[saqadri]

Awesome work @StreetLamb !