Added submodule for Alquimia quickstart

[seanmerrow]

Publication suggestion PR

Thank you for suggesting a quickstart for publication on redhat(dot)com! Please complete the sections below fields so reviewers have enough context and leave checkboxes unchecked.

Summary

Publication readiness checklist

• README is clear, concise, and free of typos
• README is accurate and includes vertical use case
• README is complete and follows template structure
• Quickstart runs end-to-end without errors and is reproducible
• Titles, descriptions, and tags adhere to MIST guidelines
• Insert redhat(dot)com requirements here
• Confirm markdown links use accessible "alt text" descriptions
• Technical review complete
• Peer review complete
• Marketing review complete (accurately convey purpose of quickstart ->
  demo, not production-ready code supported by RH)
• Known issues and requests are documented (or resolved)

[github-actions]

📋 Automated Documentation Review

Submodule: quickstart/Evaluate-agents-with-gaussia-evalhub
Source Repo: rh-ai-quickstart/Evaluate-agents-with-gaussia-evalhub
Commit: 2a8fad6fbc952f3050422cc8461a241e2f36bbd8

---

📊 Summary

Severity	Count
🚫 Blockers	0
⚠️ Major	5
ℹ️ Minor	2
💡 Suggestions	1
Total	8

---

⚠️ Major Issues (5)

Trademark symbol used on non-first mentions throughout README

Category: style | Location: Throughout document: Title, Detailed description (multiple instances), Requirements section, Deploy section, and other locations

The trademark symbols ® appear multiple times throughout the document on repeated mentions of 'Guassia®', 'Alquimia®', 'Red Hat OpenShift AI®', and other product names. According to the 'First Mention Rule', trademark symbols should only appear on the first mention of a product name in the document.

Suggested Fix: Use 'Guassia®' and 'Alquimia®' only on first mention in the title or detailed description. All subsequent mentions should be 'Guassia' and 'Alquimia' without the ® symbol. Same applies to 'Red Hat OpenShift AI®' - use the symbol only on first mention, then 'Red Hat OpenShift AI' or 'OpenShift AI' thereafter.

Requirement: Publishing Guidelines > Naming and branding consistency > The 'First Mention' Rule

---

Placeholder text '[Gaussia]' used instead of proper product name

Category: content | Location: Multiple locations: 'This AI quickstart helps platform, product, and model teams measure autonomous agent quality before agent updates reach production. It uses [Gaussia] as the evaluation provider', 'For details on available [Gaussia] metric families', 'The chart registers the [Gaussia] provider', etc.

Throughout the README, the text '[Gaussia]' appears in brackets as if it's a placeholder or markdown link, rather than the proper product name 'Guassia'. This appears in multiple locations including the detailed description, technical details sections, and architecture flow summary.

Suggested Fix: Replace all instances of '[Gaussia]' with 'Guassia' (without brackets). If these are intended to be links, use proper markdown link syntax: 'Guassia'. If they are just product name mentions, remove the brackets entirely.

Requirement: Publishing Guidelines > Naming and branding consistency

---

Broken or incomplete external reference links

Category: content | Location: References section: 'Gaussia documentation' and 'EvalHub provider adapter entrypoint' both link to https://github.com/gaussia-labs/pygaussia

The References section contains links that appear to be placeholders or incomplete. Specifically, both 'Gaussia documentation' and 'EvalHub provider adapter entrypoint' link to 'https://github.com/gaussia-labs/pygaussia' (the same URL), which seems incorrect for two different purposes.

Suggested Fix: Verify and update the reference links to point to the correct documentation. The 'EvalHub provider adapter entrypoint' should likely point to a specific file or subdirectory within the repository, not the repository root. Ensure each link serves its stated purpose.

Requirement: README Quality Standards > BLOCKER criteria > Broken README links (treating as MAJOR since links exist but may be incorrect rather than 404)

---

Missing 'See it in action' section with video/demo links

Category: structure | Location: Missing between 'Detailed description' and 'Architecture diagrams' sections

The README structure requirements indicate that a 'See it in action' section (H3) with links to videos/Arcades is recommended for users without environments. This section is completely absent from the README.

Suggested Fix: Add a '### See it in action' section after the detailed description and before the architecture diagram. Include links to any demo videos, Arcade experiences, or recorded walkthroughs that show the quickstart in action. If none exist yet, consider creating an Arcade demo as recommended in Contributing Guidelines.

Requirement: README structure requirements > Order 5: 'See it in action' (H3, Recommended)

---

Instructions assume undocumented knowledge about judge/guardian models

Category: content | Location: Deploy > Prepare the quickstart project section

The Deploy section references judge and guardian models multiple times and states 'You will need judge and guardian models to run the full suite of benchmarks' but doesn't clearly explain what these are, why they're needed, or what happens if users don't have them. The link to 'Deploy judge and guardian models' helps, but the main flow lacks context.

Suggested Fix: Add a brief explanation before the deployment steps: 'Judge and guardian models are LLMs used by Guassia to evaluate agent conversations for bias, toxicity, and other quality metrics. You can run basic evaluations without them (humanity-only benchmarks), but the full evaluation suite requires these models.' Then proceed with the existing instructions.

Requirement: README Quality Standards > MAJOR criteria > Unclear instructions: Steps work but are confusing, lack context, or assume undocumented knowledge

---

ℹ️ Minor Issues (2)

Inconsistent code block formatting for environment variables

Category: formatting | Location: Deploy > Prepare the quickstart project section, particularly the environment variable table and the 'Model and run metadata' section under Technical details

The README uses inconsistent formatting when showing environment variable examples. Some use inline code (VARIABLE_NAME), others use code blocks with export statements, and the table format mixes both styles.

Suggested Fix: Standardize the format: use inline code for variable names in prose (GAUSSIA_JUDGE_*), use code blocks for actual commands users should run (export VARIABLE=value), and use consistent formatting in tables (either all inline code or all plain text for variable names in the first column).

Requirement: README Quality Standards > MINOR criteria > Minor formatting inconsistencies

---

Missing 'Reference' section as H2 heading

Category: structure | Location: Section titled '## References' (should be '## Reference')

The README has a 'References' section but the structure requirements specify it should be 'Reference' (singular) as an H2 heading. While this is a minor naming inconsistency, following the standard structure improves consistency across quickstarts.

Suggested Fix: Change '## References' to '## Reference' to match the standard structure template.

Requirement: README structure requirements > Order 12: 'Reference' (H2, Recommended)

---

💡 Suggestions (1)

Consider adding troubleshooting guidance for common deployment issues

Category: content | Location: Could be added after the 'Delete' section and before 'References'

Given the complexity of this deployment (multiple components, external model dependencies, environment variables, Helm charts), users would benefit from a dedicated Troubleshooting section. The README links to 'docs/troubleshooting.md' in References, but common issues could be surfaced in the main README.

Suggested Fix: Consider adding a '## Troubleshooting' section with the most common 3-5 issues users encounter (e.g., 'EvalHub pod not starting', 'MLflow connection refused', 'Judge model authentication failed') with quick fixes, then link to the full troubleshooting doc for comprehensive guidance.

Requirement: README Quality Standards > SUGGESTION criteria > Future enhancements or ideas

---

🤖 Automated review powered by Claude

[seanmerrow]

Working on addressing feedback. Will resubmit