What happened
On PR #30, the review agent applied generic code-quality criteria to proof-of-concept experiment code. It produced some findings that were noise for this context (naming convention for _example suffix, "no linked issue" warning) while missing the most impactful concerns (methodology soundness of scorers, statistical validity of baselines). The review run log shows the agent was given no experiment-specific review guidance — the repo's AGENTS.md covers experiment conventions (numbering, frontmatter, linting) but contains no guidance about what to prioritize during code review of experiment/PoC code.
What could go better
The review agent treated this PoC experiment code the same as production code, spending attention on naming conventions and production-grade error handling while missing design-level methodology concerns that matter most for experiments. Experiment repos have different review priorities than production repos: methodology soundness, hypothesis validity, and reproducibility matter more than production-grade error handling or naming conventions. The existing AGENTS.md in fullsend-ai/experiments does not communicate these priorities to the review agent. Confidence is moderate: the current AGENTS.md may not have existed at the time of the June 9 review (the run log noted injecting an org-level AGENTS.md), but it exists now and still lacks review-specific guidance.
Proposed change
Add a review-specific section to fullsend-ai/experiments/AGENTS.md that guides the review agent's priorities for experiment code. Suggested content:
### Review priorities for experiment code
Experiments are proof-of-concept code. When reviewing, prioritize:
1. **Methodology soundness** — Do metrics, scorers, and evaluation criteria measure what they claim? Are statistical assumptions valid?
2. **Hypothesis alignment** — Does the implementation actually test the stated hypothesis?
3. **Reproducibility** — Can someone else run this experiment from the README instructions?
4. **Security** — No hardcoded secrets, credentials via env vars, sensitive data excluded via .gitignore.
Deprioritize for experiments:
- Production-grade error handling (try/except on every call)
- Naming conventions and code style beyond readability
- Missing linked issues (experiments may not use issue-driven workflows)
This is complementary to fullsend-ai/fullsend#1722 (repo-level review profiles), which is a platform feature. This proposal uses the existing AGENTS.md mechanism available today.
Validation criteria
After updating AGENTS.md, the next 2 review agent runs on experiment PRs in this repo should: (1) not flag naming conventions or missing linked issues as findings, and (2) include at least one methodology-level or hypothesis-alignment observation in their review output. Verify by comparing the review output against the updated guidance priorities.
Generated by retro agent from #30
What happened
On PR #30, the review agent applied generic code-quality criteria to proof-of-concept experiment code. It produced some findings that were noise for this context (naming convention for
_examplesuffix, "no linked issue" warning) while missing the most impactful concerns (methodology soundness of scorers, statistical validity of baselines). The review run log shows the agent was given no experiment-specific review guidance — the repo's AGENTS.md covers experiment conventions (numbering, frontmatter, linting) but contains no guidance about what to prioritize during code review of experiment/PoC code.What could go better
The review agent treated this PoC experiment code the same as production code, spending attention on naming conventions and production-grade error handling while missing design-level methodology concerns that matter most for experiments. Experiment repos have different review priorities than production repos: methodology soundness, hypothesis validity, and reproducibility matter more than production-grade error handling or naming conventions. The existing AGENTS.md in fullsend-ai/experiments does not communicate these priorities to the review agent. Confidence is moderate: the current AGENTS.md may not have existed at the time of the June 9 review (the run log noted injecting an org-level AGENTS.md), but it exists now and still lacks review-specific guidance.
Proposed change
Add a review-specific section to
fullsend-ai/experiments/AGENTS.mdthat guides the review agent's priorities for experiment code. Suggested content:This is complementary to fullsend-ai/fullsend#1722 (repo-level review profiles), which is a platform feature. This proposal uses the existing AGENTS.md mechanism available today.
Validation criteria
After updating AGENTS.md, the next 2 review agent runs on experiment PRs in this repo should: (1) not flag naming conventions or missing linked issues as findings, and (2) include at least one methodology-level or hypothesis-alignment observation in their review output. Verify by comparing the review output against the updated guidance priorities.
Generated by retro agent from #30