experiment: Agent evaluation via MLflow + OpenTelemetry

[ascerra]

Summary

• Adds agent-eval-mlflow-otel/ experiment validating MLflow 3.x + OTLP as a complete eval platform for autonomous AI agents
• Includes simplified example scripts: mechanical scorers, LLM-as-judge scorers, trace export, prompt registration, regression detection
• All 5 hypotheses validated: trace capture, scoring, PR gates, regression detection, prompt versioning

Contents

File	Purpose
README.md	Full experiment write-up with architecture, results, and analysis
examples/scorer_mechanical.py	5 pure-Python scorers (validation, cost, efficiency, confidence, iterations)
examples/scorer_llm_judge.py	4 Claude Opus semantic quality scorers via Vertex AI
examples/run_eval.py	Score traces via mlflow.genai.evaluate()
examples/check_regression.py	Compare recent traces against golden baselines
examples/register_prompts.py	MLflow Prompts Registry with @staging/@production aliases
examples/send_trace_example.py	Minimal OTLP trace export to MLflow
examples/harness-explore.yaml	Example harness config with eval section
fixtures/	Example fixture input and LLM judge rubric

Security

• No hardcoded secrets — all credentials via environment variables
• No internal IPs or hostnames
• .gitignore covers .env, venv/, results/, output/

Made with Cursor

[fullsend-ai-review]

🤖 Review · Started 5:25 PM UTC
Commit: ba204cb · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:28 PM UTC · Completed 5:43 PM UTC
Commit: ba204cb · View workflow run →

[fullsend-ai-review]

Review

Findings

Medium

• [error-handling] agent-eval-mlflow-otel/examples/scorer_llm_judge.py:34 — _llm_judge() parses LLM response as JSON with no error handling. If the judge model returns malformed JSON or an unexpected structure (e.g., missing score key), json.loads raises JSONDecodeError and crashes the entire evaluation run. The markdown-fence stripping logic is also fragile.
  Remediation: Wrap json.loads(content) in a try/except and return a fallback Feedback (e.g., score=0 with rationale indicating parse failure). Validate the returned dict contains the expected score key before accessing it.

• [error-handling] agent-eval-mlflow-otel/examples/scorer_mechanical.py:28 — In tool_efficiency, int() cast on get_attribute values may raise ValueError if the attribute is a non-numeric string. The or 0 fallback only handles None/falsy, not arbitrary strings.
  Remediation: Use try/except around the int() casts, e.g., try: tools = int(...) except (ValueError, TypeError): tools = 0.

• [api-contract] agent-eval-mlflow-otel/examples/run_eval.py:93 — mlflow.log_param and mlflow.log_metrics are called after mlflow.genai.evaluate() returns without an active MLflow run context. If evaluate() manages its own internal run, these calls will fail with MlflowException.
  Remediation: Wrap the evaluation and logging in a with mlflow.start_run(): block.

Low

• [logic-error] agent-eval-mlflow-otel/examples/check_regression.py:80 — The regressions list is always empty (comparison logic is commented out). The script prints "To complete: fetch recent traces..." acknowledging the stub, but still reports "All scorers within threshold" which could be misleading if used in CI without reading the output carefully.

• [prompt-injection] agent-eval-mlflow-otel/examples/scorer_llm_judge.py:72 — _get_trace_summary() interpolates trace data (reasoning text, agent name) directly into LLM judge prompts without delimiting. An adversarial trace could influence scoring. Risk is low since this is an internal evaluation tool scoring the team's own agent traces.

• [credential-handling] agent-eval-mlflow-otel/examples/check_regression.py:35 — connect() hardcodes admin as default MLflow username via setdefault. Could lead to unintended admin-level access if MLFLOW_TRACKING_USERNAME is unset while MLFLOW_OTLP_TOKEN is set.

• [edge-case] agent-eval-mlflow-otel/examples/register_prompts.py:68 — client.search_prompt_versions() may raise RestException for non-existent prompt names rather than returning an empty list. First-time registration could fail.

• [logic-error] agent-eval-mlflow-otel/examples/harness-explore.yaml:22 — iteration_count returns a raw count (e.g., 3) not a normalized 0–1 score. Gating logic like min_quality_score: 3.0 would interact confusingly with unnormalized values in metrics aggregation.

• [naming-convention] agent-eval-mlflow-otel/examples/send_trace_example.py — The _example suffix is redundant when the file is already in the examples/ directory.

• [missing-authorization] agent-eval-mlflow-otel/README.md — This PR adds 12 new files with no linked issue. For an experiments repo this is a minor process gap — the thorough README provides sufficient context — but linking to an authorizing issue improves traceability.

Info

• [secrets-handling] .gitignore correctly excludes .env, venv/, results/, output/. All credentials loaded from environment variables. No hardcoded secrets found.

• [scope-alignment] README clearly states production versions live at fullsend-ai/features and these are simplified standalone excerpts. Scope is well-documented.

• [architectural-coherence] Post-hoc trace export design (avoiding coupling agents to observability libraries) is architecturally sound and well-justified.

[maruiz93]

Retracted — posted prematurely, will follow up with a proper inline review.

[maruiz93]

This is a PoC so my reviews aren't blockers but I think at least those could be noticed somewhere in the readme

[ascerra]

@maruiz93 Thanks for the review — folded the three methodology notes into the experiment README (and fixed the gate scale mismatch) in 66dd0e7:

• Mechanical scorer caveats (cost_within_budget binary ceiling; confidence_coherence on LLM self-report) under design decisions + a new Methodology notes (from review) section
• min_quality_score: 3.0 → 0.6 in harness-explore.yaml so gates match 0–1 scorer Feedback
• Golden baselines: prefer fixture-controlled inputs for evaluation; prod means for monitoring/trends only

Replied on each inline thread as well.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:19 PM UTC · Completed 8:33 PM UTC
Commit: 66dd0e7 · View workflow run →

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 8:22 PM UTC · Completed 8:31 PM UTC
Commit: 66dd0e7 · View workflow run →

[fullsend-ai-retro]

Retro: PR #30 — Agent evaluation via MLflow + OpenTelemetry

Timeline

1. June 9: ascerra opens PR experiment: Agent evaluation via MLflow + OpenTelemetry #30 (12 new files, 1,176 additions) — a PoC experiment adding MLflow 3.x + OTLP evaluation infrastructure. No linked issue. Co-authored with Cursor.
2. June 9 (17:23–17:43): Review agent runs (run 27223655310 cancelled after ~2 min, run 27223800207 succeeds in ~14 min). Posts 3 medium, 7 low, and 3 info findings. Applies requires-manual-review label.
3. June 16: Human reviewer maruiz93 posts 3 inline review comments — all methodology-level concerns the agent missed entirely.
4. July 20: ascerra addresses feedback in commit 66dd0e7 (documentation caveats, gate scale fix 3.0→0.6, fixture-based baselines guidance). PR merged without formal GitHub approval.

Review quality analysis

Agent strengths: Found real crash-path issues (json.loads without try/except, int() cast on potentially non-numeric values, MLflow API calls outside active run context) and a prompt-injection surface. One finding (scale mismatch in harness-explore.yaml) overlapped with the human reviewer's findings.

Agent gap — methodology/design-level critique: The human reviewer identified three findings the agent missed entirely, all at the methodology level rather than the code-path level:

• cost_within_budget uses a binary pass/fail ceiling rather than deviation-from-baseline measurement
• confidence_coherence puts non-deterministic LLM self-reported data under "mechanical" scoring, compounding non-determinism into baselines
• Golden baselines from heterogeneous production traces create noisy regression signals; fixture-driven baselines are needed

The agent operates at the code-path level (crash paths, missing error handling, API contract violations). The human operates at the design level (is this metric measuring what it claims? is this statistical approach sound?). This abstraction-level gap is the primary review quality finding.

Agent noise: The naming-convention finding (_example suffix) and "no linked issue" warning are low-value for a PoC experiment repo.

Notes on existing issues

• Review agent should assess feature-level design, not just implementation fullsend#1469 ("Review agent should assess feature-level design, not just implementation") is related to Proposal 1 below but addresses feature design broadly. This retro provides specific evidence and a narrower scope (methodology/statistical-soundness critique for scoring/evaluation code).
• Review agent: support repo-level review profiles to calibrate severity for experiment repos fullsend#1722 ("Review agent: support repo-level review profiles to calibrate severity for experiment repos") is a platform feature request. Proposal 2 below is a complementary, immediate-action alternative using existing AGENTS.md guidance.
• Calibrate 'no linked issue' finding severity for repos without issue-tracking conventions fullsend#5008 ("Calibrate 'no linked issue' finding severity for repos without issue-tracking conventions") — this PR provides another data point: the experiments repo uses a standalone-PR workflow for experiments, making the "no linked issue" finding consistently low-value.

Autonomy readiness

The review agent is not ready for increased autonomy on this class of change (evaluation/scoring code). It missed the three highest-impact findings — all methodology-level. The requires-manual-review label was appropriate. The agent performs well on code-path correctness checks, which may support autonomy for simpler change types (documentation, config) in this repo.

Proposals filed

• Review agent should add methodology/design-soundness critique for evaluation and scoring code (in fullsend-ai/fullsend)
• Add experiment-specific review guidance to AGENTS.md (in fullsend-ai/experiments)

[fullsend-ai-review]

Review skipped — this PR is already merged.

The /fs-review command only reviews open pull requests.

_{Posted by fullsend post-review check}