[Feedback] Codex vs. Codex + InferMatrix on vllm-omni #5559: preserve coverage, reduce latency

[hsliuustc0106]

Summary

A comparison on vllm-project/vllm-omni#5559 suggests that InferMatrixCopilot adds valuable, project-specific defect coverage, but the current review experience pays too much latency before returning anything useful.

The product direction I would recommend is:

• keep a fast Codex merge/CI/diff pass;
• inject only the owner rules triggered by the changed files into that same pass;
• run targeted InferMatrix specialists only for uncovered high-risk contracts;
• publish a preliminary verdict early, then finish the deeper audit;
• pin the reviewed SHA and revalidate only the delta if the PR changes.

This is related to the usage discussion in #4, but focuses specifically on review quality, latency, freshness, and measurement.

Important comparison caveat

This was a directional case study, not a controlled A/B benchmark:

• InferMatrix-guided review inspected 85d7463f and reported 7 findings.
• Codex GitHub review inspected the later e85c8a90 and reported 5 findings.
• The broken argparse test found by Codex was introduced after the InferMatrix snapshot, so it must not be counted as an InferMatrix miss.

Future comparisons should freeze the same head SHA, base SHA, model, environment, and time budget.

Observed quality comparison

Result	Codex alone	Codex + InferMatrix
Findings reported	5	7
Shared root causes	2	2
Unique findings	3	5
Historical union		10 distinct root causes

Shared root causes

1. Tokenizer validation was not checkpoint/topology aware.
2. The advertised server/offline shared prompt seam was not actually shared.

Codex-only strengths

1. Current-main merge/rebase conflict and preservation of upstream integration.
2. A newly added argparse test exits before reaching its assertion.
3. The CLIP score test has no pass/fail assertion, although this one is more dependent on the intended test contract.

These findings show strength in repository state, localized test behavior, formatting, CI, and merge readiness.

InferMatrix-only strengths

1. DiT-only topology incorrectly activates an AR hook.
2. Tokenizer loading silently falls back.
3. The T2I default changes from think mode to plain mode.
4. One-sided height/width input is accepted despite a paired-dimension contract.
5. The advertised negative_prompt input is ineffective.

These findings show complementary strength in model topology, defaults, fallbacks, public behavior, and producer-to-consumer contracts spanning registry, tokenizer, serving, scripts, and model code.

Current performance signal

Default Direct mode does not run a second model; it supplies knowledge to the host Codex review. The extra wait therefore comes mainly from a broader knowledge-driven investigation, repeated evidence gathering, and late aggregation.

The repository's 20-PR autonomous benchmark artifacts are useful proxies, although they are not measurements of this exact #5559 session:

Metric	copilot_v4_pr20_r2	copilot_v4_pr20_r3
Mean wall time	6.15 min	6.64 min
Median wall time	5.94 min	6.76 min
Mean time to first result	369 s	398 s
Mean LLM calls	29.1	31.0
Mean tool calls	52.2	54.8

In these runs, roughly 99% of elapsed time is in the review stage, and the first result arrives almost at the end. Fetch and deterministic gates are not the main bottleneck.

Sources: r2, r3, and eval/ANALYSIS.md.

Pros and cons

Codex alone

Pros

• Faster path to a current merge verdict.
• Strong mergeability, CI, test, formatting, and local-diff checks.
• Lower risk that the PR head changes before the report is ready.

Cons

• Can miss model-specific topology, fallback, default, and cross-entrypoint contracts.
• Less systematic coverage of online/offline parity and architecture-wide behavior.

Codex + InferMatrixCopilot

Pros

• Higher incremental defect yield in this case study.
• Adds owner-specific runtime and semantic checks rather than only duplicating generic review.
• Particularly useful for registries, model topology, multimodal dimensions, prompt defaults, serving/offline seams, and fallbacks.

Cons

• Longer wall-clock time and almost no progressive visibility.
• Duplicate reading and verification across lenses.
• Greater risk of publishing a stale report after the head changes.
• Environment/setup failures can consume time without increasing defect yield.

Prioritized improvement direction

P0: Make it one review, not two

• Resolve the changed files once.
• Route them directly to exact owner/model rule groups.
• Inject a compact checklist into the first Codex review.
• Reuse files, caller searches, tests, and findings already gathered by Codex.
• Ask the deeper pass only for novel, contradictory, or uncovered high-risk issues.

P0: Pin an immutable review snapshot

• Resolve {base_sha, head_sha, updated_at} before collecting evidence.
• Compute evidence from those exact SHAs.
• Key caches by repository, PR, head SHA, and rule version.
• Recheck the head before publishing.
• If it changed, review old_head..new_head and invalidate only affected findings instead of restarting or publishing stale results.

P0: Return progressive results

• Publish mergeability, CI state, and early P1/P2 findings within 60 seconds.
• Continue targeted specialists in the background.
• Clearly label preliminary versus verified findings.

P1: Use adaptive specialist dispatch

• Start with one knowledge-aware pass.
• Activate behavior/contracts/verification specialists only when deterministic diff signals require them.
• For #5559, target Hunyuan topology, tokenizer/checkpoint validation, prompt defaults, paired dimensions, and serving/offline parity instead of sending the full diff to every generic lens.
• Partition evidence by specialist.
• Retry at most one uncovered high-risk zero-yield lens after all specialists finish.

P1: Reduce orchestration overhead

• Batch GitHub metadata and knowledge reads.
• Execute independent read-only tool calls concurrently.
• Share file, grep, caller, test, repo-map, and knowledge-route caches across the whole review.
• Replace recursive grep with bounded rg where available.
• Use one cache-warmup window for sibling lenses rather than a 0/8/16/24s staircase.
• Invoke the reducer only when candidates conflict or semantically overlap.

P1: Make verification predictable

• Maintain a dependency-matched environment keyed by target SHA.
• Run a short compatibility preflight before pytest.
• Run targeted tests and cheap static checks concurrently with model review.

P2: Instrument the user-visible timeline

Record:

request_received → queue_start/end → snapshot → knowledge_route → each lens start/first response/first candidate/end → reducer → freshness_check → report_ready → host_delivered

Expose per-stage wall time, LLM calls, tool time, tokens, cache-hit ratio, reviewed SHA, and time to first actionable candidate. The current time_to_first_result is effectively end-of-review and does not explain where the user waited.

Suggested product modes

• Fast: deterministic gates plus Codex merge/CI/diff review.
• Standard (default): one knowledge-routed Codex review plus conditional specialists.
• Audit: full ensemble, explicitly requested or run asynchronously/nightly.

Proposed acceptance targets

• First actionable finding in ≤60 seconds.
• Standard-review p50 ≤2–3 minutes.
• Full-audit p50 ≤4 minutes.
• Mean LLM calls ≤12 for Standard mode.
• Mean tool calls ≤25 for Standard mode.
• Zero reports published against a stale head SHA.
• No regression in accepted P1 recall.
• Retain at least 90% of severity-weighted valid findings from the full audit.

These are targets to validate, not currently measured results.

Recommended controlled experiment

Run at least 20–30 PRs across small/large and low/high-risk changes using:

1. Codex fast pass.
2. Codex plus targeted InferMatrix Standard mode.
3. Full InferMatrix Audit mode.

For every arm, freeze the same SHA/base/model/environment/time budget and have maintainers blindly adjudicate deduplicated root causes for validity, severity, and actionability.

Primary metric:

incremental defect yield =
  severity-weighted valid findings unique to InferMatrix
  / additional wall-clock minutes

Also track time to first valid P1, time to usable verdict, precision, overlap rate, stale-head rate, cost per accepted finding, and human adjudication minutes.

Candidate quick wins to validate first

A local prototype suggests two low-risk changes worth A/B testing:

1. Changed-file routing plus batched knowledge reads can route Hunyuan changes directly to the repository, diffusion, and Hunyuan owner rules without walking generic indexes.
2. Giving lens 0 one cache-warmup window and launching the remaining siblings together removes 16 seconds of deliberate delay from the default four-lens path without changing prompts, call topology, budgets, or reducer behavior.

These should be benchmarked on a frozen workload before being treated as production performance gains.

[zuiho-kai]

1. direct路径的 review 性能优化了一点