We Benchmarked 13 LLMs on Quant Trading: Which Capabilities Are Reliable, and Which You Still Can't Trust
After 987 real API calls, we wanted to answer an engineering question — not "which model is smartest," but "when you build a quant agent, which steps can you hand to an LLM, and which must fall back to code."
Inside a quant agent, an LLM isn't there to chat. It has to compute indicators, recognize K-line structure, execute trading rules, and emit JSON the next stage can parse. It's an execution node in a pipeline, not a chat box.
So the real question is: are these capabilities actually reliable? Which can go into a production path, which are fit only for assistive analysis, and which must be backstopped by traditional code?
To answer that, we picked four tasks that map closely to real quant work, plus JSON compliance running through all of them, and ran a standardized comparison across 13 mainstream LLMs: real BTC/USDT market data, automated ground-truth scoring, 987 API calls.
This isn't a leaderboard. It's a model-selection report for engineers.
If you only have two minutes:
- Claude Fable 5 takes the overall top spot (81.5%) — the first model past 80% — but it has been delisted. As of writing, Fable 5 is no longer publicly available, so treat its score as a "capability ceiling" reference point. The strongest models you can actually use are GPT-5.5 and China's standout, GLM 5.2.
- GPT-5.5 is the strongest on public channels (73.9%) and the only model with 100% JSON compliance throughout. For an agent's hub / data-exchange node, it's the most dependable.
- China's flagship steps up. The freshly released GLM 5.2 lands 3rd overall (72.7%), overtaking Claude Opus 4.8, and ranks 2nd in the field on calculation. With the leader delisted, GLM 5.2 and GPT-5.5 are the two strongest options you can actually deploy. GLM 5.1 and DeepSeek V4 Pro stand out on price-performance.
- "Following rules" and "spotting obvious anomalies" are close to production-ready — T4/T5 averages both exceed 92%, and all 13 models hit 100% constraint adherence.
- "Exact calculation" is finally being cracked by the new flagships (the T1 ceiling jumped from 53.7% to 83.3%) — but "K-line structure analysis" tops out at 52.4% across the entire field. It is not solved.
One line to take away:
LLMs can already reliably execute rules, detect obvious anomalies, and act as the data hub; but reading K-line structure can't be trusted yet; and for exact calculation, unless you use the very latest flagship and hand it the formula, keep using code.
We only wanted real tasks to separate the models, so the methodology answers just three things:
- What we tested: 13 models, 4 task families + JSON compliance.
- What data: real BTC/USDT 1-hour candles (Binance); SMC-structure ground truth is generated by the MobiusQuant API — no manual labeling.
- How we scored: fully automated against ground-truth answers, no subjective judgment.
Each task family maps to a real engineering step:
| Capability | Test task | Engineering scenario |
|---|---|---|
| T1 calculation | SMA / RSI / MACD by hand | indicator computation |
| T2 analysis | swing highs/lows · FVG · Order Block · BOS·CHoCH · EQH·EQL | K-line structure analysis |
| T4+T7 rules | multi-condition signal judgment + constraint adherence | signal generation |
| T5+T6 data | K-line anomaly detection (50 / 200 / 500 candles) | data-quality checks |
| JSON compliance | embedded in T2/T4/T5 | agent data exchange |
The 13 models fall into three tiers: the Big Three frontier models (GPT-5.5 / Claude Opus 4.8 / Gemini 3.1 Pro), the Chinese first tier (six flagships — DeepSeek, Qwen, Kimi, GLM, MiniMax, MiMo), and two later additions — Claude Fable 5 and GLM 5.2.
| Rank | Model | T1 calc | T2 analysis | T4 rules | T5 data | Overall | JSON |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 | 83.3% | 49.1% | 93.8% | 100% | 81.5% | 85.9% |
| 2 | GPT-5.5 | 46.3% | 49.4% | 100% | 100% | 73.9% | 100% |
| 3 | GLM 5.2 | 61.1% | 39.9% | 91.7% | 98.0% | 72.7% | 73.7% |
| 4 | Claude Opus 4.8 | 53.7% | 46.2% | 94.1% | 90.9% | 71.2% | 90.8% |
| 5 | GLM 5.1 | 48.1% | 40.0% | 96.9% | 93.9% | 69.7% | 73.6% |
| 6 | Claude Opus 4.6 | 51.8% | 42.7% | 82.0% | 91.4% | 67.0% | 80.5% |
| 6 | DeepSeek V4 Pro | 40.7% | 44.4% | 85.8% | 97.0% | 67.0% | 100% |
| 8 | Claude Sonnet 4.6 | 44.4% | 41.0% | 88.2% | 93.9% | 66.9% | 79.5% |
| 9 | MiniMax (M2.7/M3) | 27.8% | 45.5% | 93.8% | 100% | 66.8% | 81.8% |
| 10 | Gemini 3.1 Pro | 38.9% | 52.4% | 100% | 66.7% | 64.5% | 84.7% |
| 11 | Kimi K2.6 | 24.1% | 39.2% | 93.8% | 100% | 64.3% | 77.2% |
| 12 | MiMo V2.5 Pro | 16.7% | 47.0% | 90.6% | 90.3% | 61.2% | 96.6% |
| 13 | Qwen3.7 Max | 3.7% | 50.9% | 87.5% | 100% | 60.5% | 88.0% |
About the leader: the overall #1, Claude Fable 5, is delisted and no longer publicly available as of writing. Its score stays as a "current capability ceiling" reference — but you can't actually use it. The strongest deployable models are GPT-5.5 (public channels) and the recently released China standout, GLM 5.2.
The overall score is the macro-average of T1/T2/T4/T5. Fable 5 (via a local proxy) and GLM 5.2 (via Zhipu's first-party API) were added later, on channels that don't exactly match the main cohort's OpenRouter — see Limitations.
The ranking itself isn't the point. The point is the five findings below — they decide how you should build your agent.
A year ago, "indicator calculation is a glaring weakness of every LLM" was almost settled fact: before these additions, the field-wide best was just 53.7% (Claude Opus 4.8), and models routinely fumbled even an SMA ("average these 10 numbers").
The new flagships rewrote that conclusion: Claude Fable 5 pushes T1 to 83.3% (perfect SMA and RSI, 27/27 when given the formula — including the 40-step MACD recursion), and GLM 5.2 follows in 2nd at 61.1%. But note the caveat: only these two new flagships clear 60%; the other 11 models all sit below 54%.
There's a counterintuitive detail worth remembering — SMA trips models up more than RSI does: the field-wide SMA pass rate is just 33.4%, lower than the more involved RSI (62.1%). The reason: SMA forces the model to "locate the last 10 closes out of 30 candles and sum them accurately" — and LLMs are bad at counting and large-number addition. RSI has more steps, but the numbers are small, so it's actually steadier.
And MACD is the real graveyard: a 7.8% field-wide average, with 7 models at flat 0%. Even Fable 5 nails MACD 9/9 with the formula but scores 0/9 without it — the failure is a systematic bias (it defaults to a different EMA-initialization convention), not arithmetic noise. GLM 5.2's MACD is only 38.9%.
Engineering takeaway: keep computing indicators in code — deterministic, free, zero latency. When you must use an LLM, note that with Fable 5 delisted, GLM 5.2 is effectively the only remaining option (61.1%, far ahead of the next available model, Opus 4.8 at 53.7%), and you must put the formula and conventions in the prompt. Never let an LLM compute MACD from scratch.
This is the most important finding in the whole benchmark, because it shows: some capabilities don't yield to more model.
The field-wide best on T2 is Gemini 3.1 Pro at 52.4%. Fable 5, a runaway #1 on calculation, manages only 49.1% (4th) on T2; GLM 5.2 only 39.9%.
Break it down and you see where it jams:
- Swing-high/low identification (find the max/min in a window): ~98% across all models — because it's essentially numeric comparison.
- The four SMC tasks (FVG, Order Block, BOS/CHoCH, EQH/EQL): ~19.5% on average, with four models scoring a flat 0% on Order Block and on BOS.
The most telling comparison is Fable 5: it pushed calculation from 53.7% to 83.3% in one generation, yet structure understanding didn't budge. The leap in raw computational ability brought no matching leap in structural reasoning — inferring spatial structure across many candles is a different order of cognition.
Engineering takeaway: SMC / structure recognition cannot be left to any LLM as the sole decision-maker right now. If you use it, gate it behind human review or a rules-based second check. This is the step that most needs a traditional-quant backstop today.
Here's the good news.
T4 (rule execution) averages 92.0% field-wide, with a floor of 81.9%; GPT-5.5 and Gemini are perfect. More importantly: all 13 models hit 100% constraint adherence — timeframe limits, count limits, priority rules, enum values, prohibitions, no exceptions. In an agent framework, "Skill constraint-following" is already fully reliable.
T5 (anomaly detection) has 5 models at a perfect score, and every model except Gemini clears 90% F1. Anomaly detection is fundamentally "finding the irregular within the regular" — exactly where LLM pattern-matching shines.
The errors cluster at boundary conditions: T4 mistakes come mostly from volatility judgments (Bollinger-band width near the medium/low boundary) and last-candle signals (Close just barely above the BB upper band) — the residual risk in these tasks is uncertainty right around thresholds.
Engineering takeaway: "judge by rules," "flag obvious anomalies," and "run a checklist" are tasks LLMs handle reliably enough to put in a production path — but boundary conditions need test coverage. This is the most solid landing spot for LLMs in quant today.
This one is for the people building agents: a high score ≠ a stable pipeline.
GPT-5.5 and DeepSeek V4 Pro return clean JSON throughout (100%). The whole Claude family (including the latest Fable 5) is stuck at 72.5% — they love to wrap output in ```json ... ```; the content is correct, but json.loads() throws. The GLM line runs lower, often appending extra explanatory prose.
Put that in a pipeline: a model scoring 93.8% on T4 capability but only 72.5% on JSON means roughly 1 in 4 outputs needs extra cleanup. In a chain of agents, that's a steady failure point — one non-clean JSON upstream breaks the whole chain downstream.
Engineering takeaway: when choosing the hub / data-exchange node of an agent, weight JSON compliance above raw single-task capability. Use DeepSeek / GPT-5.5 for the hub; if you must use Claude / GLM, add a JSON-cleaning layer or enable strict output mode.
No model wins across every dimension.
Even the closest, Fable 5 (the only model in the top 5 on all dimensions), is just 49.1% on T2. And rankings reshuffle violently between dimensions: Gemini is tied #1 on T4 but dead last on T5 (collapsing to 0% on 500 candles); Qwen is perfect on T5 but near zero on T1 (3.7%). Every model has a clear strength and a clear weakness.
So the realistic architecture isn't "go all-in on the single strongest model" — it's multi-model collaboration, split by step:
- Indicator calculation → code
- Structure analysis → human / rules backstop
- Rule execution → a strong-JSON model
- Anomaly detection → a low-cost model
- Data hub → the model with the cleanest JSON
Single-model worship is neither cost-effective nor stable in a quant setting.
The findings, distilled into one actionable table:
| Scenario | Pick | Why |
|---|---|---|
| Indicator calculation | Code first; Fable 5 is delisted, so the only LLM option left is GLM 5.2 (and give it the formula) | #1 on calculation among usable models, far ahead of #2 |
| K-line structure analysis | Gemini / Qwen worth trying, but always with human or rules review | top two on T2, yet absolute scores still untrustworthy |
| Trading-rule execution | GPT-5.5 / GLM 5.1 | perfect / near-perfect on T4 |
| Data-anomaly detection | low-cost models like MiniMax / Kimi / Qwen | perfect on T5 at very low per-call cost |
| JSON data hub | DeepSeek / GPT-5.5 | 100% JSON + manageable cost |
| Maximum performance | Fable 5 is delisted; deployable picks are GPT-5.5 / GLM 5.2 | top three overall, but the leader is unavailable |
| Price-performance | GLM 5.1 / DeepSeek V4 Pro | low cost, high score (GLM 5.1: $0.96 → 69.7%) |
On cost, the Chinese first tier runs 1/5 to 1/15 the per-eval price of the Big Three flagships; DeepSeek in particular is the balanced choice — "100% JSON + 67% overall + low cost." The performance ceiling sits with the new flagships, but for everyday pipelines, the price-performance of Chinese models is hard to ignore.
State the boundaries up front, and the conclusions hold up better:
- Only BTC/USDT, 1-hour — a single instrument and timeframe; cross-instrument / cross-timeframe generalization is unverified.
- Some dimensions are under-sampled: in the main cohort, T2/T4/T5 ran a single round per case, so randomness isn't fully averaged out.
- The T5 anomalies are fairly obvious (a 90% price crash, negative volume, etc.); subtler data-quality issues would be harder.
- Fable 5 / GLM 5.2 were added later: the former via a local proxy, the latter via Zhipu's first-party API — channels that don't exactly match the main cohort's OpenRouter. (Both are footnoted in the reports, and Fable 5's calculation result went through a separate five-step audit.)
- A few per-call details are missing: some of Fable 5's raw single-call data was lost during debugging — it doesn't affect the aggregate scores, but it limits call-by-call review.
A year ago, "exact calculation" was treated as a fundamental LLM weakness; the new flagships have now shown it's just an engineering problem that gets solved generation by generation. "K-line structure analysis" — which needs multi-step spatial reasoning — is still waiting for its breakthrough.
One more real-world variable worth noting: the overall #1, Fable 5, has been delisted. That's a sharp reminder of how fragile it is to bet everything on a single closed flagship — a model can become unusable overnight. The recently released China standout GLM 5.2 — controllable via a first-party API, 3rd overall, 2nd on calculation — steps in as the steadier choice. The capability ceiling will keep changing hands, but building your architecture on a controllable, swappable mix of models is what's actually robust in engineering terms.
So the right way to build a quant agent today isn't to ask "which model is strongest" — it's to put each step where it belongs:
Let the LLM do what it's already reliable at — executing rules, detecting anomalies, serving as the data hub; hand what it isn't reliable at — exact calculation, structure analysis — to code and to people.
That's the LLM's real place in quant, in 2026.
Full data and methodology are in the technical report; every ground-truth answer is reproducible automatically.