🏆 OTC Desk Agent Arena — Run #134
Gemini 3.8 Flash is the more accurate model and finishes behind the one it replaces. The extra spend is not retries and not subagent fan-out: it is five times as much filesystem searching, at the same hit rate, in the same places.
Correction, 2026-09-04. This report originally read the over-execution as a property of the model. It is mostly a property of the effort setting. Run
#138re-ran the same model atlowacross all six workflows: calls fall 53%, objective gives up 1.8 points, and OVR rises from 79 to 86 — a better consolidated card than its predecessor's 83. The findings below stand as a description ofxhigh; they do not support the claim that the newer model is worse. The evidence is in The effort arm, below.
2026-09-03 · run #134 · 1 model × 6 workflows × 2 trials (5 workflows) and 1 trial
(confirmation-desk-day) · pinned to xhigh · output budget unpinned · objective-only
scoring (jury off) · 6 of 6 arms scored, zero truncated calls, zero malformed tool
calls · compared against the gemini-3-7-flash arm of run #127, the same route at
the same effort · control arm: run #135
Why this run exists
Gemini 3.8 Flash was published on 2026-09-01. The desk's previous Gemini contestant, 3.7 Flash, holds an unusual position in the standings: it has posted the highest objective score in the flash tier (94.3 on run #129) while ranking third, because its efficiency is the worst of the leading group. The obvious question for a successor was whether it kept the accuracy and fixed the cost.
It did the opposite of the second half.
Headline
Consolidated card, six workflows, published to /arena/models.html as cards only —
one model has no field, so a rank of #1 of 1 would measure nothing.
| Model | eff | OVR | GRD | ADH | SYN | PRC | EFF | CON | Objective |
|---|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | xhigh | 79 | 91 | 97 | 99 | 95 | 4 | 89 | 95.6 |
| Gemini 3.7 Flash (run #129) | xhigh | 83 | 95 | 92 | 99 | 92 | 31 | 89 | 94.3 |
On the five workflows the two arms share, at the same effort on the same route:
| Gemini 3.8 Flash | Gemini 3.7 Flash | |
|---|---|---|
| Objective score | 97.1 | 94.3 |
| Workflows won on objective | 4 of 5 | 1 of 5 |
| Perfect 100.0 scores | 2 | 0 |
| Tool calls per trial | 95.9 | 52.0 |
| EFF | 4 | 31 |
| Mean OVR | 80.6 | 83.0 |
The newer model is better at the work and ranks lower for it. It wins four of the five workflows on correctness, posts two perfect scores where its predecessor posts none, and spends 84% more tool calls to do it. EFF is scored as golf against each workflow's calibrated par, so that difference is not a rounding adjustment — it is the whole ranking.
| Workflow | 3.8 obj | 3.8 calls | 3.7 obj | 3.7 calls |
|---|---|---|---|---|
| ops-settlement-day | 100.0 | 84.5 | 89.8 | 38.5 |
| risk-limit-breach-day | 100.0 | 45.0 | 98.7 | 29.0 |
| risk-manager-control-day | 97.4 | 92.0 | 93.6 | 58.5 |
| trader-rfq-booking-day | 96.8 | 136.5 | 93.7 | 69.5 |
| high-board-portfolio-review-day | 91.4 | 121.5 | 95.7 | 64.5 |
Every row goes the same way. There is no workflow where 3.8 is both more accurate and cheaper, and none where it is cheaper at all.
Where the calls actually go
The scores say how much more it spends. The transcripts say on what. Counting every tool call in all ten trials across the five shared workflows, then averaging per trial:
| Tool | 3.8 Flash | 3.7 Flash | Delta |
|---|---|---|---|
grep |
64.5 | 14.5 | +50.0 |
ls |
22.5 | 0 | +22.5 |
glob |
23.0 | 6.5 | +16.5 |
read_artifact |
17.5 | 2.5 | +15.0 |
inspect_artifact |
11.5 | 2.5 | +9.0 |
list_artifacts |
10.0 | 3.0 | +7.0 |
Grouping those together against everything else:
| 3.8 Flash | 3.7 Flash | |
|---|---|---|
| Filesystem / search calls | 149 | 29 |
| Domain and quant tool calls | 352 | 253 |
| Search as a share of all calls | 29.7% | 10.3% |
Filesystem search is 55% of the entire increase. The remaining 45% is spread thinly
over two dozen domain tools, none of which moves by more than about ten calls. The single
most striking cell in the table is ls: across ten trials and five workflows, Gemini 3.7
Flash never calls it once, and 3.8 calls it 22 times.
Where it searches is consistent. Ranked by target, 3.8's search calls go to /skills
(53), the filesystem root (25), /large_tool_results (24) and /artifacts (8). Its
predecessor touches /skills 24 times and almost nothing else.
What it is not
Three cheaper explanations are available, and the transcripts rule out all three.
It is not retrying after failures. Gemini 3.8 Flash records fewer errors than its predecessor, 0.3 per workflow against 0.8. Whatever the extra calls are, they are not recovery.
It is not subagent fan-out. Both models dispatch essentially the same number of
task() subagents, 1.5 against 1.4 per workflow. The extra work is happening in the
orchestrator's own loop.
It is not worse searching. This is the one that matters, and it is the one that survives contact with the data least comfortably for the intuitive story. Classifying every search call by what came back:
| 3.8 Flash | 3.7 Flash | |
|---|---|---|
| Returned content | 78% | 79% |
| Returned nothing | 17% | 14% |
| Refused (foreign store) | 5% | 3% |
| Exact repeat of an earlier call | 23% | 21% |
The hit rates are the same. The repeat rates are the same. Both models also concentrate their searching late in an episode rather than front-loading it as orientation — 55% of 3.8's search calls fall in the last third, against 60% for 3.7.
So this is not a model that searches badly, or searches differently, or searches in the wrong places. It searches exactly the way its predecessor does, with the same productivity per call, and does five times as much of it before it stops. The deficit is not in the searching. It is in the stopping.
The confound, and the control that sizes it
There is a problem with everything above, and it is ours rather than the model's.
The 3.7 baseline ran on 2026-08-26. Run #134 ran on 2026-09-02, and in between we
changed the harness. Gemini 3 routes return an encrypted thought signature beside
every tool call and require it echoed back on the assistant turn carrying that call;
ChatOpenAI does not preserve that field, by design, so the second model turn of every
tool loop was being refused with a 400. Run #134 initially failed on all six workflows
before we fixed it. (The control that established this was a harness defect rather than a
new-model limitation: 3.7 Flash, which has four published boards on this route, fails
identically once the field is stripped.)
That fix hands the model back its own prior reasoning. It is entirely plausible that doing so makes a model more willing to keep pulling on a thread — which would mean we had measured our own change and attributed it to Google.
So we re-ran Gemini 3.7 Flash today, with the fix, on ops-settlement-day. That
workflow is the right instrument: 3.7's own trial-to-trial spread there is a single call,
so one trial separates the hypotheses cleanly.
Counting the transcripts, so that total calls and search calls come from one instrument:
| Arm | Tool calls | Search calls | Objective |
|---|---|---|---|
| 3.7 Flash, run #127, without the fix | 45, 46 | 1, 4 | 89.8 |
| 3.7 Flash, run #135, with the fix | 55 | 10 | 90.9 |
| 3.8 Flash, run #134, with the fix | 85, 98 | 29, 43 | 100.0 |
The fix is not inert. It moves 3.7 from 45.5 calls to 55, and its search calls from 2.5 to 10. Had we published the naive comparison, part of what we called a model regression would have been our own patch.
It does not, however, come close to explaining the gap:
| Component | Calls | Share of the gap |
|---|---|---|
| Total gap, 3.8 with fix vs 3.7 without | +46.0 | 100% |
| Attributable to the harness change | +9.5 | 21% |
| Attributable to the model | +36.5 | 79% |
The same decomposition holds for search calls alone (22% harness, 78% model), and the two independent instruments — the trace harvest that feeds EFF, and a direct count of the transcripts — agree on the split to the percentage point.
The conclusion is robust to the correction. EFF is scored on the trace harvest, which
puts 3.8 at 84.5 calls on this workflow. Strip the entire harness component out and it
still runs ops-settlement-day in 75 calls against a calibrated par of 30 — 2.5× par,
which floors EFF just as thoroughly as 2.8× does. No plausible adjustment for the harness
rescues the ranking.
The effort arm: what actually drives the searching
Added 2026-09-04, runs #136/#137.
Everything above holds xhigh fixed and varies the model. The obvious complementary
test is to hold the model fixed and vary the effort — and the arena already contained
half the answer for free. Run #127 and run #130 are Gemini 3.7 Flash on the same five
workflows with effort as the only variable:
3.7 at xhigh |
3.7 at low |
|
|---|---|---|
| Tool calls per workflow | 56.4 | 35.5 |
| Search calls per workflow | 5.8 | 0.5 |
| Search as a share of calls | 10.3% | 1.4% |
Search falls 91%. On three of the five workflows it goes to exactly zero. Whatever the file rummaging is, it is switched on by reasoning effort rather than baked into the model.
So we ran Gemini 3.8 Flash at low across all six workflows (run #138), one trial
each — same model, same route, same harness, same commit, with effort as the only
variable. It is the paired arm for run #134, and it reverses this report's framing.
low (run #138) |
xhigh (run #134) |
|
|---|---|---|
| OVR | 86 | 79 |
| GRD | 91 | 91 |
| ADH | 92 | 97 |
| SYN | 99 | 99 |
| PRC | 94 | 95 |
| EFF | 47 | 4 |
| Objective | 93.8 | 95.6 |
| Tool calls per workflow | 75.3 | 161.6 |
| Search calls per workflow | 26.7 | 51.3 |
Halving the effort halves the work and raises the score. Calls fall 53%, search calls fall 48%, objective gives up 1.8 points — and OVR rises seven, from 79 to 86, because EFF goes from 4 to 47. Grounding is identical at both efforts (91 and 91); the entire correctness cost is 5 points of adherence.
Per workflow, low wins four, ties one and loses one by a single point:
| Workflow | low OVR |
xhigh OVR |
Δ | low obj |
xhigh obj |
|---|---|---|---|---|---|
| ops-settlement-day | 98 | 83 | +15 | 100.0 | 100.0 |
| risk-limit-breach-day | 97 | 84 | +13 | 100.0 | 100.0 |
| high-board-portfolio-review-day | 80 | 72 | +8 | 85.7 | 91.4 |
| trader-rfq-booking-day | 87 | 81 | +6 | 96.8 | 96.8 |
| confirmation-desk-day | 71 | 71 | 0 | 87.9 | 87.9 |
| risk-manager-control-day | 82 | 83 | −1 | 92.3 | 97.4 |
Four of the six score identically on objective at both efforts, including both perfect
100.0s. The accuracy xhigh buys is confined to two workflows —
risk-manager-control-day and high-board-portfolio-review-day — and on the golf curve
it costs far more in calls than it returns.
The effort effect on searching is not uniform, and that is the most useful detail here.
On five workflows filesystem search all but disappears at low. On
confirmation-desk-day it barely moves: 151 search calls at low against 159 at xhigh,
even though total calls halve from 490 to 256. That workflow's rummaging is inherent to
the task, not bought by effort — which is exactly why it is the one workflow where
dropping the effort changes nothing at all, OVR 71 either way.
So the corrected reading of run #134 is simple. xhigh on this model is a
misconfiguration. At the desk default it is not merely competitive but the strongest
consolidated card the flash tier has produced — OVR 86 against the 83 its predecessor
posts at its own ceiling.
Scope. Run #138 is one trial per cell, so CON is not measured rather than
perfect, and single-sample noise is real: ops-settlement-day was measured twice at
low, in run #137 and run #138, and scored 93.2 and then 100.0. Read the per-workflow
rows as indicative and the direction — five of six workflows improving, none worse by
more than a point — as the finding.
Caveats
- The harness control is one workflow and one trial. ZenMux subscription quota ran
out mid-investigation, so the paired 3.7 arm on
trader-rfq-booking-daywas deliberately abandoned to preserve what remained. The 21% figure is measured onops-settlement-dayand is not established as a constant across workflows. It is sized, not solved. (The effort arm, by contrast, now covers all six workflows.) - A comparability break now exists for Gemini contestants, in the same way run #115 created one for Anthropic-protocol models when the output budget was raised. Every Gemini board from #134 onward runs with the thought signature preserved; #113, #127, #129 and #130 did not, and the control says that is worth roughly a fifth of the call count on at least one workflow. Read Gemini EFF across that boundary with care.
confirmation-desk-daycarries one trial, not two. Its second was lost to the same quota exhaustion and recordedinvalid/infra_blankrather than scored, so its CON is not measured, not perfect. That workflow is also excluded from every 3.7 comparison in this report, because 3.7 has no arm on it at pinned effort.- Objective scores are not adjusted for the harness change at all. The correction above applies to call counts. Whether the restored signature also helped 3.8's accuracy is untested, and the 3.7 control moved only 1.1 points on that axis.
- Zero truncation and zero malformed tool calls, measured rather than assumed:
trials_measuredequalstrials_totalon all six arms.
What would change the verdict
- ~~A paired
lowarm.~~ Run across all six workflows (#138), and it changed the verdict — see the effort arm above. Alowcard is now published beside thexhighone. What remains is depth: run #138 is one trial per cell, so it carries no CON and the twolowmeasurements ofops-settlement-daydiffer by 6.8 objective points. A two-trial re-run would turn the direction into a number. - The rest of the harness control. Four more paired 3.7 arms would turn the 21% from a single-workflow measurement into a real coefficient, and would settle whether the effect is uniform or concentrated in workflows with large artifact trees.
- A prompt-level test of the stopping rule. The finding is that the model does not
know when to stop searching, not that it searches badly. If that is right, it should be
correctable from the orchestrator prompt rather than requiring a different model — and
a cheap A/B on
ops-settlement-daywould show it.
Cards for this run are published at /arena/models.html. Nothing here
reaches the leaderboard: a rank is relative and this run had no field.