Run #110 — does reasoning_effort improve arena performance? (gpt-5.6-luna)

Date: 2026-08-17 → 2026-08-18 (single detached process, ~13.5 h) Design: predeclared in 2026-08-17-luna-reasoning-effort-plan.md before launch. One model (gpt-5-6-luna, zenmux), four pinned arms (none / low / high / max), all 5 golden workflows as paired blocks, 2 trials per cell → 40 trials. Board health: 20/20 arms scored, 0 invalid, 0 infra deaths — every predeclared validity rule held; no cell was excluded.

Headline answer

No — "higher effort ⇒ better agent" is not what the data shows. For luna on this arena, reasoning_effort behaves as two different dials at once:

1. Zero → some reasoning is the real jump. Every reasoning arm beats none by ~5 points mean objective (none 88.9 vs low 93.8 / high 93.3 / max 94.8). And low achieves that while being cheaper than none (29.1 calls, 10.2 min/trial vs 30.8 calls, 11.1 min/trial): a little thinking reduces total work by cutting flailing.
2. Above low, quality is flat but cost explodes. max buys +1.0 objective point over low for 3.3× wall-clock (33.8 vs 10.2 min/trial) and +50% tool calls (43.9 vs 29.1). high buys nothing over low (93.3 vs 93.8). Within-ladder differences are inside per-cell trial noise (stdev 1–4); the cost differences are not.

By mean OVR (which prices efficiency in), low is the best operating point: 86.9, vs 81.8 (none), 81.6 (high), 82.7 (max).

The board

workflow	none	low	high	max
risk-manager-control-day (flagship)	94.8	93.6	91.1	91.0
high-board-portfolio-review-day	71.4	84.3	78.6	90.0
trader-rfq-booking-day	94.5	93.7	96.8	96.8
risk-limit-breach-day	97.4	97.3	100.0	96.1
ops-settlement-day	86.3	100.0	100.0	100.0
mean objective	88.9	93.8	93.3	94.8
mean OVR (card)	81.8	86.9	81.6	82.7
mean calls/trial	30.8	29.1	38.6	43.9
mean wall min/trial	11.1	10.2	14.2	33.8
mean trial stdev	3.98	1.06	2.20	1.72

Effort's value depends on the workflow's shape

• Procedural, well-routed loops (flagship): effort hurts. Monotone decline 94.8 → 91.0. At max, luna skipped reading the read-risk-result skill doc entirely (0/2 vs 2/2 at none) — more internal reasoning substituted for following the documented procedure. Calls balloon (29 at low → 47–62), EFF collapses (67 → 1.5–17).
• Judgment/selection-heavy (high-board): effort pays, a lot. +18.6 points none→max (71.4 → 90.0). The none arm failed exactly the inference work: step-6 retrieval of the prior governed valuation (0/2), the final governance artifact (1/2), the required tool ordering (0/2).
• Numeric solving (trader-rfq): only high+ clears the hard check. The step-1 achieved-price-within-tolerance check went 0/2 at none and low, 2/2 at high and max — a genuine reasoning win no amount of procedure-following buys.
• Near-ceiling ops workflows: low is already enough. ops-settlement is 100.0 at every reasoning level; none dropped one trial to 72.7 by skipping settlement reads and fumbling the reopen-refusal answer (diligence failures, not capability ones).

Consistency and tails

• none is the least consistent arm (mean trial stdev 3.98; ops-settlement trials split 72.7 / 100.0). Adaptive zero-reasoning makes performance a coin flip on judgment moments.
• max has extreme latency tails: one high-board trial ran 107 minutes (median max trial ≈ 24 min).
• A parallel probe study on this same run found trap/prohibition compliance 8/8 at all four efforts — effort moves exploration volume, never policy.

Mechanism (check-level tally)

Losses at none are diligence failures — steps skipped, answers given without fetching evidence. Losses at high/max are over-execution failures — extra exploration displacing the documented procedure (skill-doc skips, occasional grounding slips at 2× par call volume), plus the EFF collapse the golf curve is designed to price. Correctness axes (GRD/ADH/SYN) barely move above low; the call count moves ~50%.

Replicates the cross-model pattern: the "Grok inversion" (EFF crash at high effort) is not grok-specific — luna shows the same shape once effort is pinned, inverting the #107/#108 deepseek observation (fewer calls at high) — the direction of the effort→calls relationship is model-family-specific.

Limitations (predeclared)

• n = 2 trials × 5 blocks per arm; within-ladder ordering (low vs high vs max) is not resolved — only none vs any is outside noise.
• Arms ran sequentially (same process), so arm order correlates with time-of-day; per-trial wall times show no drift step, and 0 infra deaths.
• Token spend is not recorded in transcripts; wall-clock and call counts are the cost proxies.
• One model, one provider route. The deepseek contrast above says do not generalize the calls-vs-effort direction across families.

Desk recommendation

Run luna at low for routine desk work (best OVR, cheapest, most consistent); escalate to high/max only for judgment-heavy or solve-type tasks (high-board-style reviews, price solving), accepting 2–3× wall-clock. Never run none for anything with judgment moments: the mean looks acceptable but the variance is where it bites.

Artifacts

• Run: arena run #110 (matches + per-trial transcripts under artifacts/arena/110/).
• Analysis script: session scratchpad analyze_run110.py (read-only over the live DB; board, card means, paired deltas, per-check tally).
• Plan (predeclared): docs/arena/2026-08-17-luna-reasoning-effort-plan.md.