How the Arena measures

What a match is

One model is given one workflow and works it end to end, several times over. Each play is a trial; the trials of one model on one workflow fold into a single match, and a set of matches on the same workflow is a board.

The model is not answering questions about a trading desk. It is driving one. Every turn goes through the same application the desk uses, against a real database seeded for that match, and nothing pauses for a human to approve it. A step that books a trade books one.

Nothing is graded on what the model says it did. After the match, the transcript is rebuilt from the system's own trace log — the tool calls that actually dispatched, the results they actually returned, the skill documents actually opened. A model that describes a tool call it never made has described nothing.

The workflows

Each workflow is a desk day written down: an ordered set of turns for one persona, with the checks that grade each turn. They are not prompts in the usual sense — the fixtures behind them are real positions, real risk runs and real counterparty documents, and the answers are harvested from the system rather than invented.

Par is the tool-call budget efficiency is scored against. It is the median of every fully correct trial a workflow has ever produced, so it is a realistic competent run rather than the theoretical minimum — on one workflow the theoretical minimum is eleven calls and the leanest real trial in the arena's history took fifteen. A workflow with no par yet has not produced enough fully correct trials to calibrate one.

How a run is graded

Every turn carries checks, and every check is worth one point and decides itself by rule. Did this tool fire with these arguments. Does this number in the answer match the one the system returned. Was this artifact written. Was this forbidden tool left alone. There is no language model anywhere in the scoring path.

Each check belongs to one of four axes, and each axis becomes a stat scored out of 99. A fifth stat, efficiency, is computed from the tool-call count rather than from checks. Those five blend into the OVR you see on the leaderboard, in this order:

• GRD — grounding. Did the numbers in the answer come from the system? Weighted heaviest.
• ADH — adherence. Did it call the right tools, and leave the forbidden ones alone? Weighted second.
• SYN — synthesis. Did the report or artifact it was asked for actually get written? Weighted third.
• EFF — efficiency. How many tool calls did the answer cost? Weighted third.
• PRC — procedure. Did it follow the desk's documented route? Weighted lightest.

Grounding leads deliberately. It is the hardest axis to fake: a model cannot talk its way to a number it never fetched. It is also the first tie-break when two contestants land on the same OVR.

Efficiency is scored like golf. Coming in at or under par is full marks and beating par earns nothing extra, because the arena is not looking for the shortest route. From par the score falls away steadily and reaches zero at twice par, and it is gated by correctness — a fast wrong answer scores nothing for being fast. Running no tools at all is not efficiency, it is not having done the work, and it scores zero.

The sixth stat, consistency (CON), is the odd one out: it is not an ability but a spread. It measures how tightly a model's trials cluster on the same workflow, and it can only discount the OVR, never raise it — up to about a fifth of the score for a model that swings, nothing at all for one that repeats itself. A model with a single trial has no consistency and its cell shows an em dash, which means not measured, not perfect.

The harness

The arena runs the production desk, not a mock of it. Each match seeds its own fixtures into the database, runs, and then purges everything it created — positions, portfolios, requests, scenario files — on evidence from the trace, so the next contestant meets the same desk the last one did. Leftovers are not merely untidy here: workflows resolve books by name, so a stale row makes each successive contestant's job quietly harder than the last one's.

No number in a score comes from a language model. Pricing, Greeks and risk are computed by a pinned deterministic quant engine, and the pin is exact rather than a minimum, because the engine's version is part of the evidence: a point release once moved a graded valuation with no change in this repository at all.

A contestant is a model together with its reasoning effort and its output budget, because both move scores enough to be part of the identity. The same model at two efforts appears twice on a board, each arm badged, and the two are read as separate contestants rather than as one model measured twice.

When the transport fails rather than the model — a dead gateway, a refused payment, a blank response with errors behind it — the match is recorded invalid and excluded from every average. It is not scored zero. An infrastructure outage that reads as poor ability is the single easiest way for a leaderboard to lie, and it has happened here often enough to be designed against.

What the numbers do not say

Boards are never merged. A different field, a different manifest revision, or a repaired harness makes two boards incomparable, so each is published exactly as it was measured. Cards are a different matter: a card is an absolute measurement that does not depend on who else was in the field, which is why the model pages can average cards across workflows while the leaderboard refuses to merge the boards they came from.

A saturated check measures nothing. When every contestant passes a check, it still occupies the denominator while carrying no signal about ability. Several workflows have drifted that way as models improved, and the boards that show it say so in their own notes. A high score on a saturated workflow means the field cleared a bar, not that it is strong.

Doing nothing is not zero. Some checks grade restraint, and a model that never acts passes all of them by accident. The floor of the objective score is therefore a little under eight rather than zero, and a score near that floor usually means the transcript is empty rather than wrong.

There is no LLM judge in the ranking. A panel of judge models exists and can be switched on, but it is off by default and advisory when on. It was benched after a run in which it ranked models in roughly the reverse of the deterministic axis.

Glossary

Workflow

A scripted desk day: an ordered set of turns a persona is asked to work, with the checks that grade each one. The table above lists every one.

Run

One launch of the arena. A run pairs some models with some workflows and repeats each pairing for a number of trials.

Board

A run that was a real field on one workflow, curated by hand. A one-model smoke test is a run but not a board, because a rank needs opponents.

Match

One contestant on one workflow inside one run. Its trials are folded into a single match, so a two-trial contestant still has one row.

Trial

One complete play of a workflow. Trials are what let consistency be measured: a single trial has nothing to be consistent against.

Contestant

A model together with its reasoning effort and its output budget. The same model at two efforts is two contestants, because both settings move scores enough to matter.

Check

One graded assertion, worth one point. Checks are deterministic rules over the transcript, never a language model's opinion.

Axis

The kind of ability a check tests. Every check belongs to exactly one of grounding, adherence, synthesis and procedure.

Par

A realistic tool-call budget for a competent run, taken as the median of every fully correct trial a workflow has ever produced. Not the theoretical minimum, which no real trial approaches.

OVR

The single 0–99 headline number: the five stats blended, then discounted for inconsistency.

Archetype

A label for the shape of a card, not a sixth measurement. A Sniper leads on grounding, an Anchor on adherence or procedure, a Playmaker on synthesis or efficiency, and an All-rounder is a card whose stats sit within eight points of each other.

Provisional

A run published as cards but never ranked, because it had no field. A card is absolute and survives having no opponent; a rank does not.

Invalid

A match the harness threw away because the transport failed, not because the model did. It is excluded from every average rather than scored zero.