GPT-5.6 on Cogny Bench: Terra Matches Opus 4.8 at $0.05 a Run

OpenAI moved the GPT-5.6 family — Sol, Terra, and Luna — to general availability this week. Three tiers, one base model: Sol ($5/$30 per MTok) is the flagship, Terra ($2.50/$15) is pitched as "GPT-5.5 performance at half the price" for production agent workloads, and Luna ($1/$6) is the volume tier for classification and routing.

We ran all three through Cogny Bench the day after GA. The pitch undersells Terra.

The numbers

Cogny Bench is our internal eval: synthetic marketing-analytics problems with a planted ground truth and a deliberately withheld trap — a Simpson's paradox, an attribution leak, a tracking outage masquerading as a CPA spike — that a model has to infer from the data using a SQL tool. Scoring is 70% deterministic field checks + 30% a fixed Opus-4.8 judge.

Three things jump out.

Terra ties Opus 4.8 — at a thirteenth of the cost. 93.1 vs 93.1, $0.052 vs $0.657 per run. Against its own predecessor the jump is bigger than OpenAI's "matches GPT-5.5" framing: Terra scores four points above GPT-5.5 (93.1 vs 89.1) at less than a quarter of the per-run cost.

Sol buys nothing here. The flagship scored 93.0 — a tenth of a point below Terra, inside the noise — at 2.3× Terra's cost. Our problems are single-session analytical reasoning, so this doesn't rule out Sol pulling ahead on long-horizon agentic work, but on data analysis the extra tier is pure margin.

Luna is the sleeper. The $1/$6 volume tier scored 92.4 — one point behind the flagship, dead even with Grok 4.3 at medium reasoning, above GPT-5.5, GLM-5.2, and Sonnet 4.6 — for $0.024 a run. OpenAI positions Luna for tagging and routing; on our board it's a frontier-class analyst at classifier prices.

Where the points went

Seven of nine problems were a clean sweep — all three tiers scored 96–99 on the retention-attribution, channel-attribution, Simpson's-paradox, tracking-outage, currency-mix, event-splitting, and trend problems. The two tool-use traps showed the only spread:

• T01 (keyword cannibalization): Sol 94, Terra 91, Luna 85. All three identified the planted trap and acted on the correct entity rather than the decoy. The deductions came from our execution-verification checks (the judge couldn't confirm every action round-tripped) and, for Luna, thinner cited evidence for its reconnaissance.
• T02 (chart the story): 53–58 for all three. Before you read that as a GPT-5.6 weakness: every frontier model lands there — Fable 5 scores 53, Opus 4.8 55, Sonnet 5 53. T02 stays our hardest open problem.

The API change that broke our first run

Terra's first attempt scored zero. Not a capability problem — a 400:

Function tools with reasoning_effort are not supported for gpt-5.6-terra in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.

With GPT-5.6, reasoning plus function tools now requires the Responses API — Chat Completions only allows tools with reasoning off. The bench grew a Responses-API driver (multi-turn state via previous_response_id, flat function-tool schemas, function_call_output round-trips), and the same constraint applies to any production agent stack built on Chat Completions: if you're wiring GPT-5.6 into an OpenAI-compatible tool loop today, you're either porting to /v1/responses or measuring what reasoning_effort: 'none' costs you in quality.

That finding directly reshaped our own tiered OpenAI mix: the swap to Terra/Sol/Luna is held until Cogny's OpenAI path speaks the Responses API, because shipping it on Chat Completions would have 400'd every tool call.

What this means for the mix

Cogny runs a managed model mix per provider — workspaces pin a provider, we move the underlying models. These numbers say the OpenAI mix should lean Terra for the report/chat slots (Opus-class accuracy, $0.05 a run) and Luna for the volume slots, with Sol reserved until we see evidence it earns its 2× premium on longer-horizon work. That change is staged behind the Responses-API port.

Next on the bench: Grok 4.5, which xAI shipped this month at $2/$6 — same run, same problems, same judge.

Cogny Bench is our internal eval — problems and fixtures are synthetic (no customer data) and deliberately unpublished so scores stay meaningful. How it's built and scored.