Cortex Router on tau2-bench

The first choice shapes every turn

A short account request and a multi-step service problem can place different demands on a model. Using one fixed model for both leaves that cost–success tradeoff untested.

Cortex OSS makes the model choice from the opening prompt and keeps it for the whole conversation. This study asks what a calibrated policy would have achieved by reusing the outcomes and costs of completed fixed-model runs.

What we compared

Policy and baseline. Cortex OSS selects from Qwen3.7 Plus, GLM-5.3 and Kimi K3. The fixed baseline uses GPT-5.6 Sol for every turn.
Paired cohort. Both configurations use the same 276 valid customer-service tasks: 247 passed for Cortex OSS and 229 for GPT-5.6 Sol.
Scoring and cost. Success requires a tau2 reward of 1.0. Mean model cost covers the full conversation, using historical prices as of 2026-08-31.

Overview

: 89.5%; Task success
: 83.0%; Task success
: 76.9%; Cortex OSS vs. Sol
: +6.5 pp; Percentage points

On the same 276 paired tasks, mean model cost was $0.022/task for Cortex OSS and $0.095/task for GPT-5.6 Sol.

Performance and cost

Ten configurations

Higher and farther left is better: the counterfactual Cortex OSS point combines higher success with lower mean model cost than fixed GPT-5.6 Sol.

Task success × mean model costHigher success ↑ · Lower cost ←

Scroll horizontally to inspect the full chart →

All valid tasks · highest success first · mean model cost in USD/task

All valid tasks · highest success first · mean model cost in USD/task

GLM-5.3	253/278	91.0%	$0.032
Claude Opus 5	252/278	90.6%	$0.809
Cortex OSS	247/276	89.5%	$0.022
Qwen3.7 Plus	234/278	84.2%	$0.017
GPT-5.6 Sol	231/278	83.1%	$0.095
Kimi K3	222/278	79.9%	$0.080
Claude Sonnet 5	219/278	78.8%	$0.355
GPT-5.6 Terra	202/278	72.7%	$0.043
GPT-5.6 Luna	169/278	60.8%	$0.017
Claude Haiku 4.5	150/278	54.0%	$0.122

Operating profile

Per-task distributions

Agent cost

USD per task

Configuration

All configurations (stacked)

Tasks in bin

0$0.578$1.157

USD per task

Exact bin counts

Scroll for all 18 bins →

Agent cost counts by configuration in USD per task; bin boundaries rounded for display. No smoothing.

Claude Haiku 4.5	53	116	59	40	9	1	0	0	0	0	0	0	0	0	0	0	0	0
Claude Opus 5	0	0	1	7	20	17	22	23	19	16	18	19	13	9	12	8	7	67
Claude Sonnet 5	2	23	46	48	32	31	20	15	10	17	11	11	6	1	1	3	1	0
Cortex OSS	275	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
GLM-5.3	255	23	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
GPT-5.6 Luna	278	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
GPT-5.6 Sol	87	135	42	10	3	1	0	0	0	0	0	0	0	0	0	0	0	0
GPT-5.6 Terra	227	51	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Kimi K3	110	137	26	4	1	0	0	0	0	0	0	0	0	0	0	0	0	0
Qwen3.7 Plus	278	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

Agent turns

Assistant turns per task

Configuration

All configurations (stacked)

Tasks in bin

014.529

Assistant turns per task

Exact bin counts

Scroll for all 18 bins →

Agent turns counts by configuration in Assistant turns per task; bin boundaries rounded for display. No smoothing.

Claude Haiku 4.5	0	3	6	23	20	48	28	34	28	16	21	8	19	10	6	7	1
Claude Opus 5	0	0	7	23	14	42	19	56	31	12	25	4	13	9	4	6	13
Claude Sonnet 5	0	2	8	22	8	33	21	48	33	17	21	6	8	15	7	13	16
Cortex OSS	0	0	5	22	26	47	36	32	35	12	23	1	11	6	4	4	12
GLM-5.3	1	1	7	35	32	52	33	21	37	7	22	3	7	4	2	4	10
GPT-5.6 Luna	2	4	13	36	25	54	22	46	36	7	12	4	6	5	1	1	4
GPT-5.6 Sol	2	1	12	24	18	51	21	44	40	11	17	6	7	4	2	4	14
GPT-5.6 Terra	3	2	9	29	15	49	28	41	34	15	26	4	10	9	2	1	1
Kimi K3	2	0	5	27	14	50	22	40	27	16	25	3	8	11	6	6	16
Qwen3.7 Plus	0	0	5	17	22	47	30	41	39	15	22	1	9	9	4	4	13

Input tokens

Tokens per task

Configuration

All configurations (stacked)

Tasks in bin

0157,492314,984

Tokens per task

Exact bin counts

Scroll for all 18 bins →

Input tokens counts by configuration in Tokens per task; bin boundaries rounded for display. No smoothing.

Claude Haiku 4.5	2	8	26	50	43	24	23	19	16	12	12	14	11	10	5	2	0	1
Claude Opus 5	0	1	16	25	34	30	28	22	15	22	8	14	12	7	4	10	6	24
Claude Sonnet 5	0	3	15	17	33	36	24	16	17	21	9	10	9	9	5	5	11	38
Cortex OSS	0	11	36	55	38	31	32	10	22	3	8	6	7	3	6	2	3	3
GLM-5.3	2	12	54	63	38	21	27	16	17	3	6	4	4	6	1	2	1	1
GPT-5.6 Luna	10	34	63	70	32	28	17	9	5	2	3	4	1	0	0	0	0	0
GPT-5.6 Sol	7	27	67	42	39	27	20	17	9	4	2	6	5	2	1	1	0	2
GPT-5.6 Terra	9	28	63	48	37	28	18	24	9	11	2	0	0	1	0	0	0	0
Kimi K3	2	13	51	51	29	22	29	21	13	3	9	13	5	3	4	2	3	5
Qwen3.7 Plus	0	7	29	41	49	33	35	16	23	5	7	6	9	3	6	2	3	4

Output tokens

Tokens per task

Configuration

All configurations (stacked)

Tasks in bin

03,182.56,365

Tokens per task

Exact bin counts

Scroll for all 18 bins →

Output tokens counts by configuration in Tokens per task; bin boundaries rounded for display. No smoothing.

Claude Haiku 4.5	3	11	37	61	45	55	38	14	8	5	0	0	0	0	1	0	0	0
Claude Opus 5	0	3	35	48	44	40	41	24	19	5	10	1	1	1	0	1	1	4
Claude Sonnet 5	0	5	29	30	27	29	32	21	23	24	15	12	9	8	4	1	3	6
Cortex OSS	0	18	34	30	22	21	18	23	18	11	14	14	9	9	9	1	5	20
GLM-5.3	0	26	45	42	36	32	16	20	14	13	8	7	4	5	6	1	1	2
GPT-5.6 Luna	12	74	79	56	32	11	8	5	0	1	0	0	0	0	0	0	0	0
GPT-5.6 Sol	14	72	77	59	28	19	3	1	2	2	1	0	0	0	0	0	0	0
GPT-5.6 Terra	21	90	78	58	20	6	2	3	0	0	0	0	0	0	0	0	0	0
Kimi K3	0	9	31	55	42	55	25	20	15	4	8	6	3	1	1	0	1	2
Qwen3.7 Plus	0	0	2	4	11	26	25	29	26	16	23	25	14	13	12	7	11	34

Paired outcomes

Where outcomes differ

Share of all 276 paired tasks · 100% total

Routing policy

One model per conversation

Selected models across 276 valid router tasks

0276

Methodology and limits

Method and limits

Evaluation

278-task official Sierra base (1 trial)
Fixed models ran live; router results are calibrated counterfactuals.
Task success requires a tau2 reward of 1.0.

Cohorts

2 infrastructure failures excluded from their configurations.
Compare Cortex OSS and GPT-5.6 Sol on matching task IDs within each domain.
Missing outcomes and costs are never counted as zero.

Calibration

First-turn scores from the live Cortex sidecar feed cost-tilted gates.
Gates were fit only on the official training split.
Published results include train and test tasks; no held-out deployment validation.
One trial; no equal-cost comparison.

Pricing

Historical model prices as of 2026-08-31.
GPT costs use recorded token usage and historical OpenAI rates, including cached input.
Open models and Claude use recorded OpenRouter usage costs.
Latency, router overhead and total serving cost are not established.

First-Turn Routing on tau2-bench

The first choice shapes every turn

What we compared

Overview

Ten configurations

Per-task distributions

Agent cost

Exact bin counts

Agent turns

Exact bin counts

Input tokens

Exact bin counts

Output tokens

Exact bin counts

Where outcomes differ

One model per conversation

Method and limits

Evaluation

Cohorts

Calibration

Pricing