Cortex Router · tau2-benchTask-level routing across Qwen, GLM, and Kimi
A retrospective 100-task pilot of one routing decision for the full agent conversation, measured on task success and recorded model cost.
Key takeaways
- The operating point: On the 99-task common cohort, Cortex reaches 81% task success at $0.159 per completed conversation.
- Point comparison: Against Kimi K3, it gives up 6 percentage points of success and uses 61% less mean cost.
- Route mix: 5 Qwen3.7 Plus / 44 GLM-5.2 / 51 Kimi K3 across the 100-task pilot cohort.
81% success at $0.159 per task
Against Kimi K3, the task-level router gives up 6 percentage points of task success and reduces mean agent cost by 61% on the 99-task common cohort. This is a point comparison, not an iso-cost result.
Every point is a complete agent run
Task success is the tau2-bench reward. Cost includes the recorded model usage for the full conversation. The table shows each arm's available valid rows and makes differing denominators explicit.
Mean cost vs. task success
Provider logos identify every model and the Cortex router · scroll chart →
| Configuration | Passed | Success | Cost / task |
|---|---|---|---|
 Qwen3.7 Plus | 20/100 | 20% | $0.103 |
 GLM-5.2 | 83/100 | 83% | $0.307 |
 Kimi K3 | 86/99 | 87% | $0.413 |
| Cortex task-level router | 81/100 | 81% | $0.159 |
Per-task distributions, not cumulative curves
Stacked histograms show raw binned task counts across cost, turns, and token use. No smoothing. Hover a legend item or bar segment to isolate a series.
Agent cost
USD per task →
Stacked binned task counts by agent cost (x truncated at the 98th percentile). No smoothing. Each bar is a raw bin total.
- Qwen3.7 Plus
- GLM-5.2
- Kimi K3
- Cortex task-level router
Agent turns
Assistant turns per task →
Stacked binned task counts by assistant turns (x truncated at the 98th percentile). No smoothing. Each bar is a raw bin total.
- Qwen3.7 Plus
- GLM-5.2
- Kimi K3
- Cortex task-level router
Input tokens
Tokens per task →
Stacked binned task counts by input tokens (x truncated at the 98th percentile). No smoothing. Each bar is a raw bin total.
- Qwen3.7 Plus
- GLM-5.2
- Kimi K3
- Cortex task-level router
Output tokens
Tokens per task →
Stacked binned task counts by output tokens (x truncated at the 98th percentile). No smoothing. Each bar is a raw bin total.
- Qwen3.7 Plus
- GLM-5.2
- Kimi K3
- Cortex task-level router
Where routing changes the answer
The same 99 valid task IDs are compared between the router and Kimi K3.
One decision for the full conversation
Cortex scores the initial task prompt, chooses one model, and keeps that model for every turn. This avoids mid-conversation model changes and preserves prefix-cache locality.
Retrospective, paired, and recomputable
This is a 100-task pilot from the tau2-bench base split. Cortex scores the initial task prompt, chooses one model, and keeps that model for every turn. Infrastructure failures are excluded from an arm's descriptive metric; headline comparisons use only task IDs valid in both the router and baseline arms.
Recompute every number
The export includes task ID, domain, arm, pass/fail, cost, turns, token counts, selected route, infrastructure status, and a claim map.