Coding tasks make different demands
A small code edit and a bug spanning several files can put different demands on a model. Using one tier for both is straightforward, but the cost of that choice still needs testing.
Routing makes model selection a decision for each task. This study asks how much fixed Sol quality credit Cortex retains, and what happens to recorded model cost, when it can choose Luna, Terra or Sol.
What we compared
- Configurations. Cortex Router selects among Luna, Terra and Sol. Luna Medium and Sol Medium use fixed models, with Sol Medium as the baseline.
- Cohort. The same 500 SWE-bench Verified task IDs in each configuration: 1,500 recorded rows in total.
- Score. Recorded quality credits divided by 500 tasks, separate from the Boolean resolved flag. Resolved Luna rows carry 0.6 credit each, so its credit and resolved-task rate differ.
- Cost. Recorded model cost per task using the July 24, 2026 rate card; excludes Modal compute and router overhead.
- Limits. Totals are recomputed from the published per-task records. Those records do not include grader execution traces or establish train/test separation, uncertainty intervals, or performance on new workloads.
Overview
75.2% quality credit at $0.365 per task.
Read these charts together: higher quality credit and lower model cost are better. Cortex retains 94.9% of fixed Sol’s quality credit at 72.8% of its model cost.
- Per-task values = recorded quality credit or model cost total / 500 tasks
- Historical rate card: July 24, 2026 (2026-07-24)
| Sol Medium | 500 | 396 | 79.2% | 396 | $250.56 | $0.501 | 100.0% | 100.0% |
| Cortex Router | 500 | 376 | 75.2% | 376 | $182.31 | $0.365 | 72.8% | 94.9% |
| Luna Medium | 500 | 219 | 43.8% | 365 | $56.22 | $0.112 | 22.4% | 55.3% |
- Cortex vs fixed Sol: 27.2% lower model cost; 4.0 percentage points lower quality credit
- No evidence for total production serving cost, latency, or savings after router overhead
What sits behind the averages.
Where Router and Sol agree.
206 tasks took a smaller model.
- Luna or Terra: 206 tasks (41.2%)
- Sol: 294 tasks (58.8%)
Inspect a recorded task.
How the study was run.
Scope and quality
Each configuration uses the same 500 SWE-bench Verified tasks: 1,500 task results across three configurations. Quality is total recorded credit divided by the task count, separate from the Boolean resolved flag. Resolved Luna tasks receive 0.6 credit.
Generation and grading
Source-reported method: the study describes the Codex harness running in official SWE-bench task images on Modal at Medium reasoning. Fixed arms requested Luna or Sol; Cortex selected a model before agent execution and kept that choice fixed for each task.
The report describes 500 completed generations and grades per configuration, with patches graded using the canonical SWE-bench test specification. It reports retrying infrastructure failures instead of counting them as unresolved model outcomes. These procedures cannot be independently verified from the recorded scores and usage: execution traces, patches and retry logs are unavailable.
Historical model cost
Mean cost is total recorded model cost divided by 500 tasks. Costs use the historical rates below for uncached input, cached input and output tokens; they exclude Modal compute and router overhead.
| GPT-5.6 Luna | $1.00 | $0.10 | $6.00 |
| GPT-5.6 Terra | $2.50 | $0.25 | $15.00 |
| GPT-5.6 Sol | $5.00 | $0.50 | $30.00 |
Cached input is priced once at its discounted rate; reasoning output is not counted again. Rates apply to each task’s recorded token totals.
| Luna Medium | 92.2% |
| Cortex Router | 91.6% |
| Sol Medium | 91.4% |
Cache shares use total cached-input tokens divided by total input tokens, rather than averaging task percentages. Re-measure cache behavior and provider pricing on your workload before applying these comparisons.
Routing and interpretation
The study measures one Cortex Router operating point. It does not establish a threshold sweep, train/test separation, uncertainty intervals, or performance on new workloads. The paired results show where outcomes differ, but do not establish which routing decisions caused the quality gap. Task examples expose recorded scores and usage, without prompts or answers.