Private evals on your work.Training-grade data from every outcome.
Improve your models with private evals and RL-ready data from the work your teams already do.
Every outcome becomes a training signal.
Cortex finds your highest-spend AI workflows, then captures the context, actions, corrections, and results that make them RL-ready.
One Work Unit. Both sides of the label.
OpenSearch retrieval fix
PreferredRestore version-aware retrieval for accepted support answers.
Why this workflow: Support retrieval absorbs 31% of this team’s AI spend and repeats across 1,284 Work Units each month.
Models + tools
+2Reviewers
18.4M
Tokens · 30 days
$42.6K
AI spend · 30 days
1,284
Similar Work Units
92%
Verified outcomes
Traceable lineage
Every source, action, review, and outcome stays attached.
Source context
01
Jira issue + Slack incident
Model trajectory
02ClaudeClaude · prompt + response
Tool evidence
03GitHub diff + Cursor test run
Human review
04Rejected, corrected, approved
Terminal outcome
05Merged + retained after release
Training asset
06SFT + hard negative + reward
Tune ranking weight
Claude proposed a ranking-only path that treated the symptom instead of the retrieval regression.
- Reviewer rejected the approach
- Root cause remained unresolved
- Support fixture still returned stale docs
Fix version filtering
Claude used migration notes, corrected filter ordering, and added a regression test before merge.
- Migration notes identified the root cause
- Support regression fixture passed
- PR merged and remained live after release
Why Cortex labeled this
3 source signals- Human rejected the first model path
- Regression suite verified the correction
- Merged fix was retained after release
Training labels from this Work Unit
4 labels · PreferredSFT positive
Accepted correction to imitate
Model
ClaudeTimestamp
10:42:18
Device
MacBook Pro
Surface
Cursor Composer
Application used
CursorSignals inferred from humans
Supporting evidence found in
GitHubSlackHard negative
Rejected ranking-only approach
Model
ClaudeTimestamp
10:31:04
Device
MacBook Pro
Surface
Cursor Composer
Application used
CursorSignals inferred from humans
Supporting evidence found in
GitHubJiraRegression eval
Version-filter behavior to preserve
Model
ClaudeTimestamp
10:48:22
Device
CI runner
Surface
GitHub Actions
Application used
GitHubSignals inferred from humans
Supporting evidence found in
CursorJiraRL-ready reward signal
Outcome reward from shipped behavior
Model
ClaudeTimestamp
14:06:13
Device
Cloud runner
Surface
Release monitor
Application used
GitHubSignals inferred from humans
Supporting evidence found in
SlackJiraInspect full provenance5 events · full source trail
Representative private Work Unit. Customer data stays private.
One Work Unit. Five training assets.
Compile the same outcome evidence into SFT, preference, verifier, eval, and RL formats.
Teach the accepted path
SFT
The objective, approved context, tool sequence, and shipped completion become a supervised example.
{
"source_work_unit": "WU-DEMO-4182",
"objective": "Restore version-aware support retrieval",
"context": ["incident thread", "migration notes", "gold fixtures"],
"assistant_actions": ["reproduce", "patch", "test", "open_pr"],
"accepted_completion": "Version filter applied before kNN fusion",
"label": "preferred",
"weight": 1.0
}Teach chosen vs rejected
Preference
The rejected ranking-only draft and the reviewed version-filter path form a comparison with visible source evidence.
{
"source_work_unit": "WU-DEMO-4182",
"prompt": "Fix retrieval without regressing support quality",
"chosen": "Patch version filtering, add regression fixture, run gate",
"rejected": "Patch ranking weight, skip the version filter, omit regression gate",
"preference_reason": [
"shipped",
"quality_gate_passed",
"no stale-document citations"
]
}Make success machine-checkable
Verifier
Deterministic assertions and rubric criteria turn production outcomes into a reward signal an agent cannot game.
{
"task": "support_retrieval_regression",
"reward": 1.0,
"deterministic_assertions": {
"current_version_is_top_1": true,
"stale_document_citations": 0,
"regression_suite_passed": true
},
"rubric": ["correctness", "evidence", "safe_release_process"]
}Test every system change
Eval
A related Work Unit from the same workflow becomes a held-out case after group-aware deduplication and partitioning.
{
"source_work_unit": "WU-DEMO-4217",
"workflow_group": "support_retrieval_versioning",
"eval_case": "support_retrieval_0042",
"starting_state": "index migrated; stale version remains searchable",
"required_context": ["product_version", "customer_tier", "live_docs"],
"allowed_tools": ["search", "read", "patch", "test"],
"grader": "assertions + criterion rubric",
"partition": "private_held_out",
"train_group_overlap": false
}Train long-horizon behavior
RL
State, actions, observations, tool results, terminal outcome, reward, and verifier lineage stay connected.
{
"trajectory_id": "traj_WU-DEMO-4182_preferred",
"initial_state": "retrieval regression with customer impact",
"actions": [
{"type": "read", "target": "migration_notes", "observation": "version filter moved after fusion"},
{"type": "search", "target": "support_gold_fixtures", "result": "stale versions reproduced"},
{"type": "edit", "target": "query_builder", "result": "filter applied before fusion"},
{"type": "test", "target": "support_regression", "result": "pass"}
],
"terminal_outcome": "shipped_and_accepted",
"reward": 1.0,
"verifier_version": "support_retrieval_v3"
}Evidence you can recompute.
Trace quality, cost, and model choice back to task-level results.
Tasks
500
Task-arm rows
1,500
Configurations
3
Run status
Complete
Luna Medium
gpt-5.6-luna · fixed
Router Medium
Cortex route · Luna / Terra / Sol
Sol Medium
gpt-5.6-sol · fixed
Quality is recomputed from task-level credits. Cost uses recorded input, cached-input, and output usage. Directional result; remeasure on customer traffic before production claims.
Build private evals from the work only your company can see.
Turn real work outcomes into private evals and training data you own.