Agentic LLMs usually fail the identical approach, time and again. A Stanford analysis staff traced this to lacking, reusable capabilities. Their system, TRACE, diagnoses these gaps and trains for them straight.
TRACE stands for Turning Recurrent Agent failures into Capability-targeted coaching Environments. It was launched open-source beneath an MIT license.
What downside does TRACE resolve?
To know the design, first contemplate why brokers fail. They lack particular abilities that duties demand, like retrieving the correct document or verifying a precondition.
Two mainstream fixes spend compute poorly. Direct RL or SFT offers sparse rewards that by no means say which ability was lacking. Broad artificial knowledge is untargeted, so finances flows to abilities the mannequin already has.
Nevertheless, TRACE observes that failures are usually not random. A small set of deficits accounts for many failed trajectories. Subsequently, every recurring deficit can grow to be its personal dense, verifiable coaching sign.
How does TRACE work?
On condition that findings, TRACE runs an automatic four-step pipeline. Every step is pushed by an LLM agent following a markdown immediate.
Step 1: Contrastive functionality evaluation
The bottom agent generates rollouts within the goal atmosphere. An evaluation agent splits them into profitable and failed units. It then labels each trajectory-capability pair as NA, PRESENT, or LACKING.
A functionality is retained solely when it’s contrastive and high-coverage. Particularly, its contrastive hole should clear δ = 0.20 and protection should clear ρ = 0.10. Consequently, the pipeline retains abilities whose absence concentrates in failures.
Step 2: Focused atmosphere synthesis
Subsequent, a technology agent builds one artificial atmosphere per retained functionality. Every atmosphere isolates a single functionality whereas preserving the goal’s instrument schemas and format.
Process situations are procedurally generated from random seeds. As a result of technology and verification are algorithmic, rewards want no human labels or LLM decide.
Step 3: Functionality adapter coaching
Then every functionality will get one LoRA (Low-Rank Adaptation) adapter, skilled on its artificial atmosphere. The coaching algorithm is GRPO (Group Relative Coverage Optimization). The bottom mannequin stays frozen all through.
GRPO teams rollouts by shared seed, so situations are similar inside a gaggle. Rewards are then normalized inside every group to isolate the coverage’s contribution.
Step 4: MoE composition with token-level routing
Lastly, TRACE composes the adapters right into a Combination-of-Consultants (MoE) mannequin. The spine and adapters keep frozen, and solely light-weight token-level gates are skilled.
At inference, every token is routed top-1 to a single functionality adapter. This lets the mannequin swap consultants mid-trajectory.
Interactive Explainer
TRACE diagnoses the capabilities an agent lacks, builds one verifiable atmosphere per hole, trains a LoRA skilled for every, then routes tokens throughout consultants. Step by way of the pipeline under.
1 · Contrastive Functionality Evaluation
Cut up rollouts into cross / fail, then maintain gaps that separate them.
Retained if Δ ≥ 0.20 and Cov ≥ 0.10
2 · Focused Setting Synthesis
One seeded, auto-verifiable atmosphere is generated per functionality.
3 · Functionality Adapter Coaching (GRPO)
Rollouts share a seed; rewards are normalized inside the group.
🔒 Base mannequin frozen · solely Δc updates
4 · MoE Composition · Token-Stage Routing
A realized gate routes every token top-1 to a single functionality skilled.
Decide a process above to route its tokens.
Outcome · τ²-Bench general cross fee (Qwen3-30B-A3B)
Focused coaching and MoE composition beat immediate optimization and single-adapter baselines.
Which capabilities did it discover?
In apply, on τ²-Bench, contrastive evaluation recovered 4 deficits. These had been structured knowledge reasoning, multi-step process completion, precondition verification, and gear calling precision.
Notably, this findings stayed steady throughout ten impartial runs. Structured knowledge reasoning alone coated the most important share of failed duties. Multi-step process completion adopted carefully behind.
Use circumstances with examples
To floor these capabilities, contemplate three concrete duties. Every maps to a definite failure mode that TRACE targets.
- First, take a customer-service airline agent. A consumer asks to cancel a primary financial system flight booked 14 days in the past. Precondition verification checks coverage eligibility earlier than calling
cancel_reservation. - Subsequent, contemplate a compound retail request. A consumer asks to cancel two reservations and modify a 3rd. Multi-step process completion stops the agent quitting after the primary sub-task.
- Lastly, contemplate a coding agent on SWE-bench Verified. Appropriately finding the related operate or file is a functionality. It’s crucial for fixing a bug or updating an API name.
How do the outcomes examine?
With these capabilities skilled, TRACE was examined on two backbones and two benchmarks. τ²-Bench measures customer-service cross fee throughout 50 airline and 114 retail duties. SWE-bench Verified measures Go@1 on 500 actual GitHub points.
| Spine | Technique | τ²-Bench Total (%) | SWE-bench Verified Go@1 (%) |
|---|---|---|---|
| Qwen3-30B-A3B | Base | 32.9 | 26.0 |
| GEPA (immediate optimization) | 39.6 | 31.0 | |
| SWE-RL | — | 32.6 | |
| Single Functionality GRPO (Ours) | 40.3 | 36.6 | |
| TRACE (Ours) | 48.2 | 41.0 | |
| Qwen3.6-27B | Base | 50.0 | 68.0 |
| GEPA (immediate optimization) | 53.0 | 69.6 | |
| SWE-RL | — | 70.4 | |
| TRACE (Ours) | 59.1 | 73.2 |
On Qwen3-30B-A3B, TRACE improved τ²-Bench by +15.3 factors and SWE-bench Verified by +15 factors Go@1. It beat the strongest exterior baselines, GEPA and SWE-RL, by +8.6 and +8.4 factors.
Furthermore, TRACE is sample-efficient. Utilizing beneath one-fourth the rollouts, it exceeded the ultimate scores of GRPO and GEPA. Its remaining accuracy ran +10.4 and +8.6 factors larger on τ²-Bench. Moreover, a 3rd benchmark, ToolSandBox, confirmed the identical sample.
On Qwen3.6-27B, TRACE reached 73.2% Go@1 on SWE-bench Verified. That 27B open-weight mannequin surpassed GPT-5.2-Codex (72.8%), GLM 5, and Claude 4.5 Sonnet on the general public leaderboard.
Implementation
For AI professionals, the pipeline is benchmark-agnostic and pushed by markdown prompts. After environments are generated, every functionality adapter is skilled in opposition to a operating vLLM server.
MODEL="Qwen/Qwen3-30B-A3B-Instruct-2507" # set your spine
NAME="structured_data_reasoning" # set the aptitude
N_GPUS=4
# 1 — serve the frozen base mannequin, with LoRA hot-swapping enabled
export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
vllm serve "$MODEL" --port 8000 --tensor-parallel-size 4
--enable-lora --max-loras 2 --max-lora-rank 32
--enable-auto-tool-choice --tool-call-parser hermes # qwen3_xml for Qwen3.6
# 2 — practice one LoRA adapter per functionality with GRPO
export VLLM_BASE_URLS=http://localhost:8000
torchrun --nproc_per_node="$N_GPUS" --master-port=29501 -m practice
--game "capability_$NAME" --model "$MODEL"Every LoRA adapter provides ~1.6B trainable parameters, or 5.3% of the spine. The skilled MoE gate provides simply 491,760 parameters in complete. Default thresholds are ρ = 0.10, δ = 0.20, and 8-of-10 cross-run consistency.
Key Takeaways
- Failure distinction is actionable. Evaluating profitable and failed trajectories surfaces trainable deficits, not generic error labels.
- Focused environments are sample-efficient. Every verifiable atmosphere rewards one functionality, so each rollout carries dense sign.
- Composition beats collapse. Separate consultants with token-level routing beat the perfect single-adapter different by seven-plus factors.
- Coaching beats prompting. Educated adapters scale monotonically, whereas prompt-only optimization plateaus after 4 capabilities.
- A 27B mannequin topped the leaderboard. Qwen3.6-27B hit 73.2% Go@1 on SWE-bench Verified, above GPT-5.2-Codex.
