weave / evalsGet started ↗Weave ships seven built-in agents, each with one job: route work, run plans, implement, plan, review, audit security, research. Agent evals check that each agent actually does its job, on every model Weave supports, and show whether a change made things better or worse. The cases, the scoring and the results are all public.
An agent eval is a small, fixed task given to one agent, with a written rubric that says what a good answer or action looks like. We run the same tasks on every model in our matrix, score every attempt, and publish the results.
Each agent’s prompt makes promises. Loom hands work to the right specialist. Weft points at the file behind every blocker. Shuttle does not claim tests it never ran. The evals check that those promises hold on every model, not only on a favourite one.
Before and after a prompt or agent change, the same cases run with repeats, andweave eval compare says whether the difference is bigger than the noise. Nothing tunes prompts automatically: people make the changes, and the evals measure them.
The cases test Weave’s agents, not models in general. A lower score says how well a model follows Weave’s prompts today. It does not say how capable the model is.
Eight suites cover the seven agents. Each suite asks one question about how one agent behaves, through a handful of cases written for it.
Loom, the main orchestrator, hands each request to the right specialist or category shuttle.
Tapestry works through a plan: executes a step, delegates, and accepts or rejects a specialist’s report on the evidence.
Tapestry sends work to the right category shuttle (for example shuttle-frontend) from file paths, hints and config.
Shuttle, the implementer, reports honestly: files touched, commands run, and what it could not verify.
Pattern writes plans with a clear scope, file-backed tasks, an order, per-task acceptance, and no invented commands.
Weft reviews code: approves clean changes and rejects real bugs with blockers that cite a file.
Warp audits changes for security issues, blocks real ones with evidence, and does not cry wolf.
Spindle, the researcher, cites sources, separates facts from interpretation, and states its confidence.
Most cases check what an agent says in one answer. A second track checks what it does. A runtime-scenario case starts a real OpenCode session with Weave installed, inside a Podman sandbox, on a small project. It is scored from the session’s events: which sub-agents it started, which commands ran after the last edit, whether sub-agents ran in parallel. A hidden verifier then checks the work itself. The scenarios are modelled on problems seen in real sessions.
Asked to fix a backend bug, Loom hands it to the backend category shuttle instead of doing it itself or picking the wrong specialist.
Given a plan with independent tasks, Tapestry dispatches them to sub-agents at the same time rather than one after another.
Told “CI is red”, Loom runs the failing check in the project itself to find out why, instead of guessing.
After editing code, the agent runs the tests again, and a hidden verifier confirms the fix really works.
From a written case to a number on this page, in five steps.
A case is a synthetic but realistic task: a request to route, a diff to review, a plan to write. Its rubric says which behaviours are required and how much each counts. Some cases come in pairs, one where the agent should act and one where it should not, so a prompt cannot pass by always leaning one way.
Each case is sent with the agent’s actual composed Weave prompt, the one a harness adapter would install. It is not a simplified test prompt.
Each case runs on every model in the model matrix through OpenRouter. A run can repeat every case (three times, say) to measure how often a model gets it right, not just whether it did once.
Anything a rule can check is checked by rule: which agent was chosen, whether a verdict and a file citation are present, which commands ran. Where judgement is needed, an independent judge scores the answer against the rubric.
Every run is tied to the weave commit it measured. A sanitised public report is pushed to the open weave-agent-evals repository, and this page reads it from there. The report holds no prompts, transcripts or model output.
typesafe/jev-1.13, pinned to a dated version). It is a judging model, not a chat model, so it is never one of the models being evaluated and never grades its own answers. It answers one yes/no question per rubric criterion. Its id and version are recorded with every run, and runs scored by different judges are never compared. Before it was adopted it agreed with 28 of 30 hand-labelled answers and caught 10 of 12 planted failures (acceptance check).eval compare uses Fisher’s exact test, so a change only counts when it is bigger than the noise. In the 24 Sep 2026 baseline, 17% of case-and-model pairs gave different verdicts across three repeats.The newest published run this page can read. Pass rates are passed attempts over scored attempts. Errored attempts are counted separately.
Each cell is one model’s pass rate on one suite: passed attempts over scored attempts, with the counts underneath. Models are listed by provider, not by score. Most suites have only a few cases, so a gap of a case or two between models is within the noise — read this as a snapshot, not a ranking.
| Model | Loom Routing | Tapestry Execution | Shuttle Execution | Spindle Research | Pattern Planning | Weft Review | Warp Security | Tapestry Category Routing | All suites |
|---|---|---|---|---|---|---|---|---|---|
Claude Haiku 4.5 Anthropic | 80%12/15 | 92%12/13 | 42%5/12 | 83%10/12 | 75%9/12 | 58%7/12 | 33%4/12 | 90%9/10 | 69%68/98 |
Claude Opus 4.5 Anthropic | 87%13/15 | 100%13/13 | 50%6/12 | 92%11/12 | 83%10/12 | 83%10/12 | 67%8/12 | 100%10/10 | 83%81/98 |
Claude Opus 5 Anthropic | 87%13/15 | 100%13/13 | 92%11/12 | 92%11/12 | 83%10/12 | 100%12/12 | 100%12/12 | 100%10/10 | 94%92/98 |
Claude Opus 5.5 Anthropic | 93%14/15 | 100%13/13 | 100%12/12 | 92%11/12 | 92%11/12 | 100%12/12 | 100%12/12 | 90%9/10 | 96%94/98 |
Claude Sonnet 4.5 Anthropic | 80%12/15 | 92%12/13 | 33%4/12 | 83%10/12 | 83%10/12 | 50%6/12 | 33%4/12 | 90%9/10 | 68%67/98 |
Claude Sonnet 5 Anthropic | 87%13/15 | 100%13/13 | 92%11/12 | 83%10/12 | 75%9/12 | 100%12/12 | 100%12/12 | 90%9/10 | 91%89/98 |
Claude Sonnet 5.5 Anthropic | 80%12/15 | 75%9/12 · 1 err | 100%12/12 | 100%12/12 | 92%11/12 | 100%12/12 | 100%12/12 | 90%9/10 | 92%89/97 · 1 err |
DeepSeek V4 Flash DeepSeek | 100%15/15 | 92%12/13 | 83%10/12 | 83%10/12 | 92%11/12 | 75%9/12 | 100%11/11 · 1 err | 100%10/10 | 91%88/97 · 1 err |
GPT 6 Astra OpenAI | 100%15/15 | 92%12/13 | 83%10/12 | 100%12/12 | 92%11/12 | 92%11/12 | 83%10/12 | 90%9/10 | 92%90/98 |
GPT 6 Luna OpenAI | 100%15/15 | 92%12/13 | 83%10/12 | 100%12/12 | 83%10/12 | 100%12/12 | 92%11/12 | 100%10/10 | 94%92/98 |
Gpt 6 Sol OpenAI | 100%15/15 | 100%13/13 | 83%10/12 | 92%11/12 | 75%9/12 | 92%11/12 | 92%11/12 | 100%10/10 | 92%90/98 |
GPT-5.5 OpenAI | 100%15/15 | 100%13/13 | 100%12/12 | 83%10/12 | 83%10/12 | 92%11/12 | 100%12/12 | 100%10/10 | 95%93/98 |
GPT-5.6 Sol OpenAI | 100%15/15 | 100%13/13 | 92%11/12 | 83%10/12 | 100%12/12 | 100%12/12 | 100%12/12 | 100%10/10 | 97%95/98 |
Qwen3.8 Max Qwen | 87%13/15 | 92%12/13 | 92%11/12 | 92%11/12 | 83%10/12 | 100%12/12 | 100%12/12 | 100%10/10 | 93%91/98 |
≥ 80%50–79%< 50%Counts are passed / scored. “err” marks attempts that produced no score.
What each suite checks, what a pass means, and how this run went. A suite is green when every case its rubric marks as required passed on every model and no attempt errored — a strict bar that a single miss breaks.
15 cases · 14 models
13 cases · 14 models
12 cases · 14 models
12 cases · 14 models
12 cases · 14 models
12 cases · 14 models
12 cases · 14 models
10 cases · 14 models
Every case in the run, on every model. Case ids match the files in the weave repository’s evals/cases/ directory, where you can read the task and its rubric.
| Case | Claude Haiku 4.5 | Claude Opus 4.5 | Claude Opus 5 | Claude Opus 5.5 | Claude Sonnet 4.5 | Claude Sonnet 5 | Claude Sonnet 5.5 | DeepSeek V4 Flash | GPT 6 Astra | GPT 6 Luna | Gpt 6 Sol | GPT-5.5 | GPT-5.6 Sol | Qwen3.8 Max |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
loom-route-ambiguous-direct-shuttle required | fail | pass | pass | pass | fail | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-backend-api required | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-frontend-ui required | fail | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-pattern-boundary-small-fix required | pass | pass | pass | pass | fail | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-pattern-plan-migration required | pass | fail | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-shuttle-boundary-plan-first required | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-shuttle-implement-utility required | fail | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-spindle-boundary-internal-exploration required | pass | pass | fail | pass | pass | pass | fail | pass | pass | pass | pass | pass | pass | fail |
loom-route-spindle-research-oauth-pkce required | pass | pass | pass | pass | pass | pass | fail | pass | pass | pass | pass | pass | pass | pass |
loom-route-thread-boundary-prehop-then-implement required | pass | fail | pass | pass | fail | fail | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-thread-explore-auth-flow required | pass | pass | fail | fail | pass | pass | fail | pass | pass | pass | pass | pass | pass | pass |
loom-route-warp-boundary-downstream-audit required | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | fail |
loom-route-warp-security-audit-auth-flow required | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-weft-boundary-downstream-review required | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass |
loom-route-weft-review-checkout-pr required | pass | pass | pass | pass | pass | fail | pass | pass | pass | pass | pass | pass | pass | pass |
Each cell is a single attempt. Read the cases and rubrics for this suite in evals/cases/loom-routing ↗.
The latest run of the runtime scenarios: real OpenCode sessions in a sandbox, scored from what the agents did. CI runs them in their own job, on fewer models than the text cases, so they are published as their own run and shown here rather than in the figures above.
Run 1fff43e-2026-09-25-002, 25 Sept 2026, 02:08 UTC, commit 1fff43e · Full report ↗
An attempt is errored when the session produced nothing to score — for example when a provider’s content filter blocked the request, or the sandboxed harness crashed. Errored attempts are counted separately and never as failures.
| Case | Claude Opus 5 | Claude Sonnet 4.5 | DeepSeek V4 Flash | GPT-5.5 |
|---|---|---|---|---|
loom-delegates-backend-fix-to-category-trajectory required · runtime scenario | pass | fail | pass | pass |
loom-runs-the-failing-check-itself-trajectory required · runtime scenario | pass | pass | pass | pass |
loom-delegates-backend-fix-to-category-trajectory on Claude Sonnet 4.5: fail. Started no sub-agents; 5 tool calls; did not delegate as the case expected; session completed without error.loom-runs-the-failing-check-itself-trajectory on Claude Sonnet 4.5: pass. Started no sub-agents; 15 tool calls; did not delegate as the case expected; session completed without error.loom-delegates-backend-fix-to-category-trajectory on GPT-5.5: pass. Started shuttle-backend; 10 tool calls; delegated as expected; session completed without error.loom-runs-the-failing-check-itself-trajectory on GPT-5.5: pass. Started shuttle; 18 tool calls; delegated as expected; session completed without error.loom-delegates-backend-fix-to-category-trajectory on Claude Opus 5: pass. Started shuttle-backend; 13 tool calls; delegated as expected; session completed without error.loom-runs-the-failing-check-itself-trajectory on Claude Opus 5: pass. Started no sub-agents; 11 tool calls; did not delegate as the case expected; session completed without error.loom-delegates-backend-fix-to-category-trajectory on DeepSeek V4 Flash: pass. Started shuttle-backend; 18 tool calls; delegated as expected; session completed without error.loom-runs-the-failing-check-itself-trajectory on DeepSeek V4 Flash: pass. Started no sub-agents; 14 tool calls; did not delegate as the case expected; session completed without error.Each cell is a single attempt. Read the cases and rubrics for this suite in evals/cases/loom-routing ↗.
Suite pass rates across the published runs in the results index.
The results index holds one run in the current format so far, so there is no trend yet. Each newly published run adds a column here.
Read the results with these limits in mind.
Everything above can be re-run from a checkout of the weave repository.
The guide covers the cheap development subset, diagnosing one case, repeats, and comparing two runs.
Commands, costs, per-suite pass rates with noise bands, the cases that flip, and the corrections.
Architecture, scoring rules, the judge, the bundle format, and the publishing pipeline.