Weaveweave / blog
Get started ↗
Blog/October 18, 2026

Spend on the Environment, Not the Model

The biggest quality gains we've had from coding agents came from what surrounds the model: tests it can run, an app it can start and see, and checks that don't lie. How we set that up for Fleet, and where it failed us.

  • practice
  • verification
  • fleet
A small violet cube at the centre of a much larger structure of cyan and teal rails, gates and nodes

When a coding agent produces bad work, the first instinct is to reach for a bigger model or a higher effort setting. Sometimes that helps. More often the model was fine and the place it was working in was not: no tests worth running, no way to start the app, no way to see what it had built.

Matt Pocock made this point well in a recent interview with Nate Herk. Asked about models, he said improving the environment your agents work in is “where I find the most gains”. His example was his own video editor. Once an agent could run it, click through it and take screenshots, he saw “a massive increase in the quality” of what it produced.

“I think I could get, you know, just as good [a] result … with Opus 4.5 and that setup.” — Matt Pocock, 33:38

Opus 4.5 came out in November 2025, and he thinks it would do just as well. The claim of this post follows from that: the money and attention you put into the agent’s environment pay back more than the money you put into the model. We build Fleet, our app for running coding agents side by side, with coding agents: since 7 September, 1,255 of the 1,265 commits on its main branch name a Claude model as co-author. Here’s what that environment looks like, what it let us do with cheaper models, and the places where it quietly lied to us.

Code is the environment the agent works in

Pocock’s framing is blunt. Put an agent in a “garbage vibe-coded crappy code base that has never had any care put into it” and you get garbage out. The agent copies what it sees, and it can only check its work with the checks that exist.

His test for a good environment is one engineering managers have used for decades. Imagine you had a hundred human developers. How would you raise the floor so that the worst of them did good work? You’d write tests that catch whole classes of bugs. You’d add checks that make the codebase better over time. People have been writing books about this for twenty years, and as he puts it, they feel like they were written last week.

“Code is the environment that the agent operates in.” — Matt Pocock, 28:32

An agent is that worst developer on a bad day: fast, confident and unable to tell plausible from correct unless something tells it. The environment is the something.

What that looks like in Fleet

Fleet is a .NET server with a Vue client that drives other people’s agents: Claude Code, OpenCode, GitHub Copilot and Pi (harnesses). That makes it awkward to test. Its core behaviour depends on a model replying, a harness turning that reply into events, and a browser showing them in the right order. Here is what we built so an agent can check its own work on it.

Tests in layers, with a map

Counted at commit 96df2b4 on 17 October 2026, Fleet has about 4,800 .NET test methods across its domain, application, infrastructure, API and integration projects, and about 3,200 client tests. On top sit 22 Playwright end-to-end tests that drive a real browser against a real server.

Numbers like those say little on their own. What makes them usable for an agent is a short table in the repo’s AGENTS.md that says which layer to start in for a given symptom. Three of its six rows:

Symptom Start here
Events not arriving at client Layer 1: SignalR contract tests
Events arrive but UI doesn’t update Layer 2: unit test applyDomainEvent with real payload
Everything works in tests but not in browser Layer 3: E2E with headed mode ($env:HEADED=1)

It turns “the UI looks wrong” into a first test to run, which is the step an agent new to a codebase is worst at.

The whole app runs without a model

Start Fleet with FLEET_HARNESS=test and it swaps every real harness for an in-process test harness. It prints a loud banner: “FLEET RUNNING IN TEST HARNESS MODE — no real model API calls.” Everything above the harness is the production code: the server, the event stream, the client.

Below that sits FakeLlmServer, a small server that speaks the OpenAI-compatible API and replies from a queue of scripted responses. Fleet’s conformance tests use it the other way round: a real OpenCode process talks to a fake model, and the tests check that the OpenCode harnesses produce the events Fleet expects.

Together these mean an agent can boot a complete Fleet, open it in a browser, click through a change and take screenshots, all at no model cost and with predictable output.

A scratch Fleet for every live check

The habit that grew out of this is written into our plans and learnings rather than as a rule in AGENTS.md. Before a change is called done, the agent starts a scratch copy of Fleet with its own home folder and its own port, often with a scripted model, and tries the change there, never in the person’s installed Fleet.

Our first try at Fleet mods is why we insist on it. The first stage was built in parallel lanes, one layer each, with the end-to-end step scheduled last. Three of its first PRs alone added 21,483 lines (#486, #487, #488). The plan had five places a mod could draw. When the stage shipped, mods could only draw on tool rows. For the second stage, each thin slice was checked live on a scratch Fleet before it merged. Its three features came in at about 2,250 lines against a budget of about 3,400, in about 3.5 hours against a budget of 11. The two stages weren’t the same size of job, so this isn’t a controlled comparison. Looking back, though, we put most of Stage 1’s waste down to nothing running end to end until the last day.

We hand the same habit to our users’ agents

Fleet ships a built-in fleet-run skill that users can switch on for their agent sessions. Its first line is the argument of this post in two sentences:

“Tests show the code does what the tests say. Running the app shows it does what the user asked.”

Same checks, smaller model

A large and a small violet block each pass through the same three cyan gates and end at identical teal squares
The gates do the work. The size of the block matters less.

If the environment catches mistakes, the model doesn’t have to avoid them all on its own. Weave’s own model choices show it. Weave runs an eval suite for each of its builtin agents, and publishes the results behind each default it changes.

Spindle researches external docs and APIs. On its 12-case suite, GPT 6 Luna, the cheapest model in the comparison, and Sonnet 5.5 both scored 56/60. Then one line changed in Spindle’s prompt: its report keeps only the facts that answer the question. Luna went to 60/60 and Sonnet to 59/60.

SPINDLE · 12 CASES × 5 RUNS

Before the prompt fix: GPT 6 Luna 56/60, Sonnet 5.5 56/60.

After it: Luna 60/60, Sonnet 59/60. Luna costs $0.00025 per attempt against Sonnet’s $0.0107, about 43× less.

THREAD · 13 CASES × 5 RUNS

Haiku 5.5: 65/65. Haiku 4.5: 52/65. Significant at p < 0.001, and Haiku 5.5 cost about a quarter as much per attempt.

Thread runs on Haiku 5.5 by default as of Weave 0.3.0.

The fix that mattered was in the environment, and it helped the cheapest model most. We still made Sonnet 5.5 Spindle’s default, because Weave puts Claude models first and Sonnet clears the bar. The report says plainly that Sonnet is not significantly better than Luna, which stays as the fallback.

Thread gathers evidence from the codebase. Its new suite tests exactly the habits a check depends on: separating what it found from what it guessed, saying “not found”, and giving exact paths and lines. Haiku 4.5 failed every attempt at the first of those. Haiku 5.5, a small model released on 7 October, passed all 65 attempts at about a quarter of the cost, so Thread moved to it.

Shuttle, which writes the code, stays on Sonnet 5.5. Implementation is where a quiet miss costs the most, and we haven’t published a comparison that would justify a cheaper model there.

Pocock lands in the same place from the other direction. He runs Opus 5.5 at medium effort and is thinking about going lower, “because I’ve kind of raised the floor on my agents … they just write stuff that passes the tests and the tests catch it.” Lower effort answers faster and uses fewer tokens.

Where the environment lied to us

An environment is software. It has bugs, and an agent will walk straight through every one of them. These are the ones that caught us.

A chained command checked the wrong thing. In one Weave eval, Sonnet 5.5 checked its work with cd /workspace && bun test 2>&1; ls … | head -3; which tsc. A shell reports the exit status of the last command, so the check measured which tsc, not the tests. It failed Sonnet on every attempt, even though the tests may have passed. The opposite is just as easy. An early fix to Shuttle’s prompt led Haiku 5.5 to end its checks with ; echo EXIT=$?, which always exits 0 and would have passed with no evidence at all. The fix had two parts. The scorer now trusts an exit code only when the test command is last or followed only by &&. And Shuttle’s prompt asks for each check as one plain command.

A demo that didn’t match the default. Push-guard, a sample mod that refuses risky pushes, passed its demo. But a permission hook only sees requests that would reach a person, and Fleet’s default permission level is “Allow everything”, where nothing asks and the hook never runs. Out of the box, the mod does nothing, and a passing demo didn’t show it. The lesson we took: run every demo under default settings.

Flaky tests turned main red. While Stage 2 was being built, main went red twice from older tests. One waited a fixed 200 ms for an event instead of waiting for the event itself (#519). The other built a zip that wasn’t deterministic and failed 4 runs in 6,000 (#522). Until each was fixed, every agent that branched from main inherited failures that had nothing to do with its change.

Limits shared across the machine. Agents working side by side share one operating system. While building Fleet’s app runner, we found that each dotnet watch took 51 of the 128 file-watcher instances Linux gives a user by default, so the third app running at once crashed. The failure looks like a bug in whatever happened to start third.

The eval had bugs too. Weave’s eval scorer treated a stream error that the harness had already recovered from as fatal. Its recorder also cut commands at 500 characters, so a bun test that followed a long heredoc vanished from the record (#325). Each of these made a model look worse than it was. A check that judges the agent needs its own checks.

Every one of these is a check that said the wrong thing. An agent trusts a lying check completely.

Raising the floor in your own repo

None of this needs Fleet or Weave. If you want to try it, this is the order we’d go in:

  1. Make the app start with one command, with fakes for anything paid or remote. A test mode that swaps out the model is worth more than you’d expect.
  2. Let the agent see the app. Browser automation and screenshots turn “I think it works” into evidence.
  3. Write down where to start. A symptom-to-test table in AGENTS.md or CLAUDE.md saves more time than any single test.
  4. Run each check as its own command, and read the exit code. Don’t chain checks with ;.
  5. Demo under default settings, on a scratch copy with its own data, port and home folder.
  6. Turn each surprise into a check. When an agent fools a check, fix the check, not just the change.
  7. Then try a cheaper model or a lower effort setting, and measure it with evals before you trust the saving.

The short version

  • The model is one part of what makes an agent good. The codebase, the tests and the ability to run and see the app are the rest, and they’re the part you control.
  • A good environment lets a smaller, cheaper model do the same work. In Weave’s evals, one line of prompt took the cheapest model in a comparison from 56/60 to 60/60, level with a model that costs 43 times as much.
  • The environment is code. It lies sometimes, and an agent believes it completely. Test your checks the way you test your product.

Sources

Figures from repositories are pinned to a commit, so the links keep showing what we read. We counted tests by [Fact] and [Theory] attributes and by it() and test() calls at Fleet commit 96df2b4, and commit co-authors from the Co-Authored-By lines on Fleet’s main branch up to the same commit.

Matt Pocock

Models

Weave (weave-io/weave at 5a39fee)

Fleet (pgermishuys/weave-fleet at 96df2b4)