What changes when an agent improves its own harness?

Two agents can use the same model and behave differently. One gets a short instruction and a shell. Another gets project guidance, a focused search tool, and a way to verify the result.

The software around the model changes what it can do and what information it sees. That surrounding system is often called its harness.

Harness², a Google Research project, explores agents that refine that system while leaving model weights fixed. Its paper describes two connected levels of refinement: adapting the execution harness for a task and retaining lessons that guide later harness improvements.

I haven’t run its model-backed benchmarks. Here I want to use the project to explain what a harness change means, and how I would test one before trusting it in an application.

The model is one part of the system

Suppose an agent maintains a reading-list application. A task asks it to fix a bug where imported URLs create duplicates.

The model needs to find the normalization policy, inspect the import flow, and verify a correction. It might fail because it never found the shared helper. It might find the helper and still stop without testing the import path.

Those failures suggest different changes. Better search guidance can help with discovery. A verification script can expose the second failure. Replacing the model is another option, but it doesn’t answer which part of the system caused the problem.

A harness experiment keeps some variables fixed so you can investigate the rest. If you change the model, tools, prompt, and task set together, you lose that clarity.

The eight editable components

Harness²’s component list names eight parts. The examples below show what each could mean in our fictional reading-list agent:

These are examples, not default settings shipped for this application. They also have different authority. A prompt describing a restriction and code enforcing that restriction are separate mechanisms.

Before changing anything, identify the behavior you want to improve. “The agent needs better memory” is less useful than “it repeatedly searches for a normalization helper whose location we already verified.”

Propose, probe, and compose

The Harness² paper describes a Propose–Probe–Compose process. Candidates change the harness, executions provide behavioral evidence, and a composition uses the contrasts between those executions to produce a refined harness. The task-level process operates without ground-truth feedback.

That last point needs care. An agent can compare what happened in several attempts without knowing a hidden benchmark answer. It can observe that one attempt found a relevant file or produced a deliverable another missed.

It can still misinterpret those observations. A confident final message is weak evidence that an application works. The quality of the comparison depends on what the executions reveal and how the refined result is evaluated afterward.

For the import bug, I would first inspect an unsuccessful trace. Did the agent search for the helper? Did it read the caller? Which check did it run?

Then propose one targeted change. For example, add a short tool-guidance sentence directing the agent to search existing normalization callers before adding another parser. Keep the rest of the harness fixed for the first comparison.

If the next execution finds the helper, that supports a narrow observation about discovery. You still need to verify the resulting code and try other tasks before turning the sentence into a permanent rule.

Retained lessons change future experiments

Harness² also retains adopted harnesses and lessons. Its experiment guide describes ordered tasks within a domain reading earlier entries, with separate libraries for experiment configurations.

This makes the system’s history part of the evaluation. A later task can benefit from earlier evidence. Two runs starting with different retained lessons aren’t using the same effective harness.

For an application experiment, save the starting memory snapshot alongside the configuration. Record which lessons the candidate could read and which ones it produced.

Keep task-specific answers out of general guidance. “The fixture’s broken row is number 17” doesn’t teach the next task how to validate imports. “Check whether imported URLs pass through the shared normalizer” describes a procedure you can examine across tasks.

A useful memory item should say what observation supports it. Otherwise a mistaken explanation can become a permanent instruction that later runs treat as established fact.

Keep an untouched evaluation set

A system that edits its own harness can fit the tasks used to improve it. To find out whether a change generalizes, reserve tasks it hasn’t used for that improvement.

For the reading-list agent, the development set might include duplicate imports and URL fragments. Hold back separate tasks, such as preserving query strings and checking cross-account ownership, with their expected outcomes unavailable to the improver.

Freeze the candidate before running that evaluation. Keep the model and tool permissions fixed too. Compare the baseline and candidate from the same memory snapshot, and verify the outputs using checks defined before inspecting either result.

If you change the harness after seeing a held-out failure, that task has now informed development. Keep a new untouched set for the next claim about generalization.

This is an evaluation protocol I would apply to a project. It doesn’t imply every benchmark run in the repository uses that exact application protocol.

Inspect what the agent can read

The Harness² experiment guide explicitly discusses public benchmark rubrics being reachable online. It notes that the release leaves executor network access unrestricted and its Codex executor runs with the sandbox disabled.

That is a concrete evaluation concern. An agent that downloads a rubric can optimize against information the experiment intended to keep separate. A score alone won’t reveal how it obtained that information.

The project’s setup guide separates dependency downloads from execution-time access. Review those requirements before running an experiment, and apply isolation outside the agent when that is the boundary you need to enforce.

For our application example, keep hidden checks and expected answers outside the agent’s readable workspace. Don’t put their path in a tool description or include them in the instruction bundle by accident.

Record the cost of the improvement process

A refined harness can make a later task cheaper while requiring several expensive attempts to produce it. Measure the improvement process as well as the adopted execution.

Record model usage and elapsed time for candidate generation and probing. Keep the number of attempts and the conditions under which a result was adopted. An experiment with many discarded attempts is different from a single successful run.

For a small project, start with one observed failure and a small task set. Inspect the proposed harness diff as you would application code, especially if it changes permissions, loads a plugin, or launches another agent.

Keep changes to the evaluation rules under separate control. A successful experiment should leave you with an understandable difference and evidence about where it helps. If it weakens a check to make the result pass, the task remains unresolved.

Want me to talk about your product? You can sponsor this site.