Same Weights, Better Agent: Teaching Models to Tune Their Own Harness

An agent can rewrite its own scaffolding and get substantially better, without a stronger model, a human engineer, or a single change to its weights. Freeze the model, let it read its own failure traces, propose small edits to the system around it, and keep only the edits that pass a regression test. MiniMax M2.5 went from 40.5% to 61.9% on held-out Terminal-Bench-2.0 tasks this way.

That is the result from “Self-Harness: Harnesses That Improve Themselves,” from Hangfan Zhang and colleagues at Shanghai AI Lab. The “harness” is everything around the model: system prompts, tool definitions, memory, runtime limits, recovery rules, orchestration logic. The same model behaves very differently under different harnesses, and hand-tuning one per model doesn’t scale as models multiply. Self-Harness has the model tune its own.

• MiniMax M2.5: 40.5% to 61.9% held-out (53% relative gain)
• Qwen3.5-35B-A3B: 23.8% to 38.1% held-out (60% relative gain)
• GLM-5: 42.9% to 57.1% held-out (33% relative gain)
• Gains hold on a held-out split the proposer never saw, so the edits generalize
• Model weights and evaluator stay frozen; only the harness changes

The Loop: Mine, Propose, Validate

Self-Harness runs three stages on a fixed model and evaluator, evolving only the configuration around them.

Weakness Mining. Run the model under the current harness on a held-in task set with verifiable pass/fail outcomes. Instead of treating failures one by one, cluster them by a “verifier-grounded failure signature,” a tuple of what the evaluator rejected (timeout, missing artifact), how the agent’s behavior caused it, and the reusable mechanism behind it. Two failures group only when they agree on all three. The output is an evidence bundle ordered by frequency and how addressable each pattern looks.

Harness Proposal. The same model, now in a “proposer” role, reads the evidence and generates K distinct candidate edits. Each one is minimal and targeted, mapped to a specific surface (system prompt, runtime control policy like tool-error and message caps, available tools and memory, subagent/skill config) and shipped with an audit record: which failure it targets, what it changes, the expected effect, and the regression risk. Diversity rules push branches at different failures; minimality rules forbid sweeping rewrites.

Proposal Validation. Re-evaluate each candidate on both the held-in split (did it fix the diagnosed failure?) and the held-out split (did it break anything unseen?). A proposal is accepted only if it doesn’t regress either split and improves at least one. Accepted edits merge into the harness; rejected ones are logged but discarded. The whole lineage is auditable and reversible.

What It Actually Changed

The discovered edits are narrow, model-specific configuration tweaks, not generic “think step by step” padding.

For MiniMax M2.5, it learned to create required output artifacts earlier, capped total tool messages to stop open-ended exploration loops, and added structured-output guidance to cut schema-invalid tool calls. For Qwen3.5, it added dependency prechecks before tool operations, redirected the agent toward missing files after a tool error, and blocked blind retries of failed commands. For GLM-5, it made environment changes (installed tools, PATH edits) persist across shell sessions and forced a transition from exploration to implementation when no artifact was being produced. Each is the kind of fix a human harness engineer would write after staring at the same failure ten times, found automatically from the traces.

Why No Bigger Model Is Needed

The trick is that bounded harness improvement only needs behavioral evidence, not a smarter optimizer. The model isn’t searching the whole harness space or reasoning about why edits work globally; it just recognizes a recurring failure pattern from verifier-grounded traces and proposes a plausible, testable patch. The regression gate does the rest. That separates this from external “meta-harness” methods that need a stronger agent to drive the search (costly, and unavailable for frontier models), and from Reflexion-style self-improvement that adapts responses or memory rather than the declared infrastructure itself.

Caveats

The scope is deliberately bounded. Everything happens inside Terminal-Bench-2.0, a 64-task subset (multimodal and external-dependency tasks were excluded, which likely tilts toward problems a text-harness edit can fix). The authors are clear that accepted edits may reflect benchmark-specific failure patterns rather than general improvement, and they don’t demonstrate transfer to code generation, planning, or web tasks. The acceptance gate is also just pass-rate non-regression; higher-stakes changes would need stronger criteria around runtime, safety, and side effects. And the whole loop leans on verifier quality, a noisy evaluator would propagate bad accept/reject calls.

Still, the direction is the interesting part. Harness engineering, today a manual craft redone for every new model, becomes a recorded, testable, reversible process the agent runs on itself. When the next model ships with its own behavioral quirks, you may not need to rewrite its scaffolding by hand. You hand it the traces and let it patch itself.