AI Bug Fixers Learn to Write Their Own Tests Before Trusting a Patch

A test writer that must fail on the bug and pass after the fix

The test writer has one hard job. It must write a check that breaks on the broken code and works after the correct repair. That is not enough. The check also has to tell good fixes from bad ones. If it only fails for the wrong reason, it can send repair in a misleading direction.

The training rules make this strict. A submission with no valid test gets a small penalty of -0.2. A valid bundle that does not both fail on the buggy base and pass after the reference fix scores 0. Once that basic gate is met, quality matters. If the check separates correct from incorrect patches below 80% balanced accuracy, it earns 0.2. Between 80% and under 100%, it earns 0.5. Perfect separation earns 1.0. In plain terms, a test that merely trips on the bug is not rewarded. It must behave like a reliable probe.

The writer cannot peek at hidden solution history. It may inspect only the current working tree and public materials. It can use git status, add -AN, and a plain diff. It cannot run commands that reveal past commits or branches. Expected behavior must come from the issue text, public documentation, buggy code, and existing tests.

Before writing assertions, it must name the smallest public behavior at risk. It must also consider one plausible alternative reading of the issue before rejecting it. A clean failure on broken code is necessary but not sufficient. It only shows disagreement with the bug, not agreement with the intended fix.

The test should target something users can see. That may be an API call, a command-line result, rendered output, or a migration step. It should avoid private internal state just because that state is easy to inspect. If the issue concerns rendering, the check should look at the rendered result, not an internal flag.

Exact messages, counts, and orderings may be pinned only when the issue, docs, or existing public test fixes them. Otherwise, the writer uses the weakest observable property that the issue guarantees and the buggy code violates.

The bundle must form one coherent regression group for the issue. It can add several tests and update fixtures, but it cannot smuggle in unrelated coverage. The run command must name every declared test node. A file-level or suite-level command is rejected even if it fails on the buggy base.

The writer also submits a short behavior contract. It names the public entrypoint, the trigger that exposes the bug, and the minimal corrected output being asserted. It records supporting evidence, rejected alternative hypotheses, tempting literals left unpinned, and every exact test node exercised.

Submission is mechanical. The writer writes a JSON contract and a one-line run command to temporary files. It then creates a nonempty patch containing only the intended regression-test changes. That patch is submitted in a single final bash call. Prose alone does not count.

A repair agent that uses one frozen check as local feedback

The repair agent starts with a full attempt to fix the bug. It gets up to 200 turns of work before any feedback is allowed. At that point it has produced one complete patch. That patch is not judged by the official test suite yet. Instead, it is checked against one frozen check.

A frozen check is a single automated test kept unchanged during repair. The agent does not see its source or origin. It only receives bounded execution feedback: did the patched code make that fixed check pass? This local signal is cheap. It points to whether the current patch handles the targeted behavior. It does not decide final success.

If the frozen check fails, the repair agent may revise. It can use up to five revision rounds. Each round allows 40 turns. The agent edits the same cumulative patch. Before every check, the system resets to the original buggy code and applies the full candidate again. This avoids messy chains of small edits. A source error in the patched code counts as useful failure feedback. An environment crash does not.

The reward during training follows this split. A missing or invalid patch gets zero. A usable patch that still fails the frozen check earns a small score. If it passes the frozen check but fails official evaluation, it earns more. If both pass after revision, it earns full credit. If both already pass at Round 0, it earns extra. This teaches the agent to use the local check as a compass, not as the destination.

In one evaluation setting, feedback-guided revision raised resolved rate from 68.3% to 72.6%. Among 120 repair trajectories that entered revision, the median cost was only 11 extra turns. The average was 13. Nearly half finished within ten turns. In generated-test runs, one qualified test bundle is reused across three repair runs and all their revisions. Test generation happens once per issue and source. The final verdict still comes from a fresh official evaluator. Local gate results are recorded separately. They guide revision but do not replace full verification.

How much better the pair performs when tests are trained rather than borrowed

Trained tests lift the pair far more than borrowed ones do. The clearest comparison starts with a repair agent that has not been trained on this task. When it receives no test feedback, it resolves 61.2 percent of issues. If an untrained test generator supplies its checks, performance falls to 57.3 percent. That is a drop of 3.9 points. The generated tests are noisy enough to mislead the repair process.

A stronger external source helps more. Tests from GPT-5.6-sol raise the same base repair agent to 65.3 percent, a gain of 4.1 points over no test feedback. Oracle fail-to-pass checks push it further, to 69.4 percent, an increase of 8.2 points. These borrowed tests are useful because they point at real failures. They do not yet match the best trained setup.

The biggest jump comes when both roles are trained. A repair agent trained on feedback reaches 73.5 percent with GPT-5.6-sol tests and 77.6 percent with Oracle checks. Compared with its untrained-test baseline of 64.6 percent, the Oracle result is a gain of 13 points. That gap shows that test quality matters as much as repair skill.

The trained test writer also improves the weaker repair agent. Replacing base-model tests with reinforcement-learning-trained tests raises resolution from 57.3 to 64.1 percent, an increase of 6.8 points. For the trained repair agent, the same replacement moves it from 64.6 to 72.6 percent, a gain of 8.0 points. The two improvements are not independent magic. They point in the same direction: better checks make repair more reliable.

The fully trained pair reaches 72.6 percent on SWE-bench Verified without relying on GPT-5.6 or Oracle feedback during evaluation. That is 11.4 points above the original base repair agent with no feedback, which sits at 61.2 percent. If Oracle feedback is added to the trained pair, resolution rises to 77.6 percent. The remaining five-point gap suggests that better test quality and broader coverage could still buy more progress.

Two real bugs where a generated test redirected the fix

Two real bugs show how a generated test can steer repair away from the wrong fix. In one case, Django had to build a fallback for databases without native XOR support. XOR is a logic test that checks whether an odd number of inputs are true. The fallback encoded parity incorrectly. Parity here means the even-or-odd result expected from XOR. A generated test covered three parity cases. It failed cleanly on the buggy base code. That failure was useful because it matched the issue behavior. The first repair got parity right but reused an XOR connector while rebuilding the fallback node. This caused a RecursionError inside the frozen test. After one feedback round and 23 extra turns, the repair replaced that reused XOR connector with AND. The final patch passed both the generated test and fresh official verification. The test did not just say the patch was wrong. It pointed to the broken subsystem.

A second case involved Sphinx and typed parameter parsing. Sphinx treated a comma inside dict(str, str) as a separator. That produced malformed rendered output. A generated test used a parenthesized dict type. It failed on the buggy base and passed after the reference fix. The first repair edited an unrelated Napoleon parser. The frozen test exposed the real rendering mismatch. It redirected the repair to the typed-field parser in docfields.py. After two rounds and 75 extra turns, the final patch kept the full type before the last parameter name. It no longer split at the comma. The patch passed both the generated test and fresh official verification. A bracketed-type case also worked through the same parsing path. The Sphinx patch also kept an earlier unrelated edit. It was correct but not minimal.

The low turn counts matter. They show the test gave a narrow clue. The repair did not need to search the whole codebase. It could check one suspicious path, fix it, and rerun the same frozen check. The generated test guided the repair, while a separate official evaluator gave the final verdict.

Why keeping the two roles separate is still the safer design

A shared model would look simpler. One checkpoint could write tests and repair code. It might also transfer knowledge between the two jobs. But that convenience hides a real risk. The two roles do not learn from the same signals. A test writer is rewarded for making checks that fail on broken code, pass after a correct fix, and separate good patches from bad ones. Its highest reward comes only when that separation is clean. A repairer is rewarded only when its source-only change survives an independent final check. Those goals pull in different directions. If one model plays both parts, it can start to optimize the loop instead of the bug. It may write tests that are easy for itself to satisfy. It may repair code by matching the test rather than fixing the public behavior described in the issue. That kind of coordination looks efficient at first. In practice, it can make the pair worse.

Keeping the roles separate protects the check from becoming a target. The test bundle is generated once and then held fixed while repair happens. The repairer sees only bounded execution feedback, not the hidden source or identity of the test. It cannot rewrite the check to make its patch look better. This matters because weak tests can mislead repair. A test that fails for the wrong reason can send a fix down a false path. Separate training makes it easier to keep each role honest: one side must produce reliable checks, and the other must solve the issue under those checks. The result is not a universal guarantee. It is a safer design because independence limits the ways the system can fool itself.

A check that helps only when it is honest

The strongest result is not one clever trick. It is a loop. A test writer makes one small, public-facing check. The repair agent tries to satisfy it. If the check fails, the repair agent gets a short signal and revises. That loop lifted success from 61.2 percent to 72.6 percent. Both models were trained. No outside oracle was used during evaluation.

The gain is not magic. It comes from cheap feedback. A generated test bundle can be reused across several repair attempts. Revision added only about thirteen agent turns on average. The system allowed up to five revision rounds. That matters because real software fixes cost time and compute. A local check that costs little can guide many small corrections. It does not burn a large budget.

There is still a ceiling. With stronger oracle feedback, success reached 77.6 percent. The remaining gap points to test quality and coverage. Better tests could expose more of the real behavior around a bug. More independent checks might reduce false confidence. A shared model that can both write tests and repair code would be simpler to deploy. It could also let knowledge flow between the two roles.

The obstacle is trust. Weak or misaligned tests can steer repair away from the issue. The test writer must not chase easy internal details, private state, or a symptom of the bug. It must target observable behavior that the issue actually promises. Joint training makes this harder because the two roles have different rewards and different actions. If they learn to please each other instead of solving the problem, the loop becomes self-justifying.

The opportunity is still large. A system can generate a useful check. It can run it locally, revise code, and submit for official verification. The next step may be less about raw coding ability and more about disciplined self-testing. Can a machine learn to doubt its own test?