When a Smarter AI Teacher Makes the Student Worse at Its Own Job

The scaffolding around a model can become its real skill

An AI agent is more than its model weights. The surrounding harness can carry much of the job skill. A harness is not just a system prompt. It can add tools, worked examples, reinforced instructions, and hooks that block bad actions. It can also encode environment conventions and multi-step calculation recipes. In other words, it behaves like an editable program around the model.

That matters because enterprise work often depends on routine. Payroll auditing, budget approval, stock alerting, IoT anomaly detection, browser automation, website management, and code refactoring all have checklists that a practitioner would follow automatically. A well-built harness can supply those missing routines. It tells the agent what to compute first, which filter to apply, and when to stop. The model still reasons and acts, but the scaffold shapes its path.

The strongest sign is how much performance changes when only the harness changes. For a weaker Qwen coder model, average test success rose from 29.2% with the original baseline harness to 78.0% after automatic evolution. That is a gain of 48.8 percentage points. The weights did not change in that comparison. What changed was the surrounding program: new tools, clearer recipes, and guardrails tuned by testing proposed changes on validation tasks. Only improvements were kept.

This means apparent competence can live partly outside the model. A scaffold can encode domain knowledge as executable steps. It can turn a general-purpose model into something that behaves like a trained specialist for a narrow job.

A better scaffold lifts both the apprentice and the expert

An agent’s performance is not set only by the model inside it. The surrounding setup matters too. That setup can add tools, examples, rules, and step-by-step recipes for a job. It was treated like an editable program. Small changes were tested, and only useful ones were kept.

The effect was large for a smaller open-weight model called Qwen3-Coder-30B-A3B-Instruct. With its original setup, it succeeded on average 29.2% of the time across seven business-style tasks. After the setup evolved, success rose to 78.0%. That is a jump of 48.8 percentage points. The model did not need new weights to get much better. It needed a better environment for acting in.

The surprising part came next. A stronger expert model, Gemini 3.1 Pro Preview, was tested with that same evolved setup. It had been built around the weaker Qwen model. Its average success rose from 84.4% to 93.6%. That is an increase of 15.6 points in the reported values. The scaffold did not just patch a weak learner. It also gave the expert cleaner routes through hard tasks.

This happened because the evolved setup added concrete support, not vague advice. It included new tools, reinforced instructions, and multi-step calculation recipes. The expert used almost all of those additions in 93.6% to 100% of its runs. One recipe that Qwen originally followed in only 30.8% of attempts was used by the expert in 94.2%. A good scaffold can raise both sides. It gives the apprentice missing structure. It also lets the expert work with fewer detours.

Copying an expert’s full work can break a weaker model

The intuitive fix seems simple. If a strong AI agent solves a job well inside a customized setup, let a smaller agent copy its successful paths. That is what happened when the weaker Qwen model was fine-tuned on complete runs made by a stronger expert under an improved harness. Its average test success fell from 78.0% to 63.1%, a drop of 14.9 points. The loss was not a one-off. It appeared across all seven enterprise-style tasks, including payroll auditing, budget approval, stock alerting, IoT anomaly detection, browser automation, website management, and code refactoring. Some hits were severe. Payroll auditing fell by 29.9 points. Website management dropped 20.3 points. Browser automation lost 16.0 points. Even the smallest decline, in anomaly detection, was still 4.2 points.

The surprise is that the expert was genuinely better in the same environment. Imitating a stronger model usually helps. Here it hurt because the copied material included more than facts. It carried the expert’s way of moving through a harness that had been shaped around particular tools, checks, and calculation recipes. The weaker model learned some of that behavior, but not enough to keep its own planning stable. A second check with Gemma showed the same pattern on a web-automation task. Its success under an evolved harness rose from 46.7% to 55.6%, yet after copying expert trajectories it fell to 41.1%. That is 14.5 points below its own improved-harness baseline and 5.6 points below the original default setup. The damage was not caused by ignoring the new scaffolding. The weaker models still used many added components. What broke was their ability to turn copied expert behavior into reliable step-by-step work.

The damage shows up as lost rhythm, not missing facts

When the weaker model is trained on successful expert runs, its failures do not look like blank memory. It still knows the job. It can pull up the right tools. It can follow the added rules. What breaks is timing. The surrounding setup was tuned to a short loop: compute, check once, finish. After imitation, the model keeps working past that point. In one payroll audit, it produced correct department totals. The scorer even saw the right numbers. Then it re-issued the same view 29 times. It reread the prompt about 20 times. Tool errors piled up. It never sent a finish signal across 78 steps. The rollout scored zero. The facts were there. The rhythm was not.

Zero-score rollouts were sorted into six kinds of trouble. Imitation did not wipe out domain sense. Implicit-knowledge slips fell from 46.2% to 44.5% among hard failures. That small drop shows the model learned some expert habits. The damage concentrated around missing implied steps and broken plans.

A second case made the same point. A web task needed a filtered count of approved reviews. The correct total was 346. The base model used the filter, read the right number, and finished. After imitation, it copied the expert’s quick style but not its competence. It reached the review grid. It never formed the sub-goal to filter first. It read the unfiltered count of 351 records and stopped there. On other similar questions, the same shortcut appeared. The model had learned brevity without the steps that made brevity safe.

The fine-tuned model did not ignore the added setup. It used a key calculation recipe in 76.1% of rollouts, up from 30.8%. The problem was not tool use.

The scaffolding around a model can become its real skill

A large language model is often treated as a single object: a set of weights that either knows something or does not. The surrounding system, the harness, gets less attention. It includes the prompt, the tools it can call, examples it sees, rules about when to stop, and checks on its actions. That framing misses what actually happens in practice.

The work tested this idea across seven enterprise-style tasks: payroll auditing, budget approval, stock alerting, IoT anomaly detection, browser automation, website management, and code refactoring. Each task had a clear right answer. The models were not asked to be creative. They were asked to do a job correctly.

A weaker open-weight model, Qwen3-Coder-30B-A3B-Instruct, started with an average success rate of 29.2 percent under the original harness. That is a low number for a model with thirty billion parameters. But the harness was not fixed. It was treated as an editable program. Proposed changes were tested against validation data. Only changes that improved performance were kept.

The result was not a small bump. Average test success rose to 78.0 percent, an increase of 48.8 percentage points. The model did not change its weights at all. What changed was the scaffolding around it: new tools, worked examples, reinforced instructions, hooks that blocked bad actions, and multi-step calculation recipes.

This matters because it reframes what a model is good at. A model’s skill is not only in its parameters. It is also in the structure of the task as presented to it. The harness can supply conventions the model would otherwise have to infer on its own. It can enforce a sequence of steps that keeps the agent from wandering. It can make implicit domain knowledge explicit.

The evolved harness was not just a patch for one weak model. When the stronger Gemini 3.1 Pro Preview used the same evolved scaffold, its average success rose from 84.4 percent to 93.6 percent. The expert did not ignore the scaffolding. It triggered nearly all of the added components in 93.6 to 100 percent of its rollouts. One multi-step domain-computation recipe that the base Qwen model used in only 30.8 percent of rollouts was used by the expert in 94.2 percent.

The harness became a shared asset. It encoded not just what to do, but how to do it in a way both models could follow. That is a different kind of capability than raw model strength. It is procedural knowledge made external and reusable.