Can AI Agents Find the Hidden Levers That Steer Language Models Toward Any Topic?

Finding a Lever Inside a Language Model

A lever inside a language model is an internal direction that can nudge the model toward a target style or topic. The test asked ten models to find such levers for twenty tasks, from Portuguese speech to clinical symptoms and real-estate listings. Each agent worked in a sandbox with no network access and a sixty-minute limit per task. It could inspect candidate directions and probe them with short texts.

The search was not like reading a label off the model. A useful lever had to respond to the right examples and stay quiet on nearby ones. In the Portuguese task, one agent first looked at its top eighty candidates from eight passages. The feature it eventually chose was not there. That mattered. The first list favored directions that looked strong on a few examples. It appeared only after the search widened to the top five hundred and then compared new contrastive texts. That candidate fired on a Sunday passage in Portuguese and on a restaurant request in Portuguese. It did not fire on an English description of Portuguese. A small test made this clearer. When a short thank-you phrase was expanded into a full sentence, activation rose from 1.01 to 8.48. A bare word list produced only 2.58. The lever tracked real language use, not isolated words or cultural topics.

Separating the Right Signal from Nearby Noise

A good feature must light up on the right texts and stay quiet on nearby ones. The tests used positive examples, hard negatives, and neutral controls. A hard negative looks close to the target but should not trigger it. A neutral control is unrelated and should produce little response. The expert reference feature had the cleanest profile. It reached 99.5% overall AUROC. Its hard-negative AUROC was 99.2%. Its neutral-control AUROC was 100%. It responded to every positive text. Still, it fired on 61.5% of hard negatives and 15.0% of neutral controls. That does not make it bad. What matters is relative strength. Positive responses were much larger than control responses.

Agent features also separated target texts well, but they leaked more. Kimi K3 responded to 99.2% of positives. Its overall AUROC was 96.5%. It still gave nonzero activations on 54.8% of hard negatives and 26.7% of neutral controls. GPT-5.6 Sol caught 94.2% of positives with a 95.0% overall AUROC. GLM-5.2 caught only 91.7% of positives, with a 92.2% overall AUROC. These numbers show the tradeoff. A feature can be broadly useful and still wake up on confusable material.

The Portuguese task made the difference visible. Sol’s chosen feature was very quiet on English controls. It activated at zero on all four neutral English texts. Most hard negatives also stayed near silence. A vocabulary sheet with Lisboa, fado, obrigado, and pastel gave only 1.9 activation. That is small noise. But its positive responses were often weaker than the expert’s. On a sentence about storing a document safely, it activated at 2.6. The expert feature reached 19.8 on the same text.

Turning Activation into Actual Output

A feature is turned into output by changing the model’s hidden state during generation. At each token position, a scaled copy of the feature’s decoder row is added to that state. Greedy decoding then picks the next word from the altered computation. The nudge is small, but it is applied continuously as text unfolds.

The strength of the nudge is calibrated before final scoring. Candidate strengths are tested against an expert’s frozen scale. The first set includes 0.5, 0.75, 1, 1.25, and 1.5 times that scale. If none gives at least 90% usable outputs, smaller values are tried: 0.0625, 0.125, 0.25, and 0.375. Five held-out prompts test each candidate. The outputs must be coherent and free of repetitive loops. A keyword check scores how well the target idea appears. Among passing strengths, the best one is chosen by target-cue success rate. Ties are broken by mean cue score. If a tie remains, the smaller strength wins. That preference keeps the nudge as gentle as possible.

Once chosen, that strength is frozen for all twenty final tasks. A random-direction control uses the same size nudge but points in a meaningless direction. This checks whether the effect comes from the selected feature itself. An automated judge compares outputs. It receives an instruction plus three completions in shuffled order. It rates them twice, independently. Generation length also varies by task. Most tasks allow sixty-four tokens. Four language tasks allow one hundred twenty-eight. The cat task allows one hundred ninety-two.

The Portuguese prompts show what this can do. Without steering, the base model answers in English. Three successful directions reliably switch it into Portuguese. Other agent-selected directions either stay in English or collapse into broken repetition. One failing direction turns an HTTP explanation into a long string of repeated words. Another produces a twelve-line poem with repeated lines. Some failing outputs remain coherent English, yet they score zero on the language target because Portuguese never appears.

Side effects matter as much as success. On an arithmetic prompt, one successful feature gets the answer right but gives a confused explanation. Another is concise and clean. A third writes fluent verse but adds extra lines to a four-line poem. The judge scores both target relevance and preservation of the original instruction. A direction can preserve the task while missing the target. It can also induce the target while damaging the requested answer. That is why raw activation alone does not prove useful output.

Why General Smarts Did Not Guarantee the Best Feature

General intelligence did not settle the contest cleanly. The benchmark’s overall score tracked a separate general-intelligence index closely. But that relationship broke down when the score was split into parts. Activation selectivity showed a perfect rank correlation of 1.000 with the index. Overall score, activation rank, and causal steering all correlated at 0.800 or lower. Rank and steering both sat at 0.400. In plain terms, smarter models were better at telling target texts from controls. They were not reliably better at finding the single best feature direction.

The comparison used four models with matching index scores: Claude Opus 5, Grok 4.6, GPT-5.6 Sol, and Gemini 3.8 Flash. Their index values ran from 52 down to 47. Their benchmark totals ran from 65.41 down to 59.57. The ordering was not identical. Claude Opus 5 had the highest index but not the highest total. Gemini 3.8 Flash had a lower index than GPT-5.6 Sol yet scored higher overall. That gap shows that raw capability is only one ingredient. Another is task-specific luck or strategy.

Steering success shifted by task. Kimi K3 and GPT-5.6 Sol got positive steering scores on real estate, while Claude Opus 5 got zero there. In Spanish, Claude Opus 5 produced nonzero steering where most other agents scored zero. The cat task gave every agent the same steering outcome. When tasks were grouped, strengths changed again. Kimi K3 led professional-task steering at 45.03, ahead of Claude Opus 5’s 32.19. Claude Opus 5 led report-task steering at 27.50, far above Kimi’s 12.81. GPT-5.6 Sol was close to Kimi on professional steering at 44.04 even though its overall category score was lower.

Repeated runs added another wrinkle. Agents picked the same feature in all three runs for only about one-third to slightly more than half of tasks, depending on agent. Gemini 3.8 Flash was most consistent at 55%. GLM-5.2 was least consistent at 20%. So a model’s general smarts could predict how cleanly it separated target material from unrelated material. It did not guarantee that the same model would choose the best feature, steer the output effectively, or repeat its success.

A Narrow Door With a Wide View

The result is not simply that smart models can find better levers. It is that the lever problem has several hidden gates. A model may separate target text from noise well and still miss the exact direction that changes output. It may find a strong direction and still produce broken or off-target answers. The results show this split clearly.

General intelligence tracks one part of the job very tightly. Activation selectivity matched the external intelligence ranking perfectly among four frontier models. Overall performance was also strongly aligned. But causal steering and dictionary-wide ranking fell much lower in correlation. In plain terms, a capable model is better at telling relevant text from irrelevant text. It is not automatically better at choosing the one internal feature that will actually move generation.

The task map reinforces this. Some agents steered real-estate prompts while others failed on them. One agent managed Spanish where most scored zero. The cat task gave everyone the same result. Strength was uneven and local. A model could lead in professional tasks and trail in report tasks. Another could be close behind overall yet match a leader on one category.

Repeatability adds another obstacle. Agents chose the same feature across three runs only about one-third to half of the time. The most consistent agent reached 55%. The least consistent reached 20%. Taking the best repeated attempt often improved steering by several points, sometimes more than ten. That gap matters if the goal is reliable control rather than a lucky hit.

The opportunities are real. If an internal direction can be found and tested, it becomes a handle for inspection. It can show what a model responds to before output appears. It can also support targeted changes, such as making a model answer in Portuguese or keeping clinical text focused on symptoms. The method gives a way to compare models not by their answers alone but by the internal features they can identify and use.

The obstacles are equally real. Steering is prompt-sensitive. A feature that works for one instruction can distort another. Some successful directions produced fluent Portuguese but repetitive poems or confused arithmetic explanations. Others ignored the target language entirely. Some failed directions collapsed into repeated words. Even clean features were not silent on every control text. Calibration, contrastive testing, and output checks are still needed.

The bigger picture is that interpretability is becoming a practical skill, not just an explanatory one. Models can be probed, ranked, and nudged through measurable internal directions. The next step is to make those discoveries stable across tasks, prompts, and runs. If a single direction can change what a model says, the question becomes how many such directions are already inside it.