Tiny Swinging Devices May Give AI a Cheaper Way to Pay Attention

How Oscillators Turn Attention Into a Physical Settling Problem

Oscillator attention replaces a digital calculation with a physical motion problem. In an ordinary transformer, each query asks how strongly it should attend to every key. The model computes scores and then applies softmax. That step needs exponentials and careful normalization. Oscillator attention does not compute those weights directly. It lets small oscillators move until they stop.

Each key supplies a fixed anchor direction. The query’s oscillator feels a pull toward the anchors it matches well. Positive couplings make that pull stronger for better alignments. The motion is constrained to the surface of a sphere, so its length never drifts. In two dimensions, the state can be written as one angle. That makes the dynamics look like ordinary phase synchronization.

The design has one important mathematical property. An energy-like measure falls as the oscillator moves, and it has only one minimum. That removes a common danger in coupled oscillators: many competing stable states. Training can therefore use the exact final point. Hardware does not need to solve equations symbolically. It only needs time to settle.

In three or more dimensions, no single angle is enough. The oscillator points in a multidirectional state on a sphere. It cannot be reduced to several independent phase clocks. That is why the mechanism needs hardware that can hold vector orientations, not just phases.

The settling behavior was checked with numerical integration of trained language models. The check used 57,900 trajectories per model. Those came from validation sequences and repeated starting points. At dimension 2, 84.3% converged and 11.7% appeared to fail by landing near the opposite direction. At dimension 8, convergence rose to 94.9%, with 3.4% failures. At dimension 32, it reached 98.3%, leaving only 0.81% apparent failures.

This is what makes attention a physical settling problem rather than a table lookup. The network does not ask a processor for an exponential value. It asks matter to move toward the same answer that training already knows. If the hardware settles, the readout can use the final orientation as the attention weight.

Why the Missing Piece Is Directional Capacity, Not Extra Tuning

Oscillator attention leaves a gap behind on language modeling, and the evidence points to one missing ingredient: directional capacity. The mechanism forms attention from anchors that live in an oscillator space. That space has only as many independent directions as its dimension allows. A rank ceiling follows from this geometry. It is roughly one more than the oscillator dimension. Trained models get close to that ceiling, but they do not exceed it. On TinyStories, effective rank was 2.31 at dimension 2. At dimension 4 it was 3.95. At dimension 8 it was 5.79. At dimension 16 it was 9.59. The ceilings were 3, 5, 9, and 17. The models use most of the available directions, yet they still fall short of full softmax attention.

Three extra tuning ideas did not close the gap. Learned position-dependent phase offsets changed WikiText-2 perplexity by less than half a point. The average shift was about -0.42, with a standard deviation of 0.25. Position already enters through input embeddings, so rotating anchors adds little new direction. Per-head coupling gains also failed. A learned scalar gain for each head and query position moved perplexity by only about -0.06. The gains varied from roughly 0.10 to 2.68. But a scalar gain cannot enlarge the anchor subspace that controls rank. Adding heads while holding model width fixed did not help either. With four heads, oscillator attention sat near 110.7 perplexity on WikiText-2. With eight heads, it moved to about 111.1. Softmax stayed near 99 in both cases.

The rank evidence makes the diagnosis sharper. Truncating a trained softmax model at inference is much worse than training under that restriction from the start. On TinyStories, post-hoc truncation raised perplexity to about 10.6 at rank 33 and about 49.7 at rank 3. A softmax variant trained with a matched score-rank ceiling did better. Its perplexity fell smoothly as rank grew: about 9.41 at rank 2, 9.18 at rank 4, 8.96 at rank 8, and 8.67 at rank 32. The full softmax baseline was also about 8.67. Oscillator attention at dimension 2 remained about 2.24 points worse than that baseline. So score-rank limitation explains part of the gap, but not all of it.

The missing piece is therefore capacity in directions, not another knob to turn. A scalar gain can make anchors stronger or weaker. It cannot add a new axis for attention to separate patterns. A phase offset can rotate an anchor. It still lives inside the same low-dimensional space. More heads with fixed width split that space thinner instead of widening it. The likely path to parity is matching oscillator dimension to the effective rank needed by softmax on the task. Exact parity remains uncertain because the two mechanisms build weights in different ways.

Attention works like a spotlight over a sentence.

It decides which earlier tokens should influence the current one. In oscillator attention, that spotlight is built from cosine similarity between query and key directions. A simple readout step can turn this raw signal into sharper choices. The knob is an exponent called p. When p equals 1, the physical hardware produces the weights directly. Larger values of p act like cooling a mixture. They push probability mass toward the strongest matches. Smaller values flatten the distribution and let weaker matches matter more.

On keyword spotting, raising p from 1 to 4 lifted accuracy from 88.79% to 90.02%. The task used two-dimensional oscillators and no positional encoding. That gain came from suppressing faint background alignments. On TinyStories, the same change cut perplexity by 0.74 at dimension 2 and by 0.80 at dimension 8.

This sharpening changes how focused the spotlight is, not how many independent directions it can use. Before sharpening, a standard spread measure for oscillator attention was above 0.90. Softmax sat near 0.60. A value close to one means almost uniform spreading. Raising p concentrates the mass on better matches. The order of matches stays tied to cosine similarity. The limit on independent directions stays at one more than the oscillator dimension. That limit does not depend on p.

In practice, p is a software-side choice. Hardware can stop at p=1. A digital post-processing step can raise p after the physical readout. This is useful because weak alignments can dilute attention. The result gets more selective without adding new directional capacity.

Settling Time, Hardware Size, and Where Savings Appear

The physical version does not compute attention in one digital step. It lets oscillators settle toward the answer that training expects. That settling takes time, but the measured times are short. On simple edge tasks, the dimensionless settling horizon is about 5. On TinyStories, it is about 30. These are units of oscillator dynamics, not seconds or minutes. The actual energy cost depends on the material and circuit used. No absolute energy figure can be given without naming a substrate.

Settling is not just an average claim. Numerical checks followed many trajectories on trained TinyStories models. At oscillator dimension 2, 84.3% reached the target state and 11.7% looked like antipodal failures. At dimension 8, convergence rose to 94.9%, with only 3.4% apparent failures. At dimension 32, it reached 98.3%, leaving just 0.81%. Each model was tested on 57,900 trajectories from validation sequences. The higher the dimension, the fewer bad starts remain.

Hardware size follows oscillator dimension. In two dimensions, only the free oscillators count as evolving hardware elements. Anchors are fixed learned forcing points and do not add moving parts. The keyword-spotting setting needs 98 two-dimensional phase oscillators in total. A WikiText-2-style causal model at dimension 32 would need about 6,200 multi-dimensional oscillators if the substrate supports it. Two-dimensional oscillator hardware is comparatively mature. It appears in mechanical resonators, Josephson junctions, MEMS devices, and electrical LC circuits. Higher-dimensional oscillator hardware is less established. Some systems with intrinsic multidirectional order may be compatible in principle, but practical demonstrations remain open. The immediate deployment target is therefore dimension 2. Higher dimensions are currently realized in software.

Savings appear only where oscillator attention differs from softmax: forming the attention weights. All surrounding transformer computations stay identical and are excluded. The cost of writing coupling values into a physical array is also excluded because it depends on the device. Counts use a first-order convention. One exponential counts as ten multiply-adds; all other operations count as one. Moving more of the weight-forming step into hardware gives larger reductions for edge tasks. For keyword spotting and subject–verb agreement, the reduction grows from under 2× in an all-digital implementation to about 12× when only the coupling nonlinearity remains digital. TinyStories is the exception. At oscillator dimension 8 its readout dominates. The all-digital oscillator version has a reduction factor of 0.6, meaning it uses more counted operations than softmax. Once equilibration is physical, the factor is about 1.1, roughly parity. Once readout and affine sum/division are also physical, it reaches 4×.

The large edge-task figure assumes ReLU coupling. The trained models use softplus. With that nonlinearity, the operation-count reduction is much smaller, about 1.2×. These counts are first-order. Fused kernels, quantization, streaming, and device details can change them. In a physical implementation, equilibration has no digital operation count. Its cost is time.

The Promise and the Price of Physical Attention

The oscillator idea is not just another attention trick. It moves part of a transformer’s heaviest step into hardware that settles by itself. Instead of computing exponentials for every query-key pair, physical states drift toward a weighted direction. That motion can replace digital math with time. For small tasks, the savings look real. Keyword spotting and subject-verb agreement can see large operation-count reductions when the nonlinear coupling is handled physically. The numbers are first-order counts, not energy bills. Still, they point to a route where attention becomes cheaper because it stops being purely arithmetic.

The hard part is capacity. Oscillator attention has a directional ceiling set by its dimension. Trained models approach that ceiling but do not exceed it. Adding learned position offsets, per-head gains, or more heads at fixed width does not remove the language-modeling gap. The missing resource is rank, not another knob. A softmax model trained under a matching low-rank restriction loses far less than one cut after training. That comparison isolates the bottleneck: attention needs enough independent directions to represent what it sees.

This creates both opportunity and obstacle. Two-dimensional oscillator hardware is mature. Mechanical resonators, Josephson junctions, MEMS devices, and LC circuits can already support phase states. Such systems may serve narrow, low-rank tasks well. Sharpening the readout can improve accuracy without changing the ceiling. Robustness to coupling mismatch and frequency disorder adds practical confidence. But language modeling wants more directions. Higher-dimensional oscillator hardware is less established. Readout costs can erase savings if they remain digital. Masking also changes the final attention matrices, so the clean low-rank story applies before that step.

The bigger question is not whether oscillators can mimic attention. They already do. The real test is whether physical substrates can supply enough directional states, fast settling, and cheap readout at the scale language models need. If they cannot, oscillator attention remains a clever bridge for small tasks. If they can, it may turn part of inference into something closer to physics than software.