Stabilizing Backward Signals Lets Recurrent Models Use Much Longer Contexts
Recurrent language models promise a cheaper way to handle very long texts because they carry a compact memory instead of storing every previous token. Yet when trained on short passages, that memory often fails to connect distant causes and effects. The key question is whether the backward signal that teaches earlier hidden states from later mistakes can be kept alive without changing what the model predicts or how it is scored.