Introduction: New benchmark for state tracking
StateBench tests state tracking, a capability that a long context window alone does not establish: whether a language model can keep track of information as it changes. In its introduction of the benchmark, published November 3, 2026, Ai2 reports that frontier models handle simpler updates well but lose accuracy on harder sequences as inputs grow. Recurrent and hybrid models trained directly on the tasks generalize substantially farther, although their strongest results depend on the difficulty level and memory-update design.
The benchmark separates state tracking into four levels, from recovering a value that never changes to following dependencies between variables through successive updates. That distinction makes its findings more useful than a single overall score. Recovering an initial assignment, finding the latest replacement, and computing a result through a chain of changes impose different demands.
The central finding is therefore conditional: architecture matters, but so does training. Pretrained recurrent models do not automatically solve difficult state tracking. Direct training reveals capabilities that pretraining failed to establish, while the tested transformers still fail to learn the hardest level. These are results reported by the benchmark’s authors, rather than an independent replication. The details and qualifications below come from Ai2’s StateBench introduction.
Why state tracking matters for AI agents
State tracking is the work of maintaining the current condition of an entity as new information arrives. The source identifies code reading and narrative comprehension as examples: variables are assigned and reassigned, characters move, and objects change hands. Understanding the final situation requires interpreting what those changes do.
Ai2 illustrates the difference with a knife changing hands in a mystery. If John gives Mary the knife, the update is explicit. If John instead gives Mary the item in his backpack, the reader must also remember that Edward previously put a knife there. The later event depends on an earlier state. Missing that connection changes the answer even if both passages remain available in the document.
For AI agents, the relevance is the distinction between access to a history and accurate use of that history. An extended interaction can contain information that remains fixed, information that is replaced, and information whose meaning depends on previous updates. StateBench isolates those differences rather than treating every long-document question as the same kind of memory test.
That is also the limit of the inference. StateBench evaluates controlled state-tracking problems, not complete agent workflows. Its results identify a capability relevant to agents; they do not establish how a particular deployed agent performs across all its tasks. The useful question is which kinds of changes a model can follow reliably, and how that reliability holds up as the sequence grows.
The four levels of difficulty in StateBench
StateBench’s hierarchy permits increasingly complex update operations. The authors explain the levels through variable assignments, while grounding the distinctions in Krohn–Rhodes theory, an algebraic framework for decomposing state-tracking problems into primitives.
| Level | Permitted update | What the model must track |
|---|---|---|
| R-trivial | Immutable assignment, such as
| The value first assigned to a variable |
| Aperiodic | Reassignment by value, with the second assignment occurring later:
| The most recent assigned value |
| Soluble | Self-referential updates, such as
| Changes accumulated from the current value |
| General finite | Assignment by reference, such as
or swaps | Dependencies between variables across updates |
The jump from aperiodic to soluble problems is particularly important. A model can answer an aperiodic problem by locating the latest assignment and ignoring earlier values. With self-referential updates, that shortcut is insufficient: the current result depends on carrying earlier changes forward.
General finite problems add dependencies across variables. The source uses a ball hidden under one of five cups, followed by repeated swaps, to illustrate the S₅ state-tracking problem. Finding the ball requires following how the swaps rearrange the objects. This belongs to the hardest level in the hierarchy.
To make comparisons fairer, StateBench starts with a general finite sequence and derives three easier versions. Each uses the same number of variables and distinct values, and assignments occur at the same positions. The assignment type changes. That construction helps separate the difficulty of the update operation from differences in sequence structure.
The theoretical claims also need their conditions preserved. Under standard theoretical assumptions, the authors say transformers can handle r-trivial problems robustly, and aperiodic and soluble problems to some degree, but are predicted to fail on general problems. Gated DeltaNet with a more flexible memory update can, in principle, handle all four. Those statements concern theoretical capability; the experiments test what particular models actually learn.
Evaluation of pretrained transformer models
The open pretrained evaluation covered r-trivial, aperiodic, and general finite problems. It included the 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, which adds Gated DeltaNet layers, alongside smaller models with 275 million and 2.7 billion parameters. The authors report that all were pretrained on the same data, with training context lengths no greater than 64,000 tokens. Most were tested beyond that length.
Almost all performed well on r-trivial problems. Accuracy declined on aperiodic problems as inputs lengthened, while performance on general finite problems stayed at chance. The observed weakness therefore extended beyond transformers: the pretrained models did not demonstrate the hardest capability simply by having a recurrent or hybrid architecture.
Presentation also changed the results. The 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems, yet near chance when the same underlying problems were framed as spreadsheet edits. The source does not provide a numerical breakdown for that comparison, so it supports a qualitative finding about format sensitivity rather than a precise estimate of the gap.
This matters for interpreting benchmark success. A high score on one presentation does not establish equally strong performance on another presentation of the same state updates. StateBench uses several formats for that reason. Its findings caution against treating success on a familiar task format as proof of a broadly learned state-tracking procedure.
Performance of frontier LLMs
The frontier evaluation tested Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna. Each model was asked to read a sequence of variable assignments and generate the final value of every variable.
All four performed well on r-trivial and aperiodic problems. On soluble and general finite problems, however, accuracy fell sharply as sequences grew longer. On the longest general finite problems, it dropped to 13% or lower. The supplied source does not give separate scores for each frontier model or specify the length of those longest problems, so it does not support a detailed ranking or a precise failure threshold.
The authors suggest sensitivity as one likely factor: on harder levels, more individual steps can affect the final result. A missed update has more opportunities to invalidate the answer. This is an explanation proposed by the source, not a demonstrated accounting of every model’s errors.
The contrast between levels is more informative than a broad claim that frontier models cannot track state. They handle the simpler operations well in this evaluation. Their weakness emerges when updates accumulate or depend on other variables, particularly across longer sequences. For agent evaluation, that suggests examining the structure of the required updates alongside the amount of text.
Success of recurrent and hybrid architectures
To separate the effects of pretraining from architectural constraints, the researchers trained six 275-million-parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints.
Training sequences contained up to 256 assignments. Testing extended to 64,000 assignments, or 250 times the maximum training length. These are assignment counts, distinct from the token-based context lengths reported for the pretrained evaluation.
Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems through the longest tested sequences. The transformers learned the easier levels but degraded as sequences lengthened, and never learned general finite problems in these experiments.
The hardest level requires a separate qualification. The Gated DeltaNet variant with a more flexible memory update generalized the farthest, remaining near perfect to roughly 10 times the maximum training sequence length. That is a strong result, but it is not the same as maintaining near-perfect performance through the full 250-fold extension reported for the easier levels.
Together, the experiments support two different diagnoses. For the recurrent and hybrid models, direct training established capabilities that their pretrained counterparts did not demonstrate. That points to training as a material part of the gap. For the tested transformers, direct training still did not produce success on the hardest level, consistent with the architectural limitation discussed by the authors.
The evidence consequently supports interest in recurrent and hybrid designs without establishing that every such model solves every form of state tracking. Memory-update constraints, task difficulty, training, and test length all affect the result.
Implications for future AI‑agent design
StateBench suggests that evaluating an agent’s memory should involve more than asking whether it can retrieve an old fact. Immutable facts, replacement values, accumulated changes, and dependencies between entities require different tests. A system that performs well on the first two may still fail on the latter two.
The directly trained results also make architecture and training separate design questions. Recurrent and hybrid models carry a running memory forward, and the benchmark shows that some can learn to preserve useful state across much longer sequences than those used in training. Yet their pretrained counterparts’ failures show that this potential does not automatically become a reliable capability.
A reasonable design implication is to match evaluation to the updates an agent must follow. Where the task requires accumulated changes or cross-variable dependencies, performance on simple retrieval offers incomplete evidence. Likewise, format sensitivity makes it useful to examine whether a capability survives changes in how the same underlying operations are expressed. These are implications of the reported experiments, not deployment results measured by StateBench.
The findings support further work on recurrent and hybrid approaches, but do not establish a universal architecture choice for agents. The strongest supported conclusion is narrower: explicit state-tracking training exposes substantial differences between the tested architectures that pretrained performance alone can obscure.
Open questions and future work
Several limitations remain unresolved. The source reports a new proof for Gated DeltaNet with a more restricted memory update, described as having non-negative eigenvalues. Under the paper’s fixed-precision arithmetic with rounding, that variant can solve all aperiodic tasks and certain periodic tasks, including tracking whether a repeated input has occurred an even or odd number of times.
Its exact position in the hierarchy remains open. The authors do not establish whether it can solve all soluble tasks, and questions remain about how numerical precision and combining the model with attention layers affect its capabilities. These qualifications matter because the broader label “Gated DeltaNet” covers variants whose memory updates have different constraints.
StateBench also assumes a fixed number of variables. Real interactions can introduce additional entities over time; the source gives a long conversation in which new people and objects appear as an example. The current framework does not cover that setting. Success within its fixed-variable tasks therefore leaves an important question about expanding state unanswered.
The benchmark’s contribution is a more precise account of where models break down. It separates easy updates from dependent ones, reveals the difference direct training can make, and identifies limits that persist as sequences grow. For future agent research, the unresolved challenge is to extend that precision to changing sets of entities while establishing which architectural and training choices produce reliable state tracking.
