{"paper":{"title":"Learning State-Tracking from Code Using Linear RNNs","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Linear RNNs track states from code traces of permutation compositions where Transformers fail.","cross_cats":["cs.CL"],"primary_cat":"cs.LG","authors_text":"Babak Rahmani, Hitesh Ballani, Julien Siems, Kirill Kalinin, Korbinian P\\\"oppel, Riccardo Grazzi","submitted_at":"2026-02-16T15:07:51Z","abstract_excerpt":"Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the REPL trace conversion of permutation composition preserves the core state-tracking difficulty without introducing artifacts that favor linear RNNs over Transformers","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Linear RNNs track states from REPL code traces of permutations better than Transformers, but non-linear RNNs outperform them in partially observable probabilistic automata.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Linear RNNs track states from code traces of permutation compositions where Transformers fail.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"986ffa43995e33653ea64d3e58a5610c716a3f469ea9eda25c837b9c0f770727"},"source":{"id":"2602.14814","kind":"arxiv","version":3},"verdict":{"id":"67f84ff7-8d9f-4f1d-8d9c-349aa9324fc5","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T21:43:00.825735Z","strongest_claim":"linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail","one_line_summary":"Linear RNNs track states from REPL code traces of permutations better than Transformers, but non-linear RNNs outperform them in partially observable probabilistic automata.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the REPL trace conversion of permutation composition preserves the core state-tracking difficulty without introducing artifacts that favor linear RNNs over Transformers","pith_extraction_headline":"Linear RNNs track states from code traces of permutation compositions where Transformers fail."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2602.14814/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}