{"id":"e38ce8cf-fc38-4c5a-901c-f4dee58abe2a","arxiv_id":"2606.07254","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Held-out transition-pair falsifier on S3 x S3 shows projected recurrent state model achieves perfect final-state prediction up to 1M+ tokens while matched baselines fail.","lead":"The paper introduces a held-out transition-pair falsifier protocol to test sequence models on tracking non-commutative states in finite groups like S3 x S3 over extremely long horizons. A projected recurrent model achieves perfect accuracy while baselines fail, offering a controlled way to check for genuine state tracking versus memorization.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Held-out pairs may allow indirect inference of missing transitions via group closure in finite S3×S3","rationale":"The reader's weakest assumption is exactly the load-bearing point; the finite-group setting makes algebraic closure a plausible alternative explanation that the abstract does not close off. Full-text mechanism diagnostics (hard projection, commutator separation) would need to be checked against this specific test to decide whether the claim survives.","tokens_in":1774,"tokens_out":362,"duration_ms":20818,"concrete_test":"After training, feed every held-out generator pair in isolation (single transition, no preceding context) to the final projected model and measure homomorphism error on those pairs alone; if error is near zero while the matched non-projected baselines remain at chance, the long-horizon result is explained by full-table recovery rather than by the falsifier.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The protocol withholds selected ordered generator pairs from all training sequences of length 8, yet S3×S3 has only 36 elements. A recurrent model that learns a faithful representation of the group operation from the observed pairs can algebraically deduce the withheld multiplications (via inverses, associativity, and the relations already seen) and therefore apply the held-out transitions correctly at evaluation time without ever having encountered them locally. The clean-split audits confirm zero verbatim or template overlap but do not test whether the learned map is a full homomorphism on the withheld generators. If the projected model simply recovers the complete multiplication table, the perfect 250/250 scores at 1M+ horizon length no longer demonstrate that the held-out falsifier blocked all local memorization routes; they only show that the architecture can represent the group once the table is known.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces a held-out transition-pair falsifier protocol for finite non-Abelian group state tracking. In an S3×S3 benchmark, a projected recurrent state model trained exclusively on length-8 sequences achieves perfect final-state prediction accuracy (250/250 per horizon) on held-out transitions for evaluation horizons up to 1,048,576 tokens across five seeds. Matched baselines (bag, GRU, structured SSM) remain near chance under the same protocol. The protocol is supported by mechanism diagnostics (homomorphism error, state-consistency drift, commutator separation) and clean-split audits showing zero verbatim or structural-template overlap between partitions. The scope is limited to this controlled falsifier rather than general architecture ranking.","tokens_in":1933,"tokens_out":572,"duration_ms":19947,"significance":"If the falsifier protocol demonstrably prevents all memorization routes (direct and indirect), the result supplies a concrete empirical demonstration that explicit projection onto finite-group prototypes supplies a useful inductive bias for accurate long-horizon non-commutative state composition. The extreme horizon lengths and perfect per-seed accuracy constitute a strong positive signal for the architecture class under the stated conditions; the clean-split design and mechanism diagnostics are positive features of the evaluation.","major_comments":[{"comment":"Abstract and protocol description: the claim that withholding selected ordered generator pairs 'blocks one direct local-transition memorization pathway' is load-bearing for the interpretation of the perfect 250/250 scores. In the finite S3×S3 group (36 elements), a model that recovers a faithful homomorphism from the observed pairs can algebraically deduce the withheld multiplications via closure, associativity, and inverses. The clean-split audits verify zero verbatim and template overlap but do not test whether the learned map is a full homomorphism on the withheld generators; therefore the results do not yet rule out that the model simply reconstructs the complete multiplication table rather than performing genuine held-out state tracking.","section":"Abstract / protocol section"},{"comment":"Mechanism diagnostics paragraph: the reported 'low homomorphism error' under hard projection is not stated to have been evaluated on the held-out pairs themselves. If the metric is computed only on observed transitions, it does not address whether the model has inferred the withheld transitions via group structure, weakening the link between low homomorphism error and the claimed blocking of memorization routes.","section":"Mechanism diagnostics"}],"minor_comments":[{"comment":"The abstract states 'perfect 250/250 per horizon' across five seeds; the main text should report per-seed variance or confirm that every seed achieved exactly 250/250 rather than an aggregate.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful review and constructive comments on the held-out transition-pair falsifier. We respond point-by-point to the major comments below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the clean-split audits confirm absence of verbatim and template overlap but do not test whether the learned map constitutes a full homomorphism on the withheld generators. The protocol is explicitly scoped to blocking one direct local-transition memorization pathway (exposure to the specific ordered pairs), not to excluding all algebraic reconstruction routes via group closure. The perfect accuracy of the projected model versus matched baselines (including projection-equipped variants) under this split provides evidence that the projection supplies a useful inductive bias within the protocol's constraints. To address the concern, we will add an explicit evaluation of homomorphism error on the held-out pairs and report the results in the revised mechanism diagnostics section.","revision_made":"yes","referee_comment":"[Abstract / protocol section] Abstract and protocol description: the claim that withholding selected ordered generator pairs 'blocks one direct local-transition memorization pathway' is load-bearing for the interpretation of the perfect 250/250 scores. In the finite S3×S3 group (36 elements), a model that recovers a faithful homomorphism from the observed pairs can algebraically deduce the withheld multiplications via closure, associativity, and inverses. The clean-split audits verify zero verbatim and template overlap but do not test whether the learned map is a full homomorphism on the withheld generators; therefore the results do not yet rule out that the model simply reconstructs the complete multiplication table rather than performing genuine held-out state tracking."},{"response":"The referee correctly notes that the manuscript does not state the homomorphism error was evaluated on held-out pairs; the reported values were computed on observed transitions. We will revise the mechanism diagnostics paragraph to separately report homomorphism error on both observed and held-out pairs, clarifying the scope of the diagnostic and its relation to the falsifier protocol.","revision_made":"yes","referee_comment":"[Mechanism diagnostics] Mechanism diagnostics paragraph: the reported 'low homomorphism error' under hard projection is not stated to have been evaluated on the held-out pairs themselves. If the metric is computed only on observed transitions, it does not address whether the model has inferred the withheld transitions via group structure, weakening the link between low homomorphism error and the claimed blocking of memorization routes."}],"tokens_in":1555,"tokens_out":513,"duration_ms":24721,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is a held-out transition-pair protocol that withholds specific ordered generator pairs from short training sequences while requiring them at long evaluation horizons, plus clean results on S3 x S3 where a projected recurrent model hits perfect accuracy out to over a million tokens. Baselines with matched readouts stay near chance. The clean-split audits and mechanism diagnostics tying hard projection to low homomorphism error and commutator separation are useful additions.\n\nWhat the paper does well is scope the claim narrowly to this controlled finite-group case and supply explicit checks for verbatim and template overlap. The fact that softening the projection collapses performance gives some evidence the inductive bias matters. The long-horizon scale with short training is a clear strength of the benchmark design.\n\nThe soft spot is the one the stress-test note flags. S3 x S3 has only 36 elements, so a model that recovers the group operation from the observed pairs can use associativity, inverses, and closure to compute the withheld transitions without local exposure. The audits rule out direct copying but do not test whether the learned map is a full homomorphism on the missing generators. If that route remains open, the perfect scores show the architecture can represent the group once the table is known, not that the falsifier blocked every memorization pathway. The abstract does not report such a check.\n\nThis is for researchers building or using benchmarks for compositional state tracking in sequence models. A reader working on inductive biases for non-commutative composition would find the protocol worth examining. The work shows clear thinking on its own terms and the central empirical claim is falsifiable, so it deserves a serious referee even if the closure issue needs direct testing in revision. I would send it out.","headline":"The held-out pair protocol is a concrete new falsifier for non-Abelian tracking with striking S3xS3 results, but finite-group closure leaves open whether models truly avoid indirect inference of missing transitions.","tokens_in":2403,"tokens_out":431,"would_cite":false,"duration_ms":14862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A projected recurrent state model tracks non-Abelian group states perfectly over million-token horizons when trained only on length-8 sequences under a held-out transition-pair protocol.","keywords":["non-Abelian state tracking","held-out transition pairs","sequence models","finite groups","inductive bias","recurrent models","long-horizon prediction","S3 group"],"falsifier":"Observing even one error in the 250/250 final-state predictions on a held-out transition pair at a long horizon, or observing a baseline model succeeding under the identical split, would falsify the reported advantage.","tokens_in":2655,"feed_emoji":"","tokens_out":735,"duration_ms":18759,"temperature":0.7,"pith_summary":"The paper introduces a held-out transition-pair falsifier that withholds specific ordered generator pairs from training data while requiring those same pairs during evaluation to test long-horizon state tracking in non-Abelian groups. In the controlled S3 × S3 benchmark, a projected recurrent state model achieves perfect final-state predictions across all tested horizons up to 1,048,576 tokens, while GRUs, structured SSMs, and bag baselines remain near chance even when equipped with similar readouts. The protocol includes clean-split audits confirming zero overlap in reduced words or structural templates between partitions. Results show that hard projection onto group elements correlates with low homomorphism error and commutator separation, whereas softened projection leads to collapsed accuracy. The evidence applies specifically to this finite-group setup rather than general architecture comparisons.","feed_headline":"Projected model tracks non-Abelian states perfectly to 1M tokens","feed_subtitle":"Trained on length-8 sequences, it succeeds on held-out transitions where GRUs and SSMs fail across million-token horizons.","key_machinery":"The held-out transition-pair falsifier, which forbids selected ordered generator pairs from training while mandating them in evaluation to isolate non-local state composition.","core_discovery":"The held-out transition-pair falsifier blocks selected ordered generator pairs during training and requires the same local patterns during evaluation. In an S3 × S3 benchmark, a projected recurrent state model trained only on length-8 sequences produces error-free final-state predictions through evaluation horizons up to 1,048,576 tokens across five seeds, while matched native-readout baselines remain near floor and projection-matched baselines also fail.","pith_inferences":["The falsifier protocol could be extended to larger finite groups or other algebraic structures to test whether the projection bias scales.","Success on this controlled benchmark raises the question of whether similar explicit composition mechanisms would improve performance on permutation-based planning tasks.","The gap between projected and native-readout models suggests that architectural incorporation of group structure may matter more than raw capacity in non-commutative tracking."],"forward_implications":["Hard projection onto finite-group elements is necessary, since softening the projection causes final-state accuracy to collapse.","The successful model exhibits low homomorphism error, low state-consistency drift, and non-trivial commutator separation.","Explicit projected non-commutative state composition supplies an inductive bias that supports long-horizon hidden-state tracking in this regime.","Clean audits confirm zero verbatim reduced-word overlap and zero structural-template overlap between the training and evaluation partitions."],"fun_headline_variants":["Projected states track non-Abelian groups to 1M tokens","Falsifier requires held-out pairs for non-Abelian evaluation","Recurrent projection tracks states across 1M token horizons","Non-Abelian state models tested with held-out generator pairs","Projection maintains tracking in S3 benchmark to million tokens"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The held-out transition pairs during training combined with their requirement in evaluation fully block direct local-transition memorization pathways without leaving other memorization routes open.","fun_headline_variants_meta":{"raw":{"variants":["Projected states track non-Abelian groups to 1M tokens","Falsifier requires held-out pairs for non-Abelian evaluation","Recurrent projection tracks states across 1M token horizons","Non-Abelian state models tested with held-out generator pairs","Projection maintains tracking in S3 benchmark to million tokens"]},"model":"grok-4.3","cost_usd":0.00646,"raw_usage":{"total_tokens":3057,"prompt_tokens":731,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":64599500,"prompt_tokens_details":{"text_tokens":731,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2246,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":731,"tokens_out":80,"duration_ms":14241,"temperature":1.0,"reasoning_tokens":2246,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T22:46:09.437249+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing even one error in the 250/250 final-state predictions on a held-out transition pair at a long horizon, or observing a baseline model succeeding under the identical split, would falsify the reported advantage.","supporting_citations":[],"review_version":1}