{"id":"99cfaaa6-c170-4fca-92f9-98230b418632","arxiv_id":"2411.11061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-2 models trained on character-reversed neuroscience text perform as well on a neuroscience abstract-selection benchmark as models trained on normal text, despite higher perplexity.","lead":"Researchers trained GPT-2 models from scratch on 20 years of neuroscience papers, once in normal order and once with every character reversed, then tested them on a neuroscience abstract-selection benchmark. The reversed-text models matched or beat the normal-text models and rivaled human experts, which the authors interpret as evidence that LLMs are general pattern learners rather than human-like language processors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Character-level reversal moves BrainBench's discriminating outcome statement to the start of the input, confounding the forward-backward equivalence claim.","rationale":"The reader identified the same core weakness: the evaluation assumes character-level reversal does not alter which information is available early in the sequence, and this assumption fails for BrainBench. I agree that this is the most load-bearing concern because it directly undercuts the interpretation of the non-significant accuracy difference as evidence for general pattern learning. The paper's other issues—including the overstatement that humans were tested on backward text and the lack of an equivalence test—are real but secondary; if the positional confound is confirmed, the main conclusion is unsupported regardless of those fixes. The proposed test is a targeted way to separate positional access from format-invariant processing. It requires no new training, only a re-scoring of the existing models on isolated outcome sentences or on reordered abstracts. Because the reader already conditioned acceptance on addressing this confound, my recommendation is UNCHANGED: the verdict stays CONDITIONAL pending this control.","tokens_in":8753,"tokens_out":2512,"duration_ms":28167,"concrete_test":"For each BrainBench item, identify the sentence containing the altered outcome (the final sentence in the forward abstract and the corresponding first sentence in the backward abstract). Recompute the model's decision using Eq. 1 restricted to only that outcome sentence, for both forward- and backward-trained models at all three sizes. If forward-trained models drop to chance while backward-trained models remain above chance, the backward advantage is due to early placement of the outcome rather than format-invariant pattern learning. As a complementary check, reverse the comparison: score forward-trained models on abstracts whose outcome sentence has been moved to the beginning (keeping forward word order) and see whether their accuracy rises to match backward-trained models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs are format-invariant pattern learners rests on the non-significant accuracy difference between forward-trained and backward-trained models (Fig. 3, F(1,2)=2.77, p=.238). This comparison is confounded by the position of the disambiguating information in BrainBench. Each BrainBench item pairs an original abstract with an altered version that differs mainly in the outcome statement, typically located near the end of the forward abstract. The decision rule (Methods 4.2, Eq. 1) averages log-likelihood over the entire sequence. In the forward condition the discriminative tokens appear late, after a long shared context; in the backward condition, because text is reversed character-by-character (Methods 4.8), those same tokens appear near the beginning, so the model encounters the outcome difference immediately. Thus the two conditions do not present the same prediction problem: backward models receive early access to the very tokens that determine the correct choice, while forward models must carry information over most of the abstract. The paper does not control for this asymmetry. If early access to the outcome drives accuracy, the observed \"equivalence\" says nothing about format-invariant pattern learning; it only reflects a positional artifact. The higher perplexity of backward models (Section 2.2) further indicates that reversed text is not equally predictable, yet the benchmark decision may still favor backward models for the artifact described.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains GPT-2 models of three sizes (124M, 355M, 774M) from scratch on two decades of neuroscience literature, either in normal forward order or with all characters reversed, and evaluates them on the BrainBench benchmark, where models must choose which of two abstracts is the original. The central claim is that forward- and backward-trained models perform equally well on this task, and that this equivalence supports the view that transformer-based LLMs are general pattern-learning machines rather than human-like language processors. The paper also reports that backward-trained models have higher perplexity on validation and test items, that forward-trained models correlate more with human difficulty judgments, and that models at the largest size match or exceed human expert performance.","tokens_in":8915,"tokens_out":5267,"duration_ms":51667,"significance":"If the central claim were established, this would be an important result: it would demonstrate format-invariant predictive learning in transformers and strengthen the case that LLM success on linguistic tasks should not be interpreted as evidence for human-like mechanisms. The paper has strengths: it retrains models from scratch with domain-specific tokenizers, uses a challenging expert-level benchmark, and provides a direct forward/backward comparison. However, the main inference is currently undercut by the weak statistics and by a positional confound in the backward condition, so the significance of the result in its present form is limited.","major_comments":[{"comment":"The claim that forward- and backward-trained models 'performed equally well' rests on a non-significant ANOVA (F(1,2)=2.77, p=.238). A non-significant difference does not establish equivalence, especially when only three model sizes are available and there are no repeated training runs. The authors should report an equivalence test (e.g., TOST or a Bayesian analysis) or a sensitivity power analysis showing what effect sizes the design could detect. Without this, the central conclusion is unsupported.","section":"Section 2.1, Fig. 3"},{"comment":"The character-level reversal confounds the forward/backward comparison. BrainBench abstracts differ primarily in the outcome statement, which is typically located near the end of the forward abstract. Reversing the characters moves these discriminating tokens to the beginning of the backward input, while the decision rule (Eq. 1) averages log-likelihood over the entire sequence. Thus backward-trained models receive early access to the exact tokens that determine the correct choice, whereas forward-trained models must process the shared context first. The paper does not control for token position, so the observed equivalence may reflect a positional artifact rather than format-invariant pattern learning. An analysis that conditions on the position of the altered tokens, or that truncates the input to remove the early-outcome advantage, is needed.","section":"Methods 4.2, 4.8 and Section 2.1"},{"comment":"The backward-trained models use a separately trained tokenizer on reversed text, which produces a different vocabulary (27% neuroscience-related tokens versus 25.4% for the forward tokenizer). This tokenizer difference is a confound: the similar accuracy could be due to the tokenizer rather than to format invariance. The authors themselves hypothesize this possibility in Section 2.1 but do not test it. A control experiment using the same tokenizer for both directions, or evaluating a forward model with the backward tokenizer, would be needed to separate these factors.","section":"Methods 4.8, Section 5.1"},{"comment":"The statistical model is underspecified. The text states that model size was coded as a continuous variable and that model was treated as a within-subjects factor, but does not give the actual repeated-measures error term or the model formula used in aov(). The reported degrees of freedom (1,2) for the direction effect suggest that the comparison is effectively across only three model sizes, which provides very low power. Please clarify the design and consider a more appropriate analysis that treats items as the random factor and models the nested structure explicitly.","section":"Methods 4.9"}],"minor_comments":[{"comment":"The abstract states that models performed equally well 'eclipsing human expert performance for both forward and backward orders,' but Section 2.1 and Fig. 3 show that only the 774M models surpass human experts, while the 124M models match them. Please soften this wording to avoid overstatement.","section":"Abstract"},{"comment":"It is unclear whether the prompt 'You are a neuroscientist...' is also reversed in the backward condition or whether it is prepended after character-level reversal. Please clarify, as this affects the input distribution seen by the backward models.","section":"Methods 4.2 vs. 4.8"},{"comment":"The reported t(2)=20.848, p=0.002 appears implausible for a t-distribution with 2 degrees of freedom; please verify the statistic and the degrees of freedom.","section":"Section 2.3"},{"comment":"There is a typo in the text: 'ANOV A' should be 'ANOVA'. Additionally, the description of the error term in aov() should be explicit (e.g., Error(item/(direction*model_size))).","section":"Methods 4.9"},{"comment":"There is a typo in 'Moreover. our study' in the Discussion; it should be 'Moreover, our study'.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from the same group that created BrainBench and collected the human expert data (ref. 38), which is not a problem per se, but the human comparison should be presented carefully. The main issues are the underpowered equivalence claim and the positional confound in the backward condition; both are fixable with additional analyses or experiments, so I do not recommend rejection at this stage. The paper would also benefit from a more cautious framing of the 'general pattern learning machine' conclusion, since the current evidence does not rule out a simpler positional explanation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result here is real but narrower than the title. What's new: they train GPT-2 from scratch on neuroscience text reversed character-by-character, retrain a tokenizer on that reversed text, and evaluate on BrainBench. Backward-trained models score about as well as forward-trained ones, and the largest backward model beats the human expert average. That is a fresh data point in the impossible-language literature, which mostly used word/token reversal and retained the original tokenizer. The paper is also honest about the non-significant difference in the ANOVA and about the higher perplexity of backward models. The error-correlation analysis is a nice extra.\n\nThe soft spots are serious. First, 'perform equivalently' is not established. Six models, one per size and direction, no repeated seeds. A non-significant F with 2 denominator degrees of freedom supports nothing. They need an equivalence test, or at least more runs. Second, the abstract says 'eclipsing human expert performance for both forward and backward orders'—but humans were only tested on forward text. That is an overstatement, and it matters because the whole framing is about comparing to human experts.\n\nThird, the stress-test note about positional confounding holds up on reading. BrainBench items differ mainly in the outcome statement near the end of the forward abstract. Reversing character-by-character moves that discriminating information to the start. So the backward model gets the decisive tokens early, while the forward model has to maintain the context. The perplexity decision rule in Eq. 1 averages over the whole sequence, but the conditional probabilities at the boundary are not comparable across conditions. This is a genuine confound, not a nitpick. The equivalence result could be a position artifact. They could address it by comparing forward models on truncated abstracts where the outcome appears early, or by controlling for token positions.\n\nThe paper is still worth engaging. The observation that a model trained on reversed text can do well on a difficult domain benchmark at all is interesting, and the confound has a fix. But the central claim as stated, that LLMs are format-invariant pattern learners, is not yet supported by the data. I'd send it out; a good referee could push them to fix the equivalence test and the human comparison, and to deal with the position confound. I would not cite the equivalence claim in its current form.","headline":"A genuinely interesting training experiment, but the equivalence claim is unsupported by the statistics and is confounded by where the outcome appears in reversed text.","tokens_in":9515,"tokens_out":2212,"would_cite":false,"duration_ms":22982,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that language models trained on character-reversed neuroscience text predict experimental outcomes as well as models trained on normal text, showing they act as general pattern learners rather than human-like language…","keywords":["large language models","language model evaluation","backward text","character-level reversal","neuroscience prediction","perplexity","pattern learning","human-like language processing"],"falsifier":"Take the 200 BrainBench items and score both model types on versions with the final outcome sentence removed, or with the outcome sentence moved to the start of an otherwise forward abstract; if backward-trained models lose their parity once the decisive sentence is no longer automatically early, the equivalence is a position effect rather than evidence of format-general learning.","tokens_in":8476,"feed_emoji":"🧠","tokens_out":6486,"duration_ms":56451,"temperature":0.7,"pith_summary":"This paper asks whether large language models succeed on language tasks because they model human language or because they are flexible pattern learners. To separate the two, the authors trained GPT-2 models from scratch on twenty years of neuroscience publications in normal order and in character-reversed order, then tested both on BrainBench, a benchmark where models and human experts choose which of two versions of an abstract reports the real experimental outcome. Backward-trained models performed as well as, and at the largest size better than, forward-trained models, and both matched or exceeded human experts. The authors conclude that the forward structure of language is not needed for these models' predictive success, so their linguistic performance should not be treated as evidence for human-like processing.","feed_headline":"Backward-trained LLMs match human experts at neuroscience prediction","feed_subtitle":"Reading scientific text in reverse does not hurt LLM accuracy, suggesting these models learn patterns, not rules.","key_machinery":"The central mechanism is character-level reversal combined with retraining: a fresh byte-pair-encoding tokenizer is learned on the reversed neuroscience corpus, and GPT-2 models of three sizes are trained from scratch on the reversed sequence, preserving the next-token objective. Evaluation uses perplexity: for each BrainBench item the model chooses the version with lower perplexity, and confidence is the perplexity gap. This design isolates whether the forward direction of natural language is necessary for the model's predictive success.","core_discovery":"The paper reports that GPT-2 models trained from scratch on twenty years of neuroscience text reversed character-by-character perform as well on the BrainBench neuroscience-outcome prediction benchmark as models trained on normal text, with the largest backward model exceeding the published human-expert average. Because the reversed text violates the forward direction of human language, the authors read this as evidence that transformer language models are general pattern-learning machines: they extract useful predictive structure from any sufficiently regular input, and their success on language tasks should not be taken as evidence of human-like language mechanisms.","pith_inferences":["A testable extension: move the outcome sentence to the front of forward-order abstracts; if forward models then match backward models, the equivalence is positional rather than format-invariant.","The same forward-versus-reversal comparison could be applied to non-linguistic sequences, such as time series or protein sequences, to see whether the equivalence is a general property of next-token training or particular to text.","A caution about the benchmark: because the altered BrainBench abstracts change the outcome, a model that finds the outcome sentence early in the reversed input gets a shortcut; scoring models on result-free portions would test whether they actually track the reasoning of the abstract."],"forward_implications":["Because backward-trained models match forward-trained models, the forward direction of natural language is not necessary for accurate next-token prediction on scientific text.","At the largest model size tested, both directions exceed the published human-expert average on BrainBench, so expert-level neuroscience prediction does not require human-like language order.","Backward-trained models score lower on human-like difficulty judgments yet remain accurate, meaning human and model errors on BrainBench only weakly overlap.","The authors recommend that strong LLM performance on linguistic tasks should not be read as evidence for human-like cognitive mechanisms."],"supporting_citations":[{"why":"Supplies the BrainBench benchmark, the 200 test cases, and the human-expert comparison data that the central claim depends on.","marker":"[38]"},{"why":"Supplies the GPT-2 architecture and tokenization strategy that the models are trained from scratch on.","marker":"[33]"},{"why":"Shows that domain-specific tokenizers improve model performance, motivating the retrained tokenizer used in both forward and backward conditions.","marker":"[37]"},{"why":"Provides prior evidence that large language models can be trained on character-level reversed text, the method directly adapted here.","marker":"[32]"},{"why":"Offers the word- and token-level reversal contrast that this paper extends to character-level reversal with a retrained tokenizer.","marker":"[31]"}],"fun_headline_variants":["LLMs read backward science as well as forward","Backward text: LLMs learn patterns, not human rules","Reverse science still beats human experts for LLMs","LLMs match human experts on backward neuroscience text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that choosing the less surprising abstract measures the same predictive skill in both formats, even though character reversal relocates each abstract's decisive outcome statement to the beginning of the sequence.","fun_headline_variants_meta":{"raw":{"variants":["LLMs read backward science as well as forward","Backward text: LLMs learn patterns, not human rules","Reverse science still beats human experts for LLMs","LLMs match human experts on backward neuroscience text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1246,"prompt_tokens":796,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":388}},"tokens_in":412,"tokens_out":450,"duration_ms":116353,"temperature":1.0,"reasoning_tokens":388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:56:44.799870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 200 BrainBench items and score both model types on versions with the final outcome sentence removed, or with the outcome sentence moved to the start of an otherwise forward abstract; if backward-trained models lose their parity once the decisive sentence is no longer automatically early, the equivalence is a position effect rather than evidence of format-general learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 architecture and tokenization strategy that the models are trained from scratch on."},{"cited_title":"Matching domain experts by training from scratch on domain knowledge","cited_arxiv_id":"2405.09395","evidence_quote":"Shows that domain-specific tokenizers improve model performance, motivating the retrained tokenizer used in both forward and backward conditions."}],"review_version":1}