{"id":"3e79b89d-d964-42f3-bbf1-948e5556909f","arxiv_id":"2501.11833","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study applies the human mental set experiment to three LLMs, reporting that in-context examples improve exact match accuracy but that evidence for actual mental set rigidity is absent or confounded.","lead":"This paper tests whether large language models get stuck in a mental set, a psychological tendency to keep using a familiar strategy. It runs a classic math problem experiment from child psychology on three models and finds only weak, inconsistent evidence of such rigidity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mental-set conclusion rests on the 'Steps' metric, but the step counts in Table 2 are an artifact of chain-of-thought prompting, not a measure of strategy persistence.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the 'Steps' metric is confounded with the chain-of-thought prompting instruction. The paper's strongest claim appears in Section 6, where the leap from 'more than one step under CoT' to 'mental set' is made without any independent evidence that the step count reflects strategy rigidity. Table 2 is especially telling: steps are uniformly 3 across all conditions and models, so the metric carries no information about whether the model persisted with a previously successful strategy. The study would need a manipulation check showing that step counts differ by presentation order on the critical shortcut problems; no such check exists, and the aggregate data suggest it would fail. I also note the experiment is tiny (six problems per condition), no statistical analysis is provided, and the 'Steps' metric is not defined in Section 4. However, the single most load-bearing issue is not sample size or missing statistics; it is the construct validity of the dependent measure. Without a valid measure of strategy persistence, the central claim cannot be evaluated, and the paper's conclusions are unsupported. I see no additional concern that would change the verdict: the reader's rejection is warranted, so my stress-test leaves the verdict unchanged.","tokens_in":5249,"tokens_out":2329,"duration_ms":26317,"concrete_test":"Re-run the experiment with a defined step-counting rubric and a prompt control. First, define a step as a distinct arithmetic operation in the model's output. Then prompt GPT-4o, Llama-3.1-70B, and Llama-3.1-8B on the same 24 problems under four conditions: (1) few-shot only, (2) few-shot plus 'Let's think step by step', (3) few-shot plus 'solve using the most efficient method, showing minimal work', and (4) few-shot plus 'show your work' without step-by-step phrasing. Count actual arithmetic operations from parsed outputs. If step counts remain near 3 under all CoT-like conditions regardless of the instruction, the step count is a prompt artifact. Additionally, test the order effect directly: compare steps on shortcut problems in SP vs SP+CPF. If steps on shortcut items do not differ by presentation order, the paper cannot claim mental set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6 — that LLMs require more than one step to achieve higher accuracy, reflecting mental sets — is built entirely on the contrast between Table 1 and Table 2. Under few-shot prompting, successful solutions take a variable, often single, number of steps; under few-shot plus zero-shot chain-of-thought (FS+CoT), every model takes exactly 3 steps in every condition. The paper interprets this increase as a 'mental set shift induced by CoT reasoning.' This interpretation is load-bearing and it fails because zero-shot CoT explicitly instructs the model to reason step by step. The step count is therefore determined by the prompting instruction, not by the model's strategy choice. The paper provides no definition of what constitutes a 'step' (Section 4 merely says 'Steps, which tracks the number of steps taken'), no validation that 3 steps correspond to an entrenched multi-step strategy rather than a verbose articulation of the same one-step solution, and no manipulation check. Critically, the design should show an order effect: shortcut problems encountered after complex problems should exhibit more steps if a mental set persists. Yet Table 2 shows identical step counts across SP, SP+CPF, CP, and CP+SPF for all models. The data thus fail to provide any evidence of mental set; they simply reflect prompt formatting. Without a valid measure of strategy persistence, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript adapts DeCaro's (2016) mathematical-equivalence problems to test whether large language models (Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, GPT-4o) exhibit mental sets, defined as persistence with previously successful strategies. Three models are tested under few-shot (FS) and few-shot plus zero-shot chain-of-thought (FS+CoT) prompting across four conditions: shortcut problems alone (SP), shortcut problems after complex problems (SP+CPF), complex problems alone (CP), and complex problems after shortcut problems (CP+SPF). The reported metrics are Exact Match (EM) and a 'Steps' count. The paper concludes that LLMs require more than one step to achieve higher accuracy under FS+CoT, which it interprets as evidence of mental sets, and claims to be the first to integrate cognitive psychology concepts into LLM evaluation.","tokens_in":5504,"tokens_out":5854,"duration_ms":54620,"significance":"If the central claim were sound, the paper would introduce a genuinely new evaluation dimension for LLM reasoning, and the adapted equivalence-problem paradigm would be a useful test bed for cognitive rigidity in models. The matching of numerical values across complex and shortcut problems is a good control, and the authors report low scores honestly rather than selecting only favorable cases. However, the core evidence is not credible: the 'Steps' metric is undefined and appears to measure prompt-induced verbosity rather than strategy persistence; the order manipulation yields no interference on shortcut problems, which is the defining prediction of a mental set; and the sample sizes (six items per condition) are too small for the strong claims made. The paper also provides no prompts or code, so the results are not reproducible. The framing of the study is interesting, but the current manuscript does not establish its conclusion.","major_comments":[{"comment":"The central conclusion in §6 ('LLMs require more than one step to achieve higher accuracy, reflecting presence of mental sets') rests entirely on the 'Steps' metric, yet the paper never defines what counts as a step. Section 4 only says that Steps 'tracks the number of steps taken to arrive at the correct solution.' Table 2 shows exactly 3 steps for every model, every condition, and every success under FS+CoT. This uniform value is most plausibly an artifact of the zero-shot chain-of-thought instruction to reason step by step, not a measure of an entrenched cognitive strategy. Without a definition, example outputs, or a manipulation check showing that the model persisted with a previously successful method, the step count cannot support the mental-set interpretation.","section":"§4, Table 2"},{"comment":"A mental set produces an order effect: shortcut problems should be solved less efficiently (or at least not better) after complex problems than when presented alone. The data show the opposite. In Table 1, GPT-4o achieves EM 0.66 in SP+CPF versus 0.33 in SP; in Table 2, Llama-3.1-8B and Llama-3.1-70B both achieve EM 0.66 in SP+CPF versus 0.50 in SP. Step counts are 1 in all successful FS conditions and 3 in all successful FS+CoT conditions, with no difference between SP and SP+CPF. These results contradict the predicted interference and thus fail to provide any evidence of mental-set rigidity.","section":"§5, Tables 1 and 2"},{"comment":"The sentence 'Incorporating in-context examples, whether under the CPF or SPG conditions, boosts EM scores, except for the SP+CPF condition with the Llama-3.1-8b-instruct model using FS prompting' is not supported by the tables. Under FS+CoT, SP+CPF is never worse than SP for any model; under FS, only Llama-3.1-8B shows a decrease (0.16 to 0), the opposite of a boost. More importantly, the accompanying claim that in-context examples 'do not influence the number of steps' depends on the undefined Steps metric, so the negative result carries no weight.","section":"§5"},{"comment":"Each condition is based on only six problems, with no repeated runs, no confidence intervals, and no significance testing. Differences such as Llama-3.1-70B moving from EM 0 in CP to 0.66 in CP+SPF under FS prompting, or GPT-4o moving from 0.33 to 0.83, are reported as substantive findings. With n=6 and no stochastic reporting, these differences cannot be distinguished from sampling noise, and the paper should state this limitation explicitly rather than drawing conclusions from point estimates alone.","section":"§5, Tables 1 and 2"}],"minor_comments":[{"comment":"The text refers to 'CPF or SPG conditions,' but 'SPG' is not defined anywhere; the intended abbreviation appears to be 'SPF'.","section":"§5"},{"comment":"The captions mention 'shortcut problems (Sh),' but the table columns use 'SP'; please standardize the abbreviations.","section":"Tables 1 and 2"},{"comment":"The claim to be 'the first study to integrate cognitive psychology concepts into the evaluation of LLMs' is not established by the related-work section, which does not discuss prior work on cognitive biases or rigidity in LLM evaluation. This novelty claim should be softened or supported by an appropriate literature review.","section":"Abstract and §1"},{"comment":"The exact prompts, temperature settings, decoding parameters, and the procedure for parsing the final answer to compute Exact Match are not reported, which limits reproducibility. The CoT prompt is particularly important because the step count may be directly dictated by its instructions.","section":"§4"},{"comment":"The column labeled 'Input' and 'Output' is clear, but the Problem Type entries and the ordering of the two problem types are easy to confuse with the experimental conditions; consider adding a separate column for 'Shortcut type' or a clearer label.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"I recommend rejection because the load-bearing evidence for the mental-set claim is invalid: the Steps metric is undefined and confounded with the prompting instruction, and the reported data show no order-interference effect, which is the core prediction of a mental set. This is not a presentation issue that can be fixed by editing; the experiment needs to be redesigned with a validated process measure, more items, counterbalancing, repeated sampling, and full prompts and outputs. The topic is worth pursuing, but this manuscript in its current form does not support its conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something new—it takes DeCaro's math equivalence paradigm and runs three LLMs through it—but the headline result, that LLMs show mental sets, is not supported by its own tables. The 'Steps' metric is the load-bearing piece and it is undefined and confounded.\n\nWhat is genuinely useful: the design is simple and reproducible. Table 3 gives the 12 problems; Tables 1 and 2 show EM and step counts for Llama-3.1-8B, Llama-3.1-70B, and GPT-4o under few-shot and few-shot+CoT. The paper is transparent enough that a reader can re-analyze the numbers without digging through code. That is credit where it is due.\n\nThe soft spots are serious. Section 4 defines Steps only as 'tracks the number of steps taken'; nowhere is a step counted. In Table 2, every model in every condition under FS+CoT shows exactly 3 steps, independent of problem type or order. That is what you would expect if the model is simply following the CoT instruction to reason step by step, not if a mental set is persisting. No manipulation check, no order effect, no validation that 3 steps reflects an entrenched strategy rather than verbose output. The paper's own data also undercut the mental-set story: in-context examples sometimes improve shortcut performance (GPT-4o SP+CPF 0.66 vs SP 0.33 under FS), where a mental set decrement should hurt. With six problems per condition and no statistics, the EM differences are also within noise.\n\nThe 'first study to integrate cognitive psychology concepts into LLM evaluation' claim is overbroad; there is prior work on cognitive biases in LLMs, although the DeCaro adaptation itself may be new.\n\nBottom line: the empirical protocol is a reasonable pilot, but the central conclusion does not follow from the measurements. A serious referee should ask for a defined step metric, a manipulation check, an order-effect analysis, and a more careful framing. The paper is worth engaging with as a starting point, not as a result. For peer review: send it out, because the question matters and the protocol is reproducible, but expect a heavy-revise or reject as stated.","headline":"The paper's new measurements on DeCaro-style math equivalence problems are real, but the central mental-set claim rests on an undefined and confounded 'Steps' metric, so the conclusion does not follow from the data.","tokens_in":6035,"tokens_out":2370,"would_cite":false,"duration_ms":24396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs show human-like mental rigidity on math problems, study finds","keywords":["mental set","cognitive rigidity","chain-of-thought","large language models","mathematical reasoning","procedural flexibility","in-context learning","evaluation metrics"],"falsifier":"Record the actual generated chains for each problem and count the number of distinct arithmetic operations; if shortcut-first and complex-first conditions produce the same operation counts, the order manipulation does not induce mental set.","tokens_in":5054,"feed_emoji":"🧠","tokens_out":4317,"duration_ms":40068,"temperature":0.7,"pith_summary":"The paper attempts to show that large language models, like humans, can fall into a cognitive \"mental set\": once a multi-step solution method works, they keep using it even when a single-step shortcut would do. To test this, the authors adapt a mathematics-equivalence experiment from cognitive psychology, presenting models with complex problems before shortcut problems or vice versa, and measuring both exact accuracy and the number of reasoning steps. They report that few-shot prompting with chain-of-thought improves accuracy but also raises step counts, concluding that LLMs need more than one step to reach higher accuracy, which they read as evidence of mental set. If right, this would mean standard benchmarks miss an important dimension of reasoning flexibility in LLMs.","feed_headline":"LLMs show human-like mental rigidity on math problems, study finds","feed_subtitle":"Chain-of-thought prompting lifts accuracy but pushes models into extra steps, a sign of entrenched strategies.","key_machinery":"The central object is the \"Steps\" metric: the number of reasoning steps the model uses before producing the correct answer, averaged over successful problems. The experimental machinery is the DeCaro equivalence-task design, with six complex and six shortcut equations of identical numbers; ordering problems (complex-first versus shortcut-first) is meant to induce or reveal entrenched strategies. The claim hinges on interpreting an increase in steps under chain-of-thought prompting as evidence of cognitive rigidity rather than as an artifact of the prompt's instruction to reason step by step.","core_discovery":"The paper claims that LLMs exhibit mental sets in mathematical reasoning: when asked to reason via chain-of-thought, all three tested models take three steps to solve problems that humans can solve in one, and this step inflation is interpreted as the models persisting with entrenched multi-step strategies rather than adopting efficient shortcuts. The authors also claim that in-context examples improve exact-match accuracy but do not change the number of steps, and that this is the first study to bring the mental-set concept from cognitive psychology into LLM evaluation for complex reasoning tasks.","pith_inferences":["The step-count evidence could be strengthened by analyzing the content of the generated chains to verify that extra steps are redundant arithmetic rather than genuine alternative strategies.","A direct test of mental set would compare shortcut-first versus complex-first order under identical prompts; the paper's tables do not clearly isolate this order effect on steps, so a reanalysis could sharpen the claim.","If mental sets are real in LLMs, then curriculum design for in-context learning could deliberately reorder examples to prevent strategy entrenchment.","The same experiment could be run with vision-language models on visually presented equations, which the authors propose as future work, to see whether mental-set effects transfer across modalities."],"forward_implications":["If chain-of-thought induces mental sets, then accuracy gains from chain-of-thought may come with an efficiency cost that current benchmarks, which score only final answers, ignore.","Evaluation protocols should include problem-order manipulations and step-efficiency metrics to capture adaptability, not just exact match.","In-context examples can raise accuracy but do not by themselves break entrenched strategies, since step counts stayed at three under chain-of-thought.","Models that solve shortcut problems in one step under few-shot prompting show that the capability exists, so the rigidity is context-dependent rather than absolute."],"supporting_citations":[{"why":"Provides the dataset and the complex-first versus shortcut-first ordering design that the experiment adapts.","marker":"DeCaro, 2016"},{"why":"Establishes the water-jug mental-set phenomenon that motivates interpreting persistence with a multi-step strategy as rigidity.","marker":"Luchins and Luchins, 1959"},{"why":"Supplies the definition of mental set and its role in problem solving, which the paper applies to LLMs.","marker":"Öllinger et al., 2008"},{"why":"Introduces chain-of-thought prompting, the technique whose step-count increase is interpreted as a mental-set shift.","marker":"Wei et al., 2022"}],"fun_headline_variants":["LLMs overthink simple math, mirroring human mental set trap","Chain-of-thought makes LLMs take 3 steps for 1-step problems","Mental set: LLMs stuck on old strategies in math reasoning","LLMs show human-like rigidity in math: extra steps, no gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion depends on the assumption that the \"Steps\" metric measures strategy persistence, not simply the model following the prompt's instruction to show step-by-step reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overthink simple math, mirroring human mental set trap","Chain-of-thought makes LLMs take 3 steps for 1-step problems","Mental set: LLMs stuck on old strategies in math reasoning","LLMs show human-like rigidity in math: extra steps, no gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2877,"prompt_tokens":878,"completion_tokens":1999,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1921}},"tokens_in":494,"tokens_out":1999,"duration_ms":14092,"temperature":1.0,"reasoning_tokens":1921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:48:17.983113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the actual generated chains for each problem and count the number of distinct arithmetic operations; if shortcut-first and complex-first conditions produce the same operation counts, the order manipulation does not induce mental set.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dataset and the complex-first versus shortcut-first ordering design that the experiment adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the water-jug mental-set phenomenon that motivates interpreting persistence with a multi-step strategy as rigidity."}],"review_version":1}