{"id":"2436caeb-9a6e-4218-9d1d-0d8bc75a7b17","arxiv_id":"2502.10338","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs frequently generate rational plans for multi-step questions but often fail to produce answers, and a new Franklin dataset is especially hard for them.","lead":"This paper separates question-answering reasoning into planning (meta-level) and execution (object-level), and tests four large language models on four datasets, including a new one called Franklin. Human annotators found that models usually produce sensible step-by-step plans but often fail to produce any answer, especially on Franklin.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AFR measures answer presence, not correctness, so the paper's central claim that LLMs lack sufficient object-level reasoning is not established; the conclusion requires an accuracy metric over the same responses.","rationale":"The reader's weakest assumption identifies exactly the same issue: the AFR metric is treated as evidence about object-level competence, but it only measures answer presence. The paper itself calls AFR an upper bound, which is a candid admission that it is not a correctness measure. A high AFR means the model did not even attempt an answer; a low AFR does not mean the answer was right. The paper's error analysis in Section 5.3 gives examples of wrong answers but does not quantify how often they occur. Because the central claim is a negative competence claim, it cannot be supported by an upper-bound metric alone. My proposed test would close this gap by computing actual accuracy on the same responses. I find no other more severe issue: the study is transparent, the dataset is novel, and the meta-level results are presented as subjective ratings without overclaiming. The paper's own limitations (retrospective narratives, GPT-4o-mini refusal, small error sample) are acknowledged but not resolved, which is consistent with a conditional acceptance rather than rejection or acceptance. Therefore I agree with the reader's conditional verdict and do not adjust it.","tokens_in":19381,"tokens_out":3472,"duration_ms":31841,"concrete_test":"Re-score the 1,024 study-1 responses (or at least the Franklin and GSM8k subsets) for final-answer correctness against gold labels: GSM8k, StrategyQA, and HotpotQA have public answer keys, and Franklin's template-based questions have deterministic answers from the FRANK data. Compute overall accuracy and accuracy conditioned on answer presence (i.e., among responses that contain an attempt). If accuracy is high on datasets where AFR is low, the \"lack of object-level reasoning\" claim collapses to \"refusal or no-answer in some cases\"; if accuracy is comparably low, the claim is supported. This directly tests whether answer-absence is a reliable proxy for incompetence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative claim (Section 5.1: \"models did not sufficiently, or consistently, demonstrate high levels of object-level reasoning,\" and the conclusion \"LLMs lack sufficient object-level reasoning\") rests on the Answer Failure Rate, defined in Section 4.4 as the proportion of responses containing no attempted answer. The authors explicitly state they \"focus on AFR rather than a standard accuracy metric\" because they want \"an upper bound for a model's object-level reasoning ability.\" That design choice means AFR cannot distinguish between three very different states: (1) the model cannot produce any answer, (2) the model refuses to answer (e.g., GPT-4o-mini in study 2, whose AFR on Franklin jumps to 1.00 because it \"declined to execute the plan\" due to guardrails), and (3) the model produces a wrong answer, which counts as an attempt and thus lowers AFR despite no object-level reasoning being demonstrated. The paper acknowledges (2) but still includes those data in the headline conclusion. It acknowledges (3) only through a small anecdotal sample in Section 5.3 (data fabrication, inaccurate data, incorrect arithmetic). Therefore the primary metric supports only a statement about answer-absence under the specific prompt conditions, not a statement about competence. The abstract's \"struggle with object-level reasoning tasks\" and the conclusion's \"lack sufficient object-level reasoning\" require a correctness/accuracy measurement over the same responses. Without it, the central claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a re-framing of multi-step question answering in terms of meta-level reasoning (high-level planning) and object-level reasoning (execution of low-level steps such as arithmetic or retrieval). It introduces FRANKLIN, a new dataset of template-based geopolitical questions paired with step-by-step narratives labeled as meta- or object-level. Four LLMs are evaluated on FRANKLIN, GSM8k, HotpotQA, and StrategyQA through two human annotation studies. The authors report that models frequently fail to provide answers (high AFR) on FRANKLIN and some other datasets, while human annotators rate their plans as rational and step-structured at high rates (RAR/PCR). They conclude that LLMs lack sufficient object-level reasoning but can emulate meta-level reasoning.","tokens_in":19614,"tokens_out":4800,"duration_ms":44615,"significance":"The meta/object-level framing is a useful conceptual contribution, and FRANKLIN is a novel dataset that targets a realistic combination of planning and execution. The paper is transparent about many limitations, including that the step-by-step content is post-hoc and that AFR is an upper bound. If the central claims were supported, the work would strengthen the case for external execution modules in LLM QA systems. However, the primary metric does not measure correctness, so the main negative conclusion about object-level reasoning is not currently established; the meta-level conclusion also rests on subjective ratings rather than plan correctness. The dataset and framing have value regardless, and the paper is a reasonable starting point for a revised evaluation with accuracy-based metrics.","major_comments":[{"comment":"The Answer Failure Rate measures the proportion of responses containing no attempted answer, not the correctness of the answers that are present. The paper itself calls AFR an 'upper bound' for object-level reasoning ability (§4.4), yet §5.1 concludes that 'models did not sufficiently, or consistently, demonstrate high levels of object-level reasoning,' and the abstract concludes that LLMs 'struggle with object-level reasoning tasks.' These claims require an accuracy metric on the same responses: a wrong answer counts as an attempt and lowers AFR, while a refusal to answer (e.g., GPT-4o-mini in study 2, whose AFR on FRANKLIN reaches 1.00 because it 'declined to execute the plan') inflates AFR without indicating object-level incompetence. The headline conclusion should be reworded to concern answer-absence under the specific prompt conditions, or supplemented with a correctness evaluation.","section":"§4.4, Table 3, §5.1"},{"comment":"The meta-level reasoning claim rests on RAR and PCR, which are subjective annotator judgments of whether a response 'takes a rational approach' or contains a step-by-step plan. These ratings do not verify that the plan is correct, executable, or would lead to the right answer. A plan can be rational-sounding while being wrong in its decomposition or factual premises. Therefore the conclusion that LLMs 'are able to sufficiently emulate meta-level reasoning' (Section 6) is not established by the reported data. The authors should either evaluate plan quality against ground-truth plans (e.g., the FRANKLIN step templates) or at least report which rational-approach ratings were associated with correct executed outcomes.","section":"§4.4, §5.2"},{"comment":"The claim that FRANKLIN presents an object-level challenge is supported mainly by high AFR values and an anecdotal error analysis. Section 5.3 refers to 'analysis of a small sample of answers' and gives three examples (Figure 4) but no counts, coding scheme, or reliability statistics. In addition, the meta/object step labels in FRANKLIN were 'a product of an annotation task completed by the authors' (§3.2) with no inter-annotator agreement reported. The dataset's validity and the error-mode claims require systematic coding of a larger sample and report of inter-annotator agreement.","section":"§3.2, §5.3"}],"minor_comments":[{"comment":"There is a typo in the title and abstract: 'La nguage' should be 'Language'.","section":"Title/Abstract"},{"comment":"The phrase 'demonstrate a ability at this task' contains a typo; it should be 'an ability'.","section":"§3.1"},{"comment":"The row for question type D is labeled 'Region comp.' but the template in Figure 1 is 'Future prediction & comparison'; the labels should be consistent.","section":"Table 2"},{"comment":"The tables report proportions without confidence intervals or significance tests; given 64 examples per combination and 4 annotations each, adding binomial confidence intervals or inter-annotator agreement statistics would help assess the reliability of differences.","section":"Tables 3 and 4"},{"comment":"The dataset is released via an anonymized GitHub link; for reproducibility, a stable, non-anonymized repository or DOI should be provided in the final version.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The central negative claim is currently not supported by the primary metric because AFR measures answer presence rather than correctness. However, the dataset and the meta/object framing are genuine contributions, and the paper acknowledges several limitations. A revision that adds an accuracy-based evaluation on the same responses, reports plan correctness for the meta-level claim, and includes a systematic error analysis would make the paper publishable. The anonymized dataset link should also be replaced with a permanent repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper gives the community a genuinely new dataset, Franklin, and runs a careful human-annotation study across four models and four datasets. But the headline claim—that LLMs lack sufficient object-level reasoning—is not supported by the metric it uses. AFR measures the proportion of responses with no attempted answer, not the correctness of the answers that are attempted. The paper explicitly calls AFR an upper bound for object-level reasoning, then uses it to conclude that models 'struggle' with object-level reasoning. That inference does not follow.\n\nWhat is actually new: Franklin is a template-generated 400-example dataset with meta/object labels on each step, inspired by the FRANK system. That is a useful resource for evaluating planning-plus-execution behaviour. The meta/object framing itself is borrowed from 1980s-90s symbolic AI and the authors say so, which is fine. The study is transparent: they acknowledge that the step narratives are retrospective rather than planned, that GPT-4o-mini refused to execute plans, and that the error analysis rests on a small sample. That honesty matters and should be credited.\n\nThe soft spot is the one I already named, and it is load-bearing. A model that produces a confident wrong answer counts as an attempt and lowers AFR, so the metric conflates three different states: inability, refusal, and wrong-but-articulate. The GPT-4o-mini study-2 AFR of 1.00 on Franklin is a refusal, not a competence measure. On the meta side, 'rational approach' ratings tell you a plan looks sensible, not that it would lead to the correct answer. The manual error analysis is anecdotal—examples, no counts, no systematic coding. Inter-annotator agreement is reported only as pilot SEM, not for the main study.\n\nThat said, this is not a fatal flaw. It is an addressable one. The dataset and framing are worth having, and the paper would be publishable after revision with either a softened claim—'answers are frequently absent'—or a correctness re-annotation of the same responses.\n\nThe right audience: system builders who want to know where LLMs break down in multi-step QA, and researchers working on evaluation methodology. It deserves a serious referee; it is not a desk reject. But the referee should ask for the central claim to be made proportionate to the metric.","headline":"Franklin is a useful dataset and the study is transparent, but the central claim about object-level reasoning rests on an answer-presence metric that cannot support it.","tokens_in":20183,"tokens_out":2900,"would_cite":true,"duration_ms":26025,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models can plan answers to multi-step questions but often fail to carry out those plans, according to a human-annotation study of four models and four datasets.","keywords":["large language models","question answering","meta-level reasoning","object-level reasoning","multi-step reasoning","FRANKLIN dataset","human annotation","answer failure rate"],"falsifier":"Re-score every response that contains an attempted answer against the gold answer, separately for each model and dataset. If accuracy on answered responses is near ceiling wherever the answer failure rate is low, then absence of an answer is not evidence of weak object-level reasoning; if accuracy is low or errors match the paper's examples of fabrication, rounding, and arithmetic mistakes, the conclusion is supported.","tokens_in":19155,"feed_emoji":"🧠","tokens_out":7198,"duration_ms":65343,"temperature":0.7,"pith_summary":"This paper argues that multi-step question answering requires two distinguishable kinds of reasoning: meta-level reasoning, which is high-level planning and problem decomposition, and object-level reasoning, which is the low-level execution of steps such as arithmetic and precise fact retrieval. Using four LLMs, four datasets including a new dataset called FRANKLIN, and two human-annotation studies, the authors report that models produce rational, step-by-step plans at high rates, often above 95%, but regularly fail to deliver any answer, with answer failure rates on the new dataset exceeding 50% for the best model. The paper concludes that LLMs can imitate meta-level reasoning but do not consistently demonstrate sufficient object-level reasoning, and that plan-first prompting lowers answer failure for most models. If correct, this means LLM-based question-answering systems may need external execution modules for arithmetic and precise retrieval.","feed_headline":"LLMs plan well but fail to execute: QA study","feed_subtitle":"Human annotators find high rates of rational planning but frequent missing or wrong answers on multi-step questions.","key_machinery":"The paper's central machinery is the re-framing of LLM reasoning into meta-level reasoning, defined as high-level planning and problem decomposition expressed in natural language, and object-level reasoning, defined as low-level execution such as arithmetic operations, fact retrieval, and deductions. This distinction is operationalised by three human-annotation metrics: Answer Failure Rate (AFR), the proportion of responses judged to contain no attempted answer; Rational Approach Rate (RAR), the proportion judged to take a rational approach; and Plan Creation Rate (PCR), the proportion containing a clear step-by-step plan. The FRANKLIN dataset, built from four question templates about geopolitical indicators, pairs each question with template-generated steps labelled meta or object, creating a testbed where the two reasoning levels can be scored separately.","core_discovery":"The central claim is that the widely reported difficulty of LLMs on multi-step question answering is not primarily a planning failure: across four models and four datasets, annotators judged the models' proposed strategies as rational 81 to 100 percent of the time and found explicit step-by-step plans in 80 to 100 percent of responses. The failure appears at the object level: in the unassisted setting, models left questions unanswered in up to 88 percent of cases on the new FRANKLIN dataset, and when answers were attempted on FRANKLIN they contained fabricated data, inaccurate values, and incorrect arithmetic. Asking the model to produce a plan before executing it lowered answer failure rates for three of the four models, which the paper reads as evidence that the meta-level process helps while the execution step remains the bottleneck. The exception was the proprietary model, whose answer failure rose to 100 percent on FRANKLIN in the plan-first condition because it produced a plan but declined to execute it, which the paper attributes to safety guardrails rather than reasoning ability.","pith_inferences":["If the planning and execution split is real, then fine-tuning or prompting that targets arithmetic and precise factual recall is likely to yield larger gains on these datasets than training that targets strategy.","A natural extension is to measure whether object-level failure scales with the precision demanded: generate FRANKLIN variants that require two-digit versus five-to-seven-digit arithmetic and see whether answer failure tracks numeric difficulty.","Because AFR treats any attempted answer as success, the paper's own error examples imply that a correctness-weighted version of the metric would make object-level competence look even weaker.","The near-ceiling planning ratings may partly reflect annotators' familiarity with plausible-sounding strategies rather than the strategy's actual correctness; a stricter test would execute each proposed plan and check whether it reaches the right answer."],"forward_implications":["If the conclusion holds, then adding an external execution module, such as a calculator, a retrieval API, or a verifier, should improve multi-step question-answering performance more than further prompting for better plans.","Plan-first prompting is a cheap and broadly effective intervention: for most model and dataset combinations, asking for a plan before the answer reduces the rate of unanswered questions.","Answer Failure Rate should not be read as an accuracy measure, because a model can be a strong planner and still fail to answer; benchmarks that mix planning and execution hide the source of failure.","A model's refusal to execute a plan can masquerade as an object-level failure, so evaluations need to separate 'did not answer' from 'answered incorrectly'.","The FRANKLIN dataset, with its labelled reasoning steps, provides a template for generating new questions that isolate meta-level and object-level requirements."],"supporting_citations":[{"why":"Supplies the GSM8k grade-school math dataset whose arithmetic demands define part of the object-level test.","marker":"Cobbe et al. 2021"},{"why":"Supplies StrategyQA, whose implicit inference strategies require both planning and fact-level deductions.","marker":"Geva et al. 2021"},{"why":"Supplies HotpotQA, the multi-hop QA dataset requiring synthesis of facts into an answer.","marker":"Yang et al. 2018"},{"why":"Describes the FRANK system whose meta- and object-level decomposition inspires the FRANKLIN dataset.","marker":"Nuamah and Bundy 2020"},{"why":"Provides the formal distinction between meta-level and object-level reasoning that grounds the paper's framing.","marker":"Bundy 1983"},{"why":"Introduces chain-of-thought prompting, the technique whose reported benefits explain the lower AFR in the plan-first condition.","marker":"Wei et al. 2022"}],"fun_headline_variants":["LLMs plan well but stumble on execution in QA","Meta-planning strong, object execution weak: LLM QA study","Plans good, answers bad: LLMs fail on multi-step QA","LLMs rational planners, poor executors on hard QA","Strategic LLMs, arithmetic failures: Franklin dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion that LLMs lack object-level reasoning assumes that failing to produce any answer is a valid sign of inability to execute the low-level steps, even though the metric never checks whether the answers that do appear are correct.","fun_headline_variants_meta":{"raw":{"variants":["LLMs plan well but stumble on execution in QA","Meta-planning strong, object execution weak: LLM QA study","Plans good, answers bad: LLMs fail on multi-step QA","LLMs rational planners, poor executors on hard QA","Strategic LLMs, arithmetic failures: Franklin dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2039,"prompt_tokens":910,"completion_tokens":1129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1046}},"tokens_in":526,"tokens_out":1129,"duration_ms":9090,"temperature":1.0,"reasoning_tokens":1046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T18:28:26.730924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score every response that contains an attempted answer against the gold answer, separately for each model and dataset. If accuracy on answered responses is near ceiling wherever the answer failure rate is low, then absence of an answer is not evidence of weak object-level reasoning; if accuracy is low or errors match the paper's examples of fabrication, rounding, and arithmetic mistakes, the conclusion is supported.","supporting_citations":[],"review_version":1}