{"id":"30a62ccc-2ea6-4bfb-baf9-6ffc5d9eba59","arxiv_id":"2505.24258","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark adapts eight classic program data-flow analyses into natural-language questions and finds that only a reasoning-focused LLM clearly beats random guessing.","lead":"FABLE is a new benchmark of 2,400 question-answer pairs that tests whether large language models can track how information flows through cooking recipes, travel routes, and robot plans. Initial results show a reasoning-focused model clearly outperforms general-purpose and code-focused models, but takes more than 20 times longer to answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Constant-answer cells and a uniform random baseline inflate FABLE's apparent reasoning signal; §3.4 admits 10 of 24 domain-analysis cells are constant, so rankings may reflect answer priors rather than data-flow reasoning.","rationale":"The paper's stated contribution is a diagnostic benchmark for data-flow reasoning, and its headline empirical result is that a reasoning-focused model outperforms general-purpose and code-specific models, which are near chance. Both halves depend on the test items actually exercising data-flow reasoning and on the baseline being a meaningful chance level. The paper itself discloses in §3.4 that 10 of 24 domain-analysis cells have constant answers, and then in §4 compares models to a uniform random baseline. For constant cells the correct trivial baseline is 100%, not 50%, so the 'gains over random' reported in Figure 2 are inflated and model rankings on those cells are not diagnostic. The reader's verdict already flags constant-answer analyses; my analysis agrees and sharpens it: the issue is not just that some items are easy, it is that the evaluation protocol's null model is misspecified for at least 41.7% of the benchmark. I do not see a need to move the verdict: the paper should remain CONDITIONAL, with the condition being that the authors re-report results excluding or rebalancing constant cells and against a majority-class baseline, and clarify the decoding protocol. The parser concern and the contradictory 'greedy decoding with sampling seeds' description are real but secondary; the constant-cell problem alone is sufficient to block the central empirical claim as currently stated. The authors deserve credit for releasing code/data and documenting limitations; the flaw is checkable rather than hidden.","tokens_in":22456,"tokens_out":5751,"duration_ms":75236,"concrete_test":"Using the released FABLE dataset (Hugging Face/GitHub), compute the per-domain-per-analysis answer distribution and drop every cell with zero answer variance (the 10 cells identified in §3.4, plus any others that are constant). Recompute Table 6, Figure 2 deltas, and the deepseek-vs-others comparison on the remaining non-constant questions only, and compare all models against a per-cell majority-class baseline rather than uniform random. If deepseek-r1:8b's margin over llama3.1:8b and granite-code:8b disappears or becomes non-significant on non-constant cells, or if a majority-class baseline matches or beats deepseek, then the current accuracy numbers do not support the claim that FABLE measures data-flow reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is that a large slice of FABLE does not require data-flow reasoning at all: Section 3.4 states that in travel routes, Available Expressions and Taint Analysis always answer Yes, while Concurrency, Type-State, and Very Busy Expressions always answer No; in recipes, Reaching Definitions and Available Expressions always answer Yes, Type-State and Concurrency always answer No, and Taint always answers Yes. That is at least 10 of 24 domain-analysis cells, or 1,000 of 2,400 QA pairs, with zero answer variance. For these cells the paper's uniform Yes/No random baseline has expected accuracy 50%, but the trivial baseline that always predicts the majority class achieves 100%. Figure 2 therefore credits models with up to 50 percentage points of 'gain over random' for reproducing a constant. The reported rankings are not robust: Table 6 shows granite-code:8b scoring 85% on Travel Available Expressions and 77% on Recipes Available Expressions, above deepseek-r1:8b's 64% and 75%, and even a constant-Yes strategy would beat several reported scores. This does not prove the benchmark is worthless, but it means the central empirical claim that deepseek-r1:8b demonstrates data-flow reasoning while the other models are near chance cannot be read off the current numbers. A diagnostic benchmark must include items that discriminate target reasoning from answer priors; constant cells and a uniform baseline do not provide that discrimination.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FABLE, a benchmark of 2,400 question-answer pairs that adapt eight classical data-flow analyses from software engineering to procedural text in three domains (recipes, travel routes, automated plans). Ground-truth answers are computed from entity-flow graphs derived by domain-specific parsers. The authors evaluate three 8-billion-parameter LLMs (DeepSeek-R1 8B, LLaMA 3.1 8B, Granite Code 8B) using majority voting over five completions, and report that the reasoning-focused model achieves substantially higher accuracy than the other two, which perform near a uniform random baseline. The paper claims to provide the first diagnostic benchmark for data-flow reasoning in LLMs.","tokens_in":22800,"tokens_out":5121,"duration_ms":67135,"significance":"If the validity concerns raised below are resolved, FABLE could be a useful diagnostic resource: it is extensible, openly released with code and data, and it operationalizes several distinct reasoning types in a structured way. The construction pipeline is transparent, with template-based question generation, no fitted parameters, and the ground-truth answers are derived deterministically from the underlying graphs, which is a normal and acceptable benchmark-construction strategy. However, the central empirical claim—that DeepSeek-R1 demonstrates data-flow reasoning while the other models are near chance—currently rests on an inappropriate baseline and on constant-answer cells that provide no discriminative signal. The paper also lacks a quantitative validation of the parser that underlies all ground-truth labels.","major_comments":[{"comment":"The evaluation uses a uniform Yes/No random baseline, but Section 3.4 states that at least 10 of the 24 domain-analysis cells (1,000 of 2,400 QA pairs) have constant answers: in travel routes, Available Expressions and Taint always answer Yes while Concurrency, Type-State, and Very Busy Expressions always answer No; in recipes, Reaching Definitions, Available Expressions, and Taint always answer Yes while Type-State and Concurrency always answer No. For these cells a trivial baseline that always predicts the majority class achieves 100% accuracy, not 50%. Consequently, the reported 'gains over random' in Figure 2 are inflated for any model that reproduces these constants; for example, deepseek-r1:8b scores 100% on Travel Concurrency, Travel Type-State, and Travel Very Busy Expressions, and granite-code:8b scores 85% on Travel Available Expressions. The central claim that deepseek-r1:8b demonstrates data-flow reasoning while the other models are near chance cannot be read off the current numbers, because the benchmark does not discriminate target reasoning from answer priors on a large slice of its items. The authors should re-analyze the results with a per-cell majority-class baseline, report accuracy separately for constant vs. non-constant cells, and state which conclusions survive when constant cells are excluded.","section":"Section 3.4 and Section 4 (random baseline), Table 6, Figure 2"},{"comment":"The scoring rule for Interval Analysis is not well specified. The text says a prediction is correct if 'the extracted value lies within the expected numerical range or correctly identifies the relevant step interval,' but the prompts shown in the appendix are heterogeneous: the recipes prompt asks for the last time interval exactly as it appears (a pure extraction task), the plans prompt asks for a before/after/between answer, and the travel prompt asks for a numeric interval. Mixing extraction and reasoning under a single accuracy measure makes the reported per-analysis scores—and the conclusion that Interval Analysis is the hardest—difficult to interpret. The authors should define a single, principled scoring criterion per question type, report separate results for each prompt format, and consider whether the extraction-style items belong in a data-flow reasoning benchmark.","section":"Section 4 (Interval Analysis scoring) and Appendix A.4, Figures 9, 16, 24"},{"comment":"The ground-truth answers are computed from entity-flow graphs and step-dependency graphs that are produced by domain-specific parsers. For recipes, the parser uses spaCy heuristics and a manually curated entity vocabulary; for travel routes, the authors report manually checking only 2% of routes; for plans, parsing relies on the formal PDDL structure. There is no quantitative evaluation of parser accuracy against human-annotated graphs, and the recipe QA pair review described in A.3.1 checks consistency with the same underlying representation rather than against an independent ground truth. If the parser misidentifies entities, dependencies, or step order, the computed answers are wrong and the resulting model rankings may reflect parser artifacts. The authors should provide a human-annotated sample of entity-flow graphs with parser agreement rates, or release the graphs so that errors can be audited.","section":"Section 3.2 and Appendix A.3.1 (parser validity)"}],"minor_comments":[{"comment":"The sentence 'These outcomes are a direct consequence of deterministic semantics in entity flow and step ordering within these procedural domains, not of annotation bias or trivial task construction' is not persuasive as written; constant-answer cells are by definition trivial for a benchmark, regardless of how they arise. The authors should acknowledge this and adjust the framing.","section":"Section 3.4"},{"comment":"The description 'five independent completions using greedy decoding, with sampling seeds randomized' is internally inconsistent: greedy decoding is deterministic. If sampling is used, the temperature and sampling parameters should be stated; if not, the phrase about randomized seeds should be removed.","section":"Section 4, Evaluation Setup"},{"comment":"The error bars are described as 'standard deviation' but it is not clear whether they are computed across the eight analyses, across the five sampled completions, or across items within an analysis. This should be clarified in the caption.","section":"Figure 2"},{"comment":"The claim 'first diagnostic benchmark to systematically evaluate data-flow reasoning' is strong and should be qualified with respect to prior benchmarks such as ProPara, OpenPI, and CREPE, which also track entity states and causal structure; the novelty should be positioned as the specific data-flow-analysis framing rather than the first evaluation of procedural reasoning.","section":"Abstract and Section 1"},{"comment":"The table lists 'Total QA Pairs Generated' as 13,424 but the paper does not explain why only 2,400 are used beyond 'balanced subset'; a brief explanation of the selection criteria (e.g., how 800 per domain were chosen from the larger pools) would improve reproducibility.","section":"Table 3"},{"comment":"The Welch t-test compares domain means per model, but with only eight data points per domain (one per analysis) the test has extremely low power; the authors acknowledge this, but the sentence 'These statistical results support the claim that LLMs tested on FABLE do not exhibit domain-specific reasoning strength' overstates what a null result from such a small sample can support.","section":"Appendix A.5, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The constant-answer-cell issue is the most serious problem: it directly affects the paper's headline comparison and the 'close to random chance' claim. The fix is re-analysis with a majority-class baseline and/or exclusion of constant cells, which is within the scope of a revision. The interval-analysis scoring and parser validation are also load-bearing and need concrete additions. The authors' GitHub/HuggingFace release is a strength and should be emphasized in any revised version. I would not reject the paper, but the empirical conclusions as currently stated are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FABLE is a real addition to the procedural-reasoning benchmark space. Adapting eight classical data-flow analyses to natural-language procedural text is a genuinely new combination; the template-based generation from entity-flow graphs makes ground-truth construction verifiable rather than hand-labeled. The related-work table is honest, the data and code are released, and the appendix gives enough detail to reproduce most of the pipeline. This is a benchmark that will plausibly get reused.\n\nThe soft spots are in the evaluation. The paper's own §3.4 admits 10 of 24 domain-analysis cells have constant answers (travel Available Expressions always Yes, travel Concurrency always No, etc.). For those 1000 QA pairs a model can hit 100% by always predicting the majority class, yet the binary baseline in the paper is uniform random, which expects 50%. That inflates every reported 'gain over random' and makes the abstract's claim—deepseek-r1:8b shows data-flow reasoning while the other models are near chance—unreadable from the current numbers. On some constant cells the weaker models actually score below even the uniform baseline (granite-code:8b at 35% on Travel Very Busy Expressions, which is always No). The ranking may survive, but it needs to be shown on the non-constant cells or against a per-cell majority baseline. There's also a reproducibility contradiction: the abstract says five sampled completions, §4 says greedy decoding with randomized seeds, and then 'no manual constraints on temperature or sampling strategy.' Those cannot all be true; the protocol needs a clear description.\n\nNone of this kills the benchmark. The construction is solid and the idea holds up; the evaluation protocol is what needs rework. I'd treat this as a conditional accept: rebalance or separate the constant cells, report a majority-class baseline, clean up the decoding description, and re-run the analysis. The qualitative conclusion might change, but at least it will be supported.\n\nThis paper is for people building or evaluating LLMs on procedural reasoning. It deserves a serious referee, and after the fixes I'd cite it. Recommend sending to review with a request for major revision.","headline":"A genuinely useful new benchmark, but the evaluation's constant-answer cells and uniform-random baseline make the central claims about model reasoning unreadable.","tokens_in":23250,"tokens_out":3953,"would_cite":true,"duration_ms":51179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FABLE is a 2,400-question benchmark that adapts eight software data-flow analyses to procedural text; its evaluation shows a reasoning-focused 8B model far outperforms general-purpose and code-specific models, which stay near random chance.","keywords":["data-flow analysis","procedural text","LLM evaluation","benchmark","entity tracking","interval analysis","reaching definitions","reasoning diagnosis"],"falsifier":"Generate a matched set of FABLE-style questions in which entity names are replaced by random tokens and step order is randomly permuted while keeping the graph-derived answers consistent; if a model's accuracy does not drop substantially, the benchmark is answerable from template surface form rather than from data-flow structure.","tokens_in":22296,"feed_emoji":"🧠","tokens_out":5762,"duration_ms":69144,"temperature":0.7,"pith_summary":"This paper argues that large language models have not been systematically tested on data-flow reasoning, the ability to track how an ingredient, location, or plan variable is created, changed, and used across the steps of a procedure. To fill that gap, it introduces FABLE, which turns eight classical program analyses into yes/no and range questions over recipes, travel routes, and automated plans. The authors report that a reasoning-focused 8B model answers well above random guessing in all three domains, while general-purpose and code-specialized 8B models stay close to chance, and the reasoning model pays for its advantage with more than 20 times slower inference. If the benchmark measures what it claims, it gives the field a reusable diagnostic for a capability that existing entity-state, causal, and temporal benchmarks do not isolate.","feed_headline":"Data-flow benchmark: reasoning model wins, others near random","feed_subtitle":"FABLE's 2,400 questions track ingredients, routes, and plan states; reasoning-focused 8B model scores ~80% while others guess.","key_machinery":"The carrying mechanism is the pair of graphs extracted by the domain-dependent parser: a step-dependency graph $G_S = (S, E_S)$ encoding ordering constraints between steps, and an entity-flow graph $G_E = (V, E_E)$ recording creation, modification, and consumption of entities. These graphs let each of the eight classical data-flow analyses be re-expressed as natural-language template questions whose answers are formally determined, rather than manually annotated. The template generator samples a balanced 2,400 question-answer pairs from a larger automatically generated pool of 13,424 candidates, with 100 pairs for each domain-analysis combination.","core_discovery":"The central claim is that data-flow reasoning over procedural text is a distinct, diagnosable capability, and that FABLE is the first benchmark to measure it systematically by adapting eight classical data-flow analyses from software engineering. Each procedure is parsed into a step-dependency graph and an entity-flow graph, and template questions are generated with answers formally determined by those graphs. The evaluation shows a reasoning-focused 8B model exceeding the random baseline by 31 to 37 percentage points across domains, while a general-purpose model and a code-specific model stay within a few points of chance. The authors also find that interval analysis is the hardest analysis for all models, and that no model shows statistically robust domain-specific strength, with the diagnostic signal living in variation across analysis types rather than across domains.","pith_inferences":["A direct stress test would be to shuffle the order of steps or replace entity names with random tokens while recomputing graph-derived answers; if accuracy does not drop sharply, the benchmark would be answerable from superficial wording rather than data-flow structure.","The same parser-template pipeline could be applied to the future domains the authors list, such as conversational dialogs and scientific protocols, making the eight analyses a reusable evaluation scaffold rather than a one-time dataset.","The absence of statistically significant domain-level differences suggests that FABLE's value is diagnostic at the analysis level, which implies that reporting only aggregate benchmark scores would hide the reasoning failures the benchmark is designed to expose.","If the graph-computed answers are valid, FABLE could serve as a fine-tuning signal: training on its 2,400 pairs and testing on newly generated procedures would show whether models learn general data-flow tracking or memorize question templates."],"forward_implications":["FABLE scores can be decomposed by analysis type and domain, so a model's specific weakness, such as interval analysis or taint analysis, can be isolated instead of averaged into a single number.","Because answers are computed from explicit graphs, new procedural domains can be added by writing a parser and reusing the same eight templates, making the benchmark extensible beyond recipes, routes, and plans.","The reported accuracy-inference trade-off means that deploying a reasoning-focused model in real-time procedural decision-making carries a more than 20x latency cost that must be weighed against its accuracy gain.","Performance near random chance for general-purpose and code-specific models suggests that broad pretraining over natural and programming language does not by itself confer data-flow reasoning."],"supporting_citations":[{"why":"Supplies the raw recipe corpus that becomes the recipes domain after filtering and entity curation.","marker":"[26]"},{"why":"Provides the routing engine that generates the turn-by-turn travel-route procedures from map data.","marker":"[25]"},{"why":"Produces ground-truth plans for the automated-plans domain from planning problems.","marker":"[6]"},{"why":"Converts grounded planning actions into fluent English step text for the plans domain.","marker":"[19]"},{"why":"Supplies the entity-extraction model used by the recipe parser to identify ingredients, tools, and intermediate products.","marker":"[8]"},{"why":"Defines entity-state tracking in procedural text, the capability FABLE extends toward full data-flow analysis.","marker":"[20]"},{"why":"Provides the causal what-if benchmark whose coverage gap motivates FABLE's broader data-flow evaluation.","marker":"[22]"},{"why":"Supplies the temporal-ordering benchmark whose human-model gap motivates FABLE's integration of temporal with state and causal reasoning.","marker":"[14]"}],"fun_headline_variants":["FABLE: data-flow reasoning defeats general and code LLMs","FABLE: reasoning LLM outscores others by 30+ points on data flow","FABLE benchmark: near-random performance from non-reasoning models","FABLE: first test for data-flow reasoning in procedural text","FABLE: reasoning model wins data-flow test, others near chance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, acknowledged in the appendix, is that the template-generated questions and their graph-computed answers actually measure data-flow reasoning rather than surface-level pattern matching; if the parser or templates let models answer from wording alone, the rankings would not reflect procedural understanding.","fun_headline_variants_meta":{"raw":{"variants":["FABLE: data-flow reasoning defeats general and code LLMs","FABLE: reasoning LLM outscores others by 30+ points on data flow","FABLE benchmark: near-random performance from non-reasoning models","FABLE: first test for data-flow reasoning in procedural text","FABLE: reasoning model wins data-flow test, others near chance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3659,"prompt_tokens":988,"completion_tokens":2671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2588}},"tokens_in":604,"tokens_out":2671,"duration_ms":24602,"temperature":1.0,"reasoning_tokens":2588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:27:15.231646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a matched set of FABLE-style questions in which entity names are replaced by random tokens and step order is randomly permuted while keeping the graph-derived answers consistent; if a model's accuracy does not drop substantially, the benchmark is answerable from template surface form rather than from data-flow structure.","supporting_citations":[{"cited_title":"Reasoning about actions and state changes by injecting commonsense knowledge","cited_arxiv_id":null,"evidence_quote":"Defines entity-state tracking in procedural text, the capability FABLE extends toward full data-flow analysis."},{"cited_title":"140,000 english recipes in computer-readable form with photos and crawl, Jul 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the raw recipe corpus that becomes the recipes domain after filtering and entity curation."},{"cited_title":"Open source routing engine for openstreetmap","cited_arxiv_id":null,"evidence_quote":"Provides the routing engine that generates the turn-by-turn travel-route procedures from map data."},{"cited_title":"spaCy: Industrial-strength Natural Language Processing in Python","cited_arxiv_id":null,"evidence_quote":"Supplies the entity-extraction model used by the recipe parser to identify ingredients, tools, and intermediate products."},{"cited_title":"TORQUE: A reading comprehension dataset of temporal ordering questions","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal-ordering benchmark whose human-model gap motivates FABLE's integration of temporal with state and causal reasoning."}],"review_version":1}