Pith. sign in

REVIEW 3 major objections 6 minor 30 references

FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FABLE is a 2,400-question benchmark that adapts eight software data-flow analyses to procedural text; its evaluation shows a reasoning-focused 8B model far outperforms general-purpose and code-specific models, which stay near random chance.

desk verdict A genuinely useful new benchmark, but the evaluation's constant-answer cells and uniform-random baseline make the central claims about model reasoning unreadable. read the letter →

arxiv 2505.24258 v1 pith:7Y2XWQQU submitted 2025-05-30 cs.AI

Vishal Pallagani , Nitin Gupta , John Aydin , Biplav Srivastava This is my paper
classification cs.AI
keywords data-flowanalysisproceduraltextLLMevaluationbenchmarkentitytrackingintervalreachingdefinitionsreasoningdiagnosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models have not been systematically tested on data-flow reasoning, the ability to track how an ingredient, location, or plan variable is created, changed, and used across the steps of a procedure. To fill that gap, it introduces FABLE, which turns eight classical program analyses into yes/no and range questions over recipes, travel routes, and automated plans. The authors report that a reasoning-focused 8B model answers well above random guessing in all three domains, while general-purpose and code-specialized 8B models stay close to chance, and the reasoning model pays for its advantage with more than 20 times slower inference. If the benchmark measures what it claims, it gives the field a reusable diagnostic for a capability that existing entity-state, causal, and temporal benchmarks do not isolate.

What carries the argument

The carrying mechanism is the pair of graphs extracted by the domain-dependent parser: a step-dependency graph $G_S = (S, E_S)$ encoding ordering constraints between steps, and an entity-flow graph $G_E = (V, E_E)$ recording creation, modification, and consumption of entities. These graphs let each of the eight classical data-flow analyses be re-expressed as natural-language template questions whose answers are formally determined, rather than manually annotated. The template generator samples a balanced 2,400 question-answer pairs from a larger automatically generated pool of 13,424 candidates, with 100 pairs for each domain-analysis combination.

What would settle it

Generate a matched set of FABLE-style questions in which entity names are replaced by random tokens and step order is randomly permuted while keeping the graph-derived answers consistent; if a model's accuracy does not drop substantially, the benchmark is answerable from template surface form rather than from data-flow structure.

Watch

Extended reading notes

Core claim

The central claim is that data-flow reasoning over procedural text is a distinct, diagnosable capability, and that FABLE is the first benchmark to measure it systematically by adapting eight classical data-flow analyses from software engineering. Each procedure is parsed into a step-dependency graph and an entity-flow graph, and template questions are generated with answers formally determined by those graphs. The evaluation shows a reasoning-focused 8B model exceeding the random baseline by 31 to 37 percentage points across domains, while a general-purpose model and a code-specific model stay within a few points of chance. The authors also find that interval analysis is the hardest analysis for all models, and that no model shows statistically robust domain-specific strength, with the diagnostic signal living in variation across analysis types rather than across domains.

Load-bearing premise

The load-bearing premise, acknowledged in the appendix, is that the template-generated questions and their graph-computed answers actually measure data-flow reasoning rather than surface-level pattern matching; if the parser or templates let models answer from wording alone, the rankings would not reflect procedural understanding.

Editorial extensions

If this is right

  • FABLE scores can be decomposed by analysis type and domain, so a model's specific weakness, such as interval analysis or taint analysis, can be isolated instead of averaged into a single number.
  • Because answers are computed from explicit graphs, new procedural domains can be added by writing a parser and reusing the same eight templates, making the benchmark extensible beyond recipes, routes, and plans.
  • The reported accuracy-inference trade-off means that deploying a reasoning-focused model in real-time procedural decision-making carries a more than 20x latency cost that must be weighed against its accuracy gain.
  • Performance near random chance for general-purpose and code-specific models suggests that broad pretraining over natural and programming language does not by itself confer data-flow reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct stress test would be to shuffle the order of steps or replace entity names with random tokens while recomputing graph-derived answers; if accuracy does not drop sharply, the benchmark would be answerable from superficial wording rather than data-flow structure.
  • The same parser-template pipeline could be applied to the future domains the authors list, such as conversational dialogs and scientific protocols, making the eight analyses a reusable evaluation scaffold rather than a one-time dataset.
  • The absence of statistically significant domain-level differences suggests that FABLE's value is diagnostic at the analysis level, which implies that reporting only aggregate benchmark scores would hide the reasoning failures the benchmark is designed to expose.
  • If the graph-computed answers are valid, FABLE could serve as a fine-tuning signal: training on its 2,400 pairs and testing on newly generated procedures would show whether models learn general data-flow tracking or memorize question templates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces FABLE, a benchmark of 2,400 question-answer pairs that adapt eight classical data-flow analyses from software engineering to procedural text in three domains (recipes, travel routes, automated plans). Ground-truth answers are computed from entity-flow graphs derived by domain-specific parsers. The authors evaluate three 8-billion-parameter LLMs (DeepSeek-R1 8B, LLaMA 3.1 8B, Granite Code 8B) using majority voting over five completions, and report that the reasoning-focused model achieves substantially higher accuracy than the other two, which perform near a uniform random baseline. The paper claims to provide the first diagnostic benchmark for data-flow reasoning in LLMs.

Significance. If the validity concerns raised below are resolved, FABLE could be a useful diagnostic resource: it is extensible, openly released with code and data, and it operationalizes several distinct reasoning types in a structured way. The construction pipeline is transparent, with template-based question generation, no fitted parameters, and the ground-truth answers are derived deterministically from the underlying graphs, which is a normal and acceptable benchmark-construction strategy. However, the central empirical claim—that DeepSeek-R1 demonstrates data-flow reasoning while the other models are near chance—currently rests on an inappropriate baseline and on constant-answer cells that provide no discriminative signal. The paper also lacks a quantitative validation of the parser that underlies all ground-truth labels.

major comments (3)
  1. [Section 3.4 and Section 4 (random baseline), Table 6, Figure 2] The evaluation uses a uniform Yes/No random baseline, but Section 3.4 states that at least 10 of the 24 domain-analysis cells (1,000 of 2,400 QA pairs) have constant answers: in travel routes, Available Expressions and Taint always answer Yes while Concurrency, Type-State, and Very Busy Expressions always answer No; in recipes, Reaching Definitions, Available Expressions, and Taint always answer Yes while Type-State and Concurrency always answer No. For these cells a trivial baseline that always predicts the majority class achieves 100% accuracy, not 50%. Consequently, the reported 'gains over random' in Figure 2 are inflated for any model that reproduces these constants; for example, deepseek-r1:8b scores 100% on Travel Concurrency, Travel Type-State, and Travel Very Busy Expressions, and granite-code:8b scores 85% on Travel Available Expressions. The central claim that deepseek-r1:8b demonstrates data-flow reasoning while the other models are near chance cannot be read off the current numbers, because the benchmark does not discriminate target reasoning from answer priors on a large slice of its items. The authors should re-analyze the results with a per-cell majority-class baseline, report accuracy separately for constant vs. non-constant cells, and state which conclusions survive when constant cells are excluded.
  2. [Section 4 (Interval Analysis scoring) and Appendix A.4, Figures 9, 16, 24] The scoring rule for Interval Analysis is not well specified. The text says a prediction is correct if 'the extracted value lies within the expected numerical range or correctly identifies the relevant step interval,' but the prompts shown in the appendix are heterogeneous: the recipes prompt asks for the last time interval exactly as it appears (a pure extraction task), the plans prompt asks for a before/after/between answer, and the travel prompt asks for a numeric interval. Mixing extraction and reasoning under a single accuracy measure makes the reported per-analysis scores—and the conclusion that Interval Analysis is the hardest—difficult to interpret. The authors should define a single, principled scoring criterion per question type, report separate results for each prompt format, and consider whether the extraction-style items belong in a data-flow reasoning benchmark.
  3. [Section 3.2 and Appendix A.3.1 (parser validity)] The ground-truth answers are computed from entity-flow graphs and step-dependency graphs that are produced by domain-specific parsers. For recipes, the parser uses spaCy heuristics and a manually curated entity vocabulary; for travel routes, the authors report manually checking only 2% of routes; for plans, parsing relies on the formal PDDL structure. There is no quantitative evaluation of parser accuracy against human-annotated graphs, and the recipe QA pair review described in A.3.1 checks consistency with the same underlying representation rather than against an independent ground truth. If the parser misidentifies entities, dependencies, or step order, the computed answers are wrong and the resulting model rankings may reflect parser artifacts. The authors should provide a human-annotated sample of entity-flow graphs with parser agreement rates, or release the graphs so that errors can be audited.
minor comments (6)
  1. [Section 3.4] The sentence 'These outcomes are a direct consequence of deterministic semantics in entity flow and step ordering within these procedural domains, not of annotation bias or trivial task construction' is not persuasive as written; constant-answer cells are by definition trivial for a benchmark, regardless of how they arise. The authors should acknowledge this and adjust the framing.
  2. [Section 4, Evaluation Setup] The description 'five independent completions using greedy decoding, with sampling seeds randomized' is internally inconsistent: greedy decoding is deterministic. If sampling is used, the temperature and sampling parameters should be stated; if not, the phrase about randomized seeds should be removed.
  3. [Figure 2] The error bars are described as 'standard deviation' but it is not clear whether they are computed across the eight analyses, across the five sampled completions, or across items within an analysis. This should be clarified in the caption.
  4. [Abstract and Section 1] The claim 'first diagnostic benchmark to systematically evaluate data-flow reasoning' is strong and should be qualified with respect to prior benchmarks such as ProPara, OpenPI, and CREPE, which also track entity states and causal structure; the novelty should be positioned as the specific data-flow-analysis framing rather than the first evaluation of procedural reasoning.
  5. [Table 3] The table lists 'Total QA Pairs Generated' as 13,424 but the paper does not explain why only 2,400 are used beyond 'balanced subset'; a brief explanation of the selection criteria (e.g., how 800 per domain were chosen from the larger pools) would improve reproducibility.
  6. [Appendix A.5, Table 8] The Welch t-test compares domain means per model, but with only eight data points per domain (one per analysis) the test has extremely low power; the authors acknowledge this, but the sentence 'These statistical results support the claim that LLMs tested on FABLE do not exhibit domain-specific reasoning strength' overstates what a null result from such a small sample can support.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FABLE's QA labels are computed from the same graphs used for question generation, which is standard benchmark construction, not a circular derivation.

full rationale

The paper's derivation chain is self-contained: FABLE's question–answer pairs are generated by applying template-based question templates to the same entity-flow and step-dependency graphs that also determine the formally computed answers (Section 3.4). This is normal benchmark construction rather than a circular prediction—no model output is fed back into the label-generation process, and no parameter is fitted to the evaluated models. There are no load-bearing self-citations, no 'uniqueness' argument imported from the authors' prior work, and no ansatz smuggled in via citation. The closest concern is Section 3.4's admission that ten of twenty-four domain-analysis cells always answer Yes or No; that weakens the benchmark's discriminative power and makes the uniform random baseline uninformative for those cells, but it is a validity and triviality issue, not a case where a derived result reduces to its own inputs. Appendix A.1's caveat that each data-flow adaptation reflects only one formal interpretation similarly limits construct validity without creating circularity. The empirical rankings may be confounded by constant-answer priors, but that is a correctness-risk concern rather than circular reasoning. Therefore the paper exhibits no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on four main premises: the procedural-text formalism, the adaptation of program analyses to text, parser correctness, and the validity of template-generated ground truth. None of these is independently verified in the paper; the parsers and templates are authored by the same team. No free parameters are fitted to data, and no new physical or theoretical entities are introduced.

assumptions (4)
  • domain assumption Procedural text can be represented as a triple (G, S, preceq) with a goal, steps, and order.
    Section 2.1 adopts this formalism from CREPE [27]; it is used to unify recipes, travel routes, and plans.
  • ad hoc to paper Classical data-flow analyses can be faithfully adapted to natural-language procedural text using the one interpretation per analysis chosen in Table 2.
    Section 3.4 and Appendix A.1 state the adaptation reflects only one formal interpretation per analysis; the entire benchmark rests on this mapping.
  • domain assumption The domain-dependent parsers correctly extract step-dependency and entity-flow graphs from raw text.
    Section 3.2 describes recipe parsing via spaCy heuristics, route parsing via OpenStreetMap metadata, and plan parsing via PDDL; errors would propagate to all ground-truth answers.
  • ad hoc to paper Template-generated answers computed from the graphs are valid ground-truth labels for the questions.
    Section 3.4 generates answers from the graphs; recipes were reviewed by two co-authors, but only 5% of travel QA pairs were manually validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation." pith.science (2026). https://pith.science/paper/7Y2XWQQU

@misc{pith2026250524258,
  author       = {Pith},
  title        = {Pith review of: FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7Y2XWQQU}},
  note         = {Machine review of arXiv:2505.24258}
}
read the original abstract

Understanding how data moves, transforms, and persists, known as data flow, is fundamental to reasoning in procedural tasks. Despite their fluency in natural and programming languages, large language models (LLMs), although increasingly being applied to decisions with procedural tasks, have not been systematically evaluated for their ability to perform data-flow reasoning. We introduce FABLE, an extensible benchmark designed to assess LLMs' understanding of data flow using structured, procedural text. FABLE adapts eight classical data-flow analyses from software engineering: reaching definitions, very busy expressions, available expressions, live variable analysis, interval analysis, type-state analysis, taint analysis, and concurrency analysis. These analyses are instantiated across three real-world domains: cooking recipes, travel routes, and automated plans. The benchmark includes 2,400 question-answer pairs, with 100 examples for each domain-analysis combination. We evaluate three types of LLMs: a reasoning-focused model (DeepSeek-R1 8B), a general-purpose model (LLaMA 3.1 8B), and a code-specific model (Granite Code 8B). Each model is tested using majority voting over five sampled completions per prompt. Results show that the reasoning model achieves higher accuracy, but at the cost of over 20 times slower inference compared to the other models. In contrast, the general-purpose and code-specific models perform close to random chance. FABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding.

Figures

Figures reproduced from arXiv: 2505.24258 by the authors.

Figure 1
Figure 1. Overview of the FABLE dataset construction pipeline. Raw procedural data is [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Domain-wise accuracy gains over random base [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Model-wise performance across domains and data-flow analyses in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: Prompt example for the Reaching Definitions analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Prompt example for the Very Busy Expressions analysis in the Travel Routes [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Prompt example for the Available Expressions analysis in the Travel Routes [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Prompt example for the Type-State Analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Prompt example for the Live Variable Analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Prompt example for the Interval Analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Prompt example for the Taint Analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Prompt example for the Concurrency Analysis in the Travel Routes domain. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Prompt example for the Reaching Definitions analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Prompt example for the Very Busy Expressions analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Prompt example for the Available Expressions analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Prompt example for the Live Variable Analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Prompt example for the Interval Analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Prompt example for the Type-State Analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Prompt example for the Taint Analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Prompt example for the Concurrency Analysis in the Plans domain. [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Prompt example for the Reaching Definitions analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: Prompt example for the Very Busy Expressions analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Prompt example for the Available Expressions analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 23
Figure 23. Figure 23: Prompt example for the Live Variable Analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Prompt example for the Interval Analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p028_24.png]
Figure 25
Figure 25. Figure 25: Prompt example for the Type-State Analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26: Prompt example for the Taint Analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p029_26.png]
Figure 27
Figure 27. Figure 27: Prompt example for the Concurrency Analysis in the Recipes domain. [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]
Figure 28
Figure 28. Figure 28: Inference latency heatmap for Plans domain. Color intensity indicates log-scale inference time (in seconds per 100 prompts) for each model–analysis pair. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_28.png]
Figure 29
Figure 29. Figure 29: Inference latency heatmap for Travel Routes domain. Deep models like deepseek-r1:8b incur significantly higher latency, especially on Interval Analysis. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_29.png]
Figure 30
Figure 30. Figure 30: Inference latency heatmap for Recipes domain. All models show increased latency for Interval Analysis due to longer and more complex completions. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_30.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages

  1. [1]

    Major State Government Buildings,

    California Office of Emergency Services (Cal OES). Major State Government Buildings,

    work page

  2. [2]

    pddl-generators: A collection of pddl problem generators

    AI-Planning Community. pddl-generators: A collection of pddl problem generators. https://github.com/AI-Planning/pddl-generators. Accessed: 2025-05-11

    work page 2025

  3. [3]

    Data flow analysis for verifying properties of concurrent programs.ACM SIGSOFT Software Engineering Notes, 19(5):62–75, 1994

    Matthew B Dwyer and Lori A Clarke. Data flow analysis for verifying properties of concurrent programs.ACM SIGSOFT Software Engineering Notes, 19(5):62–75, 1994

    work page 1994

  4. [4]

    Data flow analysis in software reliability.ACM Computing Surveys (CSUR), 8(3):305–330, 1976

    Lloyd D Fosdick and Leon J Osterweil. Data flow analysis in software reliability.ACM Computing Surveys (CSUR), 8(3):305–330, 1976. 10

    work page 1976

  5. [5]

    Elsevier, 2004

    Malik Ghallab, Dana Nau, and Paolo Traverso.Automated Planning: theory and practice. Elsevier, 2004

    2004

  6. [6]

    The fast downward planning system.Journal of Artificial Intelligence Research, 26:191–246, 2006

    Malte Helmert. The fast downward planning system.Journal of Artificial Intelligence Research, 26:191–246, 2006

    2006

  7. [7]

    USA Hospitals, 2019

    Homeland Infrastructure Foundation-Level Data (HIFLD). USA Hospitals, 2019. Ac- cessed: 2025-03-20

    work page 2019

  8. [8]

    spaCy: Industrial-strength Natural Language Processing in Python

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing in Python. 2020

    work page 2020

Show all 30 references
  1. [9]

    International planning competitions at international conference on automated planning and scheduling (icaps)

    ICAPS. International planning competitions at international conference on automated planning and scheduling (icaps). Inhttps://www.icaps-conference.org/competitions/, 2022

    2022

  2. [10]

    IBM Thomas J

    Ken Kennedy.A survey of data flow analysis techniques. IBM Thomas J. Watson Research Division, 1979

    1979

  3. [11]

    Using ollama

    Francisco S Marcondes, Adelino Gala, Renata Magalhães, Fernando Perez de Britto, Dalila Durães, and Paulo Novais. Using ollama. InNatural Language Analytics with Generative Large-Language Models: A Practical Approach with Ollama and Open-Source LLMs, pages 23–35. Springer, 2025

    2025

  4. [12]

    Pddl-the planning domain definition language

    Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl-the planning domain definition language. 1998

    1998

  5. [13]

    National Register of Historic Places, 2025

    National Park Service. National Register of Historic Places, 2025. Accessed: 2025-03-20

    2025

  6. [14]

    TORQUE: A reading comprehension dataset of temporal ordering questions

    Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. TORQUE: A reading comprehension dataset of temporal ordering questions. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Lan...

    2020

  7. [15]

    OpenStreetMap US Extract

    OpenStreetMap contributors. OpenStreetMap US Extract. https://www. openstreetmap.org, 2025. Data retrieved from https://www.geofabrik.de/

    2025

  8. [16]

    OurAirports USA, 2025

    OurAirports. OurAirports USA, 2025. Accessed: 2025-03-20

    2025

  9. [17]

    An incremental version of iterative data flow analysis.IEEE Transactions on Software Engineering, 15(12):1537–1549, 2002

    Lori L Pollock and Mary Lou Soffa. An incremental version of iterative data flow analysis.IEEE Transactions on Software Engineering, 15(12):1537–1549, 2002

    2002

  10. [18]

    Ai a modern approach.Learning, 2(3):4, 2005

    Stuart Russell and Peter Norvig. Ai a modern approach.Learning, 2(3):4, 2005

    2005

  11. [19]

    Autoplanbench: Automatically generating benchmarks for llm planners from pddl.arXiv preprint arXiv:2311.09830, 2023

    Katharina Stein, Daniel Fišer, Jörg Hoffmann, and Alexander Koller. Autoplanbench: Automatically generating benchmarks for llm planners from pddl.arXiv preprint arXiv:2311.09830, 2023

    2023 arXiv

  12. [20]

    Reasoning about actions and state changes by injecting commonsense knowledge

    Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. Reasoning about actions and state changes by injecting commonsense knowledge. InEllenRiloff, DavidChiang, JuliaHockenmaier, andJun’ichiTsujii, editors,Proceedings of the 2018 Conference on ...

    2018

  13. [21]

    Reasoning about actions and state changes by injecting commonsense knowledge.arXiv preprint arXiv:1808.10012, 2018

    Niket Tandon, Bhavana Dalvi Mishra, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. Reasoning about actions and state changes by injecting commonsense knowledge.arXiv preprint arXiv:1808.10012, 2018

    2018 arXiv

  14. [22]

    what if

    Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. Wiqa: A dataset for" what if..." reasoning over procedural text.arXiv preprint arXiv:1909.04739, 2019. 11

    1909 arXiv

  15. [23]

    A dataset for tracking entities in open domain procedural text

    Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. A dataset for tracking entities in open domain procedural text. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the ...

    2020

  16. [24]

    Bureau of Transportation Statistics

    U.S. Bureau of Transportation Statistics. Airports – Citizen Connect, 2025. Accessed: 2025-03-20

    2025

  17. [25]

    Open source routing engine for openstreetmap

    Valhalla Valhalla. Open source routing engine for openstreetmap

  18. [26]

    140,000 english recipes in computer-readable form with photos and crawl, Jul 2017

    Zachary Vance. 140,000 english recipes in computer-readable form with photos and crawl, Jul 2017

    2017

  19. [27]

    Causal reasoning of entities and events in procedural texts

    Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison-Burch. Causal reasoning of entities and events in procedural texts. In Andreas Vlachos and Isabelle Augenstein, editors,Findings of the Association for Computational Linguistics: EACL 2023,...

    2023

  20. [28]

    Temporal reasoning on implicit events from distant supervision

    Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. Temporal reasoning on implicit events from distant supervision. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Ch...

    2021

  21. [30]

    USA Hospitals

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2017

  22. [2021]

    Accessed: 2025-03-20

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.