Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering

T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Materializing a model's topical knowledge as a small 'microtheory' of reusable sentences improves verifiable grounding by up to 8% and grounded QA accuracy by up to 8 points absolute.

desk verdict Microtheories are a real, useful idea, but the headline gains come from the full fact pool and the evaluation shares an entailment engine with the selection, so the +8% claims need a grain of salt. read the letter →

arxiv 2412.17701 v2 pith:DDCQSIAI submitted 2024-12-23 cs.CL

Nathaniel Weir , Bhavana Dalvi Mishra , Orion Weller , Oyvind Tafjord , Sam Hornstein , Alexander Sabol , Peter Jansen , Benjamin Van Durme
show 1 more author
Peter Clark
This is my paper · ORCID
classification cs.CL
keywords microtheorytextualentailmentquestionansweringknowledgedistillationgroundinglanguagemodelsp-relevanceexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a language model's latent, topic-level understanding can be materialized as a microtheory: a small, inspectable set of natural-language sentences that jointly entail answers to many questions on a topic. It argues that such microtheories, extracted from the model itself and then distilled for conciseness, add topical knowledge missing from a general corpus, letting an entailment engine ground more answers (up to +8%) and answer more accurately (up to +8% absolute). The paper also claims the distilled facts are more topically critical than the raw fact pool, and introduces p-relevance to predict how much training data is needed to make a microtheory useful. If right, users get a verifiable, inspectable window into what a model knows about a topic, not just a per-question explanation.

What carries the argument

The load-bearing object is the microtheory, a list of natural-language sentences intended to capture reusable principles (for example, 'force equals mass times acceleration') rather than question-specific details. Construction runs through four stages: per-question fact generation with a chain-of-thought-style prompt; a genericness filter; soft deduplication via SBERT cosine similarity and entailment condensation via a cross-encoder; and budgeted selection using one of three objectives, namely usage count, maximum question coverage through a linear program, or maximum partial coverage through an integer linear program. The entailment engine TreeWise supplies the definition of grounded: a hypothesis is grounded when the engine finds an entailment tree whose leaves lie in the corpus plus the microtheory. The paper's p-relevance metric fits learning curves to predict how many training questions are needed to reach a given probability of containing a fact relevant to a random test question.

What would settle it

Run the full pipeline on the same ARC and MedQA splits with a different entailment engine, or with human verification of every entailment tree; if the +8% grounding gain and +8% accuracy gain disappear when the proof checker changes, the reported improvement is an artifact of the engine's own search rather than a property of microtheories.

Watch

Extended reading notes

Core claim

The central discovery is that a set of model-generated sentences selected for reuse across questions, rather than the full pool of extracted facts, can act as a concentrated theory of a topic. Starting from questions that define a topic, the method prompts the model for facts supporting correct answers, discards context-specific statements, removes paraphrases and entailed redundancies, and then chooses a budgeted set that maximizes proof coverage of training hypotheses. When these microtheories are added to Wikipedia (or Wikipedia plus textbooks), TreeWise grounds fully up to 8% more test answers than the corpus alone, and grounded QA accuracy rises by up to 8 points; the 1000-fact theories match the benefit of the hand-built WorldTree corpus at under 10% of its size. A human expert evaluation in medicine found distilled facts score significantly higher in topical relevance than random facts from the pool, and the p-relevance analysis shows how training-set size predicts topic coverage.

Load-bearing premise

The load-bearing premise is that the TreeWise entailment engine, with a fine-tuned ChatGPT step verifier, gives a trustworthy definition of 'grounded': the same engine selects microtheory facts, judges whether answers are grounded, and scores QA accuracy, so a systematic bias in its proof search would inflate all three measures.

Editorial extensions

If this is right

  • Adding microtheories to a general corpus can raise the fraction of fully groundable answers by up to 8 points, with most of the proof leaves coming from the microtheory in the science domain.
  • Grounded QA accuracy on ARC improves by up to 8 points absolute; 1000-fact microtheories add about 4 points, on par with the hand-built WorldTree corpus.
  • Distilled microtheories concentrate topically critical facts: human experts rated optimized facts 0.6 to 1.2 grades above random pool facts in the medical domain, a significant difference.
  • p-relevance curves can predict data needs, with roughly 940 training questions required for 90% relevance on ARC versus about 8,953 on MedQA, giving practitioners a budget estimate before building a microtheory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the entailment engine's 'grounded' verdicts are accepted, the method effectively re-uses the model's own knowledge to verify itself; a fair test would check grounding against human judges or an independent engine, since the same step verifier selects and evaluates the facts.
  • The large gap between ARC and MedQA suggests microtheories will pay off most in domains with a compact shared curriculum and least in open-ended factoid settings where each question is idiosyncratic; the paper gestures at this implication but does not develop it.
  • p-relevance could serve as a pre-construction data-budgeting tool: estimate whether a candidate domain's questions share enough reusable knowledge to make distillation worthwhile before spending on LLM calls for fact extraction.
  • The appendix's Mixtral replication indicates the pipeline transfers to open models, though with lower proof coverage and relevance, so the practical ceiling of the method may depend on the generating model's knowledge quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes a method for distilling an LM's topical knowledge into a concise set of natural-language statements called a microtheory. Starting from a topic's training questions, the method prompts an LLM (GPT-4) to generate supporting facts, filters these into a generic fact pool, reduces redundancy via SBERT and entailment-based condensation, and then selects a budgeted microtheory using one of three optimization objectives: most-used facts, maximum question coverage, or maximum partial coverage (Eqs. 2–4). The microtheory is then added to a general corpus (Wikipedia or Wikipedia+textbooks), and an entailment engine (TreeWise) is used to ground answers and perform multiple-choice QA on held-out ARC and MedQA questions. The paper reports that adding the full fact pool improves grounding of correct answers by up to 8 percentage points over the corpus alone and improves QA accuracy by up to 8 points absolute on ARC, that distilled microtheories are more topically relevant than random subsets according to both human experts (MedQA) and an LLM-as-judge, and that a p-relevance curve can predict how many training questions are needed for a microtheory to be relevant to a given fraction of test questions. The paper includes detailed appendices with prompts, LP formulations, and additional Mixtral-based replication results.

Significance. The paper opens a useful new direction: materializing an LLM's topical 'theory' as an inspectable, concise set of statements and evaluating whether that theory supports downstream reasoning. The method is clearly specified, with reproducible details (LP formulations, prompts, thresholds, and an appendix replication with a different generator model). A notable strength is the human-expert evaluation in the medical domain, which provides independent evidence that the distilled microtheory is more topically concentrated than random subsets of the full pool. If the central results hold, the approach could be a practical tool for auditing and augmenting corpora with model-derived knowledge. However, the load-bearing evaluations are weakened by three issues: (i) the same entailment engine (TreeWise) is used both for selecting the microtheory and for measuring grounding/QA, creating an internal-consistency loop; (ii) the largest headline gains come from the full fact pool rather than from the distilled microtheories; and (iii) no significance testing or error bars are reported, while the test sets are small (249 ARC questions).

major comments (4)
  1. [§3.2.3, Eqs. (2)–(4) and §5.1–§5.2, Figures 6–7] The fact-selection objectives in §3.2.3 are defined via LEAVES(ENGINE(h_j, C; Q_j)), i.e., TreeWise proof trees on training hypotheses, and the evaluation in Figures 6–7 uses the same TreeWise engine on test hypotheses. This is not a per-question fit to test labels, because selection uses only training questions, but it is an internal-consistency loop: the selected facts are optimized for TreeWise's proof-search preferences, and the test metric then measures TreeWise's ability to use those facts. The +8% grounding gain and the QA gains may therefore partly reflect the engine's own biases rather than the topical criticality of the facts. The full-pool F comparison is less affected by this selection loop, but the abstract's '+8%' claim is precisely the F-vs-corpus comparison, not a distilled-microtheory result. To support the central claim, the authors should evaluate test grounding and QA with an independent entailment engine or verifier (e.g., Entailer or a different NLI-based step checker) and report whether the gains transfer; alternatively, they should show that the selected microtheories also improve grounding under an independent engine.
  2. [§5.1 and §5.2, Figures 6 and 7] The largest reported gains come from the full fact pool F, not from the distilled microtheories. The abstract's 'fully grounding up to +8% more answers' corresponds to the ARC Corpus (84%) vs F (92%) comparison, and the '+8% absolute' QA gain corresponds to Corpus (69.2%) vs F (78%), while the n=1000 Mts improve QA by only 4 points (73%) and their grounding rate is described only as 'similar' to WorldTree. Since the paper's contribution is distillation, the headline claims should be restated for the distilled n-Mt results, the exact grounding rates for the n=1000 bars should be reported, and the paper should discuss whether the 4-point gain is meaningful relative to the large reduction in fact count.
  3. [§5.1–§5.2, Figures 6 and 7] No error bars, confidence intervals, or significance tests are reported anywhere in the evaluation. The ARC test set has 249 questions across 9 topics, so a 4-point QA difference is roughly 10 questions, and the claimed 'no noticeable effect' of the optimization method is a null result that needs statistical support. Please report per-topic breakdowns or bootstrap/topic-stratified confidence intervals for the Corpus, F, and n-Mt conditions, and run a paired significance test for the headline comparisons (e.g., Corpus vs F and Corpus vs 1000-Mt).
  4. [§6 and Figure 10] The p-relevance extrapolation fits a four-parameter modified Hill equation (Vmax, K, n, s) and reports R^2=1.00. With four free parameters and a small number of training-size points, R^2=1.00 is not evidence of a good fit. The paper should state how many data points were used, whether the curve was fit independently per dataset, the fitted parameter values, and confidence intervals on the extrapolated numbers (940 for ARC, 8953 for MedQA). Without this, the practical recommendation that practitioners need 'about 900' vs 'about 9000' questions is not reliable.
minor comments (8)
  1. [§3.2.2] The thresholds t=0.9 and u=0.3 are fixed without sensitivity analysis; please add a sentence or appendix showing that the main results are robust to reasonable changes in these values.
  2. [Figure 6] The inline numbers '84 92' and '80 80' above the bars are unexplained; clarify which conditions they refer to and ensure the striped/solid legend matches the caption.
  3. [§5.1] The statement that n=1000 Mts result in 'similar rates of overall grounding' to WorldTree is vague; report the actual percentages for the relevant bars.
  4. [§1 and §7] The word 'microtheories' is misspelled as 'microtheries' in the introduction and in the conclusion's final sentence; please correct.
  5. [§5.3.2, footnote 10] Restricting the relevance assessment to the top 270 facts retrieved by the entailment engine, which uses an SBERT encoder fine-tuned on science QA, may bias the ARC relevance numbers; state whether results are sensitive to this cap.
  6. [Figure 3] Clarify whether the 'Min #Fs to Cover' LP was run on the condensed pool C or the raw pool F, and how ties were resolved.
  7. [Appendix G] The Mixtral replication shows substantively lower proof coverage and grounding than GPT-4; the main text should reference this result and discuss its implications for the claim that the method is not closed-model dependent.
  8. [§2] The related-work paragraph on library learning is too brief; say explicitly how the approach differs from library learning (generalization over data vs articulating generalizations the model already has).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's distillation and evaluation chain is a held-out same-metric pipeline with independent human and gold-label checks, not a by-construction equivalence.

full rationale

The paper's central derivation is not circular. Microtheory facts are generated from an LM, condensed with SBERT and a cross-encoder, and then selected by optimizing TreeWise proof usage or coverage on training hypotheses (Eqs. 2-4). The subsequent grounding evaluation (Fig. 6) and QA evaluation (Fig. 7) are run on held-out test hypotheses, so the improvements are not forced by construction: the engine's proof preferences on training questions must generalize to new questions for the measured gains to appear. The use of the same entailment engine for fact selection and for evaluation is a methodological limitation, but it is a standard train/test evaluation with the same metric, not a reduction of the output to the input. Independent support exists outside the TreeWise loop: QA accuracy is scored against gold ARC answer labels, and the expert relevance study in Section 5.3.1 provides an external human judgment that distilled facts are more topically relevant than randomly sampled pool facts. The p-relevance analysis in Section 6 fits a curve to measured relevance rates and extrapolates to new training-set sizes; this is curve fitting and extrapolation, not a self-referential prediction. The paper explicitly acknowledges that entailment engines are trained on examples rather than being formal systems, which tempers the strength of the grounding metric but does not make the derivation circular. No load-bearing argument reduces to a self-citation: TreeWise is cited as an off-the-shelf system with its own public implementation and published evaluation, and the paper does not invoke any uniqueness theorem or ansatz from the authors' prior work to force its conclusions.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims depend on several unverified assumptions: the LLM produces true and generic facts; the entailment engine's judgments are correct; the topic clustering creates valid splits; and the Hill-equation fit is used to extrapolate training set sizes. The method introduces no new physical entities, but it does introduce a new knowledge representation (microtheory) and a completeness metric (p-relevance).

free parameters (5)
  • SBERT soft-dedup threshold t = 0.9
    Set in section 3.2.2 to remove paraphrased facts during condensation; hand-chosen, affects the size and content of the condensed pool C.
  • SBERT entailment-candidate threshold u = 0.3
    Set in section 3.2.2 to select candidate fact pairs for entailment condensation; hand-chosen.
  • Topic clustering cosine threshold = 0.6
    Used in appendix B to cluster core-fact embeddings into topics; hand-chosen and affects the mini-split construction and thus all downstream evaluations.
  • Microtheory size budget n = swept 25, 50, 100, 200, 500, 1000
    The size of the distilled microtheory is selected by the user; the paper reports results across n and uses n=1000 for headline comparisons, so the choice of n is a free design parameter.
  • Hill equation parameters (Vmax, K, n, s) = not reported individually; R-squared = 1.00
    Fitted in section 6 to p-relevance learning curves, then used to predict required training set sizes (940 for ARC, 8953 for MedQA). This is a fit used as a prediction.
assumptions (4)
  • domain assumption The entailment engine's judgments of textual entailment are correct, or at least reliable, definitions of what it means for an answer to be grounded.
    Invoked throughout sections 3.1, 4.1, and 5.1; the paper itself notes entailment lacks a fully formal definition.
  • domain assumption GPT-4 (and Mixtral in the appendix) generates factually accurate, generic facts when prompted.
    Section 3.2.1 constructs the entire fact pool from model generations; no automatic or human verification of fact correctness is applied before use.
  • domain assumption The LLM plus SBERT clustering yields topically coherent mini-splits that represent the dataset topics.
    Section 4 and appendix B use these clusters to define train and test splits; if clusters are incoherent, the measured gains do not match the intended topical setup.
  • standard math Integer linear programming solvers correctly solve the coverage objectives in appendices A.1 and A.2.
    The LP and ILP formulations are assumed solvable to optimality; standard solvers are implied but not named.
invented entities (1)
  • Microtheory (linguistic construct) independent evidence
    purpose: A set of natural-language sentences intended to encapsulate an LM's core, reusable knowledge about a topic, used to augment corpora for entailment-based QA.
    The paper provides falsifiable operationalizations: grounding rates, QA accuracy, expert relevance ratings, and p-relevance, so the construct is testable even though it is not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering." pith.science (2026). https://pith.science/paper/DDCQSIAI

@misc{pith2026241217701,
  author       = {Pith},
  title        = {Pith review of: From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDCQSIAI}},
  note         = {Machine review of arXiv:2412.17701}
}
abstract

Recent reasoning methods (e.g., chain-of-thought, entailment reasoning) help users understand how language models (LMs) answer a single question, but they do little to reveal the LM's overall understanding, or "theory," about the question's topic, making it still hard to trust the model. Our goal is to materialize such theories - here called microtheories (a linguistic analog of logical microtheories) - as a set of sentences encapsulating an LM's core knowledge about a topic. These statements systematically work together to entail answers to a set of questions to both engender trust and improve performance. Our approach is to first populate a knowledge store with (model-generated) sentences that entail answers to training questions and then distill those down to a core microtheory that is concise, general, and non-redundant. We show that, when added to a general corpus (e.g., Wikipedia), microtheories can supply critical, topical information not necessarily present in the corpus, improving both a model's ability to ground its answers to verifiable knowledge (i.e., show how answers are systematically entailed by documents in the corpus, fully grounding up to +8% more answers), and the accuracy of those grounded answers (up to +8% absolute). We also show that, in a human evaluation in the medical domain, our distilled microtheories contain a significantly higher concentration of topically critical facts than the non-distilled knowledge store. Finally, we show we can quantify the coverage of a microtheory for a topic (characterized by a dataset) using a notion of $p$-relevance. Together, these suggest that microtheories are an efficient distillation of an LM's topic-relevant knowledge, that they can usefully augment existing corpora, and can provide both performance gains and an interpretable, verifiable window into the model's knowledge of a topic.

Figures

Figures reproduced from arXiv: 2412.17701 by the authors.

Figure 1
Figure 1. Given a set of topical training questions, we construct a microtheory, a set of statements articulating [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of distillation techniques for extracting [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Histogram of training question per fact in C and 650-850 training questions. ARC facts are used more frequently than MedQA. Mass, Acceleration and Gravity”, and “Plant Growth and Reproduction.” For MedQA, we make a split with 4 clusters including “The Effects of Smoking on Health” and “Kidney Function and Disorders.” (See [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (17 more)
Figure 5
Figure 5. Figure 5: Rate at which different microtheory ap￾proaches entail training hypotheses. Lighter bars are the sum of fractional coverages for questions not fully covered by the Mt. Total coverage for ARC questions is generally higher than MedQA. The 1000-MtP C s effectively fully c…
Figure 7
Figure 7. Figure 7: Question Answering performance on ARC topical [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Average expert-annotated task relevance score for microtheory facts. PC facts are more relevant than random and usage until the 1000-fact budget. We worked with the annotators to develop a 5- point rubric for scoring how essential a fact is to the core content that app…
Figure 10
Figure 10. Figure 10: Number of training questions (x-axis) needed to create an F that is relevant to test questions p% of the time (y-axis). While datasets like ARC and Social IQa (Sap et al., 2019) need 900-1600 questions for 90-relevance, but MedQA needs far more (8900). gests that some…
Figure 11
Figure 11. Figure 11: TheoryCoT in-context learning prompt for extracting WorldTree-like facts from an LLM for a given [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Example TheoryCoT output for a MedQA question. The generated proof steps are thrown out and [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Top-10 most used and random 10 least used (at least once) facts in the ARC microtheory. The [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Top-10 most used and random 10 least used (at least once) facts in the MedQA microtheory. The [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Subgraph showing some of the statements from the 200-Mt [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Subgraph showing some of the statements from the 100-Mt [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 18
Figure 18. Figure 18: Histogram of training question per fact in C and 650 MedQA training questions us￾ing Mixtral-8x22b as the underlying LLM. While multiple GPT-4 generated facts are used for 25 or more training questions, no Mixtral-generated fact is used more than 22 times. Cp Rd Us QC…
Figure 19
Figure 19. Figure 19: % of test hypotheses fully groundable after [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 21
Figure 21. Figure 21: Fraction of a microtheory used in an entail [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 23
Figure 23. Figure 23: Rate (%) of microtheories containing at least one fact with 5/5 and 4/5 relevance as determined by [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: Rubric used to collect expert relevance ratings for medical microtheory facts (1 of 2) [PITH_FULL_IMAGE:figures/full_fig_p024_24.png]
Figure 25
Figure 25. Figure 25: Rubric used to collect expert relevance ratings for medical microtheory facts (2 of 2) [PITH_FULL_IMAGE:figures/full_fig_p025_25.png]
Figure 26
Figure 26. Figure 26: Prompt used to score the question relevance of retrieved microtheory statements. Instructions are [PITH_FULL_IMAGE:figures/full_fig_p026_26.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligning Generalisation Between Humans and Machines

    cs.AI 2024-11 unverdicted novelty 4.0 of 10

    A perspective paper maps how humans and machines generalize differently and argues that aligning these generalization behaviors is essential for human-AI teaming.

Reference graph

Works this paper leans on

31 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Instructions are based on the rubric from Jansen et al

    Figure 26: Prompt used to score the question relevance of retrieved microtheory statements. Instructions are based on the rubric from Jansen et al. (2021). 26

    work page 2021

  2. [3]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457,

    arXiv

  3. [9]

    Smith, Dan Moreno, and Huitzilin Ortiz

    11 Peter Jansen, Kelly J. Smith, Dan Moreno, and Huitzilin Ortiz. On the challenges of evaluat- ing compositional explanations in multi-hop inference: Relevance, completeness, and expert ratings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language ...

    work page 2021

  4. [10]

    URL https://doi.org/10.1007/978-3-540-24750-0_

    work page doi:10.1007/978-3-540-24750-0_

  5. [11]

    doi: 10.18653/v1/2021.emnlp-main.596

    Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.596. URL https: //aclanthology.org/2021.emnlp-main.596. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint a...

    arXiv 2021

  6. [12]

    Maieutic prompting: Logically consistent reasoning with recursive explanations

    Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1266–1279, Abu Dhabi...

    work page 2022

  7. [13]

    doi: 10.18653/v1/2022

    Association for Computational Linguistics. doi: 10.18653/v1/2022. emnlp-main.82. URL https://aclanthology.org/2022.emnlp-main.82. Nora Kassner, Oyvind Tafjord, Hinrich Sch ¨utze, and Peter Clark. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen...

    doi:10.18653/v1/2022 2022

  8. [14]

    doi: 10.18653/v1/2021.emnlp-main.697

    Association for Computational Linguis- tics. doi: 10.18653/v1/2021.emnlp-main.697. URL https://aclanthology.org/2021. emnlp-main.697. Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, and Pe- ter Clark. Language models with rationality. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference...

    doi:10.18653/v1/2021.emnlp-main.697 2021

Show all 31 references
  1. [15]

    URL https://aclanthology.org/Q19-1026

    doi: 10.1162/tacl a 00276. URL https://aclanthology.org/Q19-1026. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. Natural language inference from multiple premises. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth Interna- tional Joint Conference on Natural ...

    doi

  2. [19]

    doi: 10.18653/v1/D19-1250

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1250. URL https://aclanthology. org/D19-1250. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proc...

    doi

  3. [20]

    doi: 10.18653/v1/D19-1410

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410. Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henry Zhu, Xinchi Chen, Zhiheng Huang, Peng Xu, Andrew O. Arnold, and Dan Roth. Entailment tree expl...

    doi

  4. [22]

    doi: 10.18653/v1/D19-1454

    Association for Com- putational Linguistics. doi: 10.18653/v1/D19-1454. URL https://aclanthology.org/ D19-1454. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering questions with faithful and truthful chains of reasoning. In Yoav Goldberg, Zornitsa Kozar...

    doi

  5. [23]

    doi: 10.18653/v1/2022.emnlp-main.134

    Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.134. URL https://aclanthology.org/ 2022.emnlp-main.134. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. V oyager: An open-ended embo...

    2022 doi

  6. [24]

    Enhancing systematic de- compositional natural language inference using informal logic

    Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Zhang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, et al. Enhancing systematic de- compositional natural language inference using informal logic. arXiv preprint arXiv:2402.14798,

    arXiv

  7. [25]

    ISBN 979-10-95546-34-4

    European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.671. 13 Kaiyu Yang, Jia Deng, and Danqi Chen. Generating natural language proofs with verifier-guided search. ArXiv, abs/2205.12443,

    2020 arXiv

  8. [26]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.),Proceed...

    2018

  9. [28]

    unfolding

    int2 & Pancreatic calcifications can be a sign of chronic pancreatitis -> hypothesis: Further evaluation is most likely to show pancreatic calcifications. Figure 12: Example TheoryCoT output for a MedQA question. The generated proof steps are thrown out and the generic facts a...

    2021

  10. [29]

    When we retrieve facts from both the Microtheory and Wikipedia (or Wikipedia+Textbooks), we append the sets in that order

    for first stage retrieval and rerank using Sentence- Transformer’s ms-marco-MiniLM-L-12-v2 (Reimers & Gurevych, 2019). When we retrieve facts from both the Microtheory and Wikipedia (or Wikipedia+Textbooks), we append the sets in that order. When we retrieve from Wikipedia+Tex...

    2019

  11. [30]

    (2021) contains 4 ordinal scores; we added a 5th to differentiate between facts relevant to ruling out incorrect options (4) vs supporting correct options (5)

    The original rubric by Jansen et al. (2021) contains 4 ordinal scores; we added a 5th to differentiate between facts relevant to ruling out incorrect options (4) vs supporting correct options (5). 23 # Indicators Exemplars 5 ● 1 Statement/fact that includes relevant associatio...

    2021

  12. [46]

    Xing, and Zhiting Hu

    Shibo Hao, Bowen Tan, Kaiwen Tang, Hengzhe Zhang, Eric P. Xing, and Zhiting Hu. Bertnet: Harvesting knowledge graphs from pretrained language models. ArXiv, abs/2206.14268,

    arXiv

  13. [1990]

    doi: 10.1145/79173.79176

    ISSN 0001-0782. doi: 10.1145/79173.79176. URL https://doi.org/10.1145/79173.79176. Bernardo Magnini. Book reviews: Recognizing textual entailment: Models and applications by Ido Dagan, Dan Roth, mark sammons and fabio massimo zanzotto.Computational Linguistics, 41(1): 157–159, March

  14. [1991]

    12 Fabio Petroni, Tim Rockt¨aschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in...

    2019

  15. [1995]

    Social IQa: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- gua...

    2019

  16. [2005]

    semanticscholar.org/CorpusID:8587959

    URL https://api. semanticscholar.org/CorpusID:8587959. Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Sco...

    2021

  17. [2015]

    URL https://aclanthology.org/ J15-1008

    doi: 10.1162/COLI a 00213. URL https://aclanthology.org/ J15-1008. Stephen Muggleton. Inductive logic programming. New generation computing,

    doi

  18. [2018]

    covered” iff all its facts are kept. ∀Lkj , |Lkj |X i=1 xi ≥ yj · |Lkj | (7) • Each question is “covered

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259. 14 A U SAGE OPTIMIZATION DETAILS A.1 n-MTQC LINEAR PROGRAM We set the objective function to achieve the following goals, ordered by precedence: • Maximize the coverage ...

    doi

  19. [2019]

    doi: 10.18653/v1/P19-1470

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1470. URL https://aclanthology.org/P19-1470. Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduction through search over statement compositions. ArXiv, abs/2201.06028,

    arXiv

  20. [2020]

    URL https://aclanthology.org/2020.aacl-main

    Association for Computational Linguistics. URL https://aclanthology.org/2020.aacl-main

    2020

  21. [2021]

    doi: 10.18653/v1/2021.emnlp-main.585

    Association for Computational Linguis- tics. doi: 10.18653/v1/2021.emnlp-main.585. URL https://aclanthology.org/2021. emnlp-main.585. Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable AI for natural ...

    2021 doi

  22. [2022]

    Metgen: A module-based entailment tree generation framework for answer explanation

    Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. Metgen: A module-based entailment tree generation framework for answer explanation. ArXiv, abs/2205.02593,

    arXiv

  23. [2023]

    doi: 10.18653/v1/2023.emnlp-main.877

    Association for Computational Linguis- tics. doi: 10.18653/v1/2023.emnlp-main.877. URL https://aclanthology.org/2023. emnlp-main.877. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob ...

    2023 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.