{"id":"17dff51f-7276-4e6a-bed5-188f09e216f6","arxiv_id":"2412.17701","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Microtheories, distilled sets of model-generated sentences, improve entailment grounding and QA accuracy when added to a general corpus.","lead":"The paper shows that a language model's topic-level knowledge can be distilled into a small set of sentences, called a microtheory, which can be added to Wikipedia to help answer science and medical exam questions. The added sentences improve how often answers can be grounded in verifiable text and raise question-answering accuracy by up to 8 points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grounding and QA gains are measured with the same TreeWise engine used to select microtheory facts; an independent-engine evaluation is needed before accepting the +8% claims.","rationale":"The reader's weakest assumption identifies the entailment engine as the load-bearing risk, and my analysis agrees: the same engine is used for selection and evaluation, so within-engine gains could reflect overfitting to TreeWise's proof-search preferences. The held-out test split, the random baseline, and the full-pool F comparison reduce but do not eliminate this concern, because every outcome metric is still produced by TreeWise. A cross-engine evaluation is the natural check. I also note the QA claim is further strained by the fact that the largest accuracy gain is from the full fact pool, not from the distilled microtheory; the reader's rationale already flags this. These are addressable empirical concerns rather than demonstrations of failure, so the existing CONDITIONAL verdict is appropriate and no verdict change is needed.","tokens_in":28926,"tokens_out":5370,"duration_ms":56755,"concrete_test":"Re-run the Figure 6 and Figure 7 ARC conditions with an independent entailment engine (e.g., Entailer or NLProofS) used only at evaluation time, keeping the same n-MtPC, n-MtQC, and n-Mtrandom theories and the same Wikipedia index. Compare (a) full grounding rates and (b) QA accuracy deltas over corpus-only. If the optimized microtheories no longer outperform the random baseline, or if the +8% grounding delta collapses, the reported gains are TreeWise-specific rather than general properties of the distilled theories.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims in §5.1–5.2 are evaluated with TreeWise, the same entailment engine used during construction: Eqs. (2)–(4) select n-Mt facts by maximizing TreeWise's proof usage or coverage on training hypotheses, and Figures 6–7 then measure TreeWise's ability to ground and answer test hypotheses from those facts. Because TreeWise includes a fine-tuned step verifier and SBERT-based retrieval, the selection is implicitly optimizing for that engine's retrieval and proof-search biases, such as preferred phrasings and acceptable inference patterns. The +8% grounding gain over the corpus is therefore not yet evidence that microtheories are verifiable by textual entailment in general; it may partly reflect that the facts were chosen because TreeWise finds them easy to use. The full-pool F result partially mitigates this, since F was not selected by TreeWise, but the distilled-Mt comparisons and the QC/PC-versus-random advantages are all within-engine. Separately, the QA claim is weakened by the fact that the largest accuracy gain in Figure 7 comes from the undistilled pool F (78%) rather than from the distilled n=1000 microtheories (73%), so the headline '+8% absolute' accuracy improvement is not actually a distilled-microtheory result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for distilling an LM's topical knowledge into a concise set of natural-language statements called a microtheory. Starting from a topic's training questions, the method prompts an LLM (GPT-4) to generate supporting facts, filters these into a generic fact pool, reduces redundancy via SBERT and entailment-based condensation, and then selects a budgeted microtheory using one of three optimization objectives: most-used facts, maximum question coverage, or maximum partial coverage (Eqs. 2–4). The microtheory is then added to a general corpus (Wikipedia or Wikipedia+textbooks), and an entailment engine (TreeWise) is used to ground answers and perform multiple-choice QA on held-out ARC and MedQA questions. The paper reports that adding the full fact pool improves grounding of correct answers by up to 8 percentage points over the corpus alone and improves QA accuracy by up to 8 points absolute on ARC, that distilled microtheories are more topically relevant than random subsets according to both human experts (MedQA) and an LLM-as-judge, and that a p-relevance curve can predict how many training questions are needed for a microtheory to be relevant to a given fraction of test questions. The paper includes detailed appendices with prompts, LP formulations, and additional Mixtral-based replication results.","tokens_in":29170,"tokens_out":8448,"duration_ms":78668,"significance":"The paper opens a useful new direction: materializing an LLM's topical 'theory' as an inspectable, concise set of statements and evaluating whether that theory supports downstream reasoning. The method is clearly specified, with reproducible details (LP formulations, prompts, thresholds, and an appendix replication with a different generator model). A notable strength is the human-expert evaluation in the medical domain, which provides independent evidence that the distilled microtheory is more topically concentrated than random subsets of the full pool. If the central results hold, the approach could be a practical tool for auditing and augmenting corpora with model-derived knowledge. However, the load-bearing evaluations are weakened by three issues: (i) the same entailment engine (TreeWise) is used both for selecting the microtheory and for measuring grounding/QA, creating an internal-consistency loop; (ii) the largest headline gains come from the full fact pool rather than from the distilled microtheories; and (iii) no significance testing or error bars are reported, while the test sets are small (249 ARC questions).","major_comments":[{"comment":"The fact-selection objectives in §3.2.3 are defined via LEAVES(ENGINE(h_j, C; Q_j)), i.e., TreeWise proof trees on training hypotheses, and the evaluation in Figures 6–7 uses the same TreeWise engine on test hypotheses. This is not a per-question fit to test labels, because selection uses only training questions, but it is an internal-consistency loop: the selected facts are optimized for TreeWise's proof-search preferences, and the test metric then measures TreeWise's ability to use those facts. The +8% grounding gain and the QA gains may therefore partly reflect the engine's own biases rather than the topical criticality of the facts. The full-pool F comparison is less affected by this selection loop, but the abstract's '+8%' claim is precisely the F-vs-corpus comparison, not a distilled-microtheory result. To support the central claim, the authors should evaluate test grounding and QA with an independent entailment engine or verifier (e.g., Entailer or a different NLI-based step checker) and report whether the gains transfer; alternatively, they should show that the selected microtheories also improve grounding under an independent engine.","section":"§3.2.3, Eqs. (2)–(4) and §5.1–§5.2, Figures 6–7"},{"comment":"The largest reported gains come from the full fact pool F, not from the distilled microtheories. The abstract's 'fully grounding up to +8% more answers' corresponds to the ARC Corpus (84%) vs F (92%) comparison, and the '+8% absolute' QA gain corresponds to Corpus (69.2%) vs F (78%), while the n=1000 Mts improve QA by only 4 points (73%) and their grounding rate is described only as 'similar' to WorldTree. Since the paper's contribution is distillation, the headline claims should be restated for the distilled n-Mt results, the exact grounding rates for the n=1000 bars should be reported, and the paper should discuss whether the 4-point gain is meaningful relative to the large reduction in fact count.","section":"§5.1 and §5.2, Figures 6 and 7"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in the evaluation. The ARC test set has 249 questions across 9 topics, so a 4-point QA difference is roughly 10 questions, and the claimed 'no noticeable effect' of the optimization method is a null result that needs statistical support. Please report per-topic breakdowns or bootstrap/topic-stratified confidence intervals for the Corpus, F, and n-Mt conditions, and run a paired significance test for the headline comparisons (e.g., Corpus vs F and Corpus vs 1000-Mt).","section":"§5.1–§5.2, Figures 6 and 7"},{"comment":"The p-relevance extrapolation fits a four-parameter modified Hill equation (Vmax, K, n, s) and reports R^2=1.00. With four free parameters and a small number of training-size points, R^2=1.00 is not evidence of a good fit. The paper should state how many data points were used, whether the curve was fit independently per dataset, the fitted parameter values, and confidence intervals on the extrapolated numbers (940 for ARC, 8953 for MedQA). Without this, the practical recommendation that practitioners need 'about 900' vs 'about 9000' questions is not reliable.","section":"§6 and Figure 10"}],"minor_comments":[{"comment":"The thresholds t=0.9 and u=0.3 are fixed without sensitivity analysis; please add a sentence or appendix showing that the main results are robust to reasonable changes in these values.","section":"§3.2.2"},{"comment":"The inline numbers '84 92' and '80 80' above the bars are unexplained; clarify which conditions they refer to and ensure the striped/solid legend matches the caption.","section":"Figure 6"},{"comment":"The statement that n=1000 Mts result in 'similar rates of overall grounding' to WorldTree is vague; report the actual percentages for the relevant bars.","section":"§5.1"},{"comment":"The word 'microtheories' is misspelled as 'microtheries' in the introduction and in the conclusion's final sentence; please correct.","section":"§1 and §7"},{"comment":"Restricting the relevance assessment to the top 270 facts retrieved by the entailment engine, which uses an SBERT encoder fine-tuned on science QA, may bias the ARC relevance numbers; state whether results are sensitive to this cap.","section":"§5.3.2, footnote 10"},{"comment":"Clarify whether the 'Min #Fs to Cover' LP was run on the condensed pool C or the raw pool F, and how ties were resolved.","section":"Figure 3"},{"comment":"The Mixtral replication shows substantively lower proof coverage and grounding than GPT-4; the main text should reference this result and discuss its implications for the claim that the method is not closed-model dependent.","section":"Appendix G"},{"comment":"The related-work paragraph on library learning is too brief; say explicitly how the approach differs from library learning (generalization over data vs articulating generalizations the model already has).","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written paper from a strong group, and the method is likely to be of interest to the CL community. The main risks are that the headline numbers are drawn from the full pool rather than the distilled microtheory, and that the evaluation is circular with respect to TreeWise. Both are fixable: an independent-engine evaluation and more careful statistical reporting would substantially strengthen the paper. I recommend major revision rather than rejection. I would also ask the authors to provide the raw data and fitted parameters for the p-relevance curves, since the reported R^2=1.00 with a four-parameter model is a likely red flag for many reviewers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the microtheory idea is genuinely new and worth engaging with, but the headline +8% numbers are not as clean as the abstract suggests. The largest gains come from dumping the entire fact pool into the corpus, not from the distilled microtheory, and the entailment engine that selects facts is also the one that measures grounding. That said, this is a solid paper with a real contribution, and it deserves referee time.\n\nWhat's new: prior work gives per-question explanations or knowledge graphs; this paper tries to extract a reusable topic-level theory from the LM, condense it, and evaluate whether it helps ground and answer held-out questions. The combination of fact generation, soft-dedup, entailment condensation, and coverage-optimized selection (usage, QC, PC) is not something I've seen in the cited literature. The p-relevance metric is a useful addition. The human evaluation in MedQA (two senior med students) is a nice orthogonal check, and the Mixtral replication in the appendix shows the method isn't locked to GPT-4.\n\nWhere it's soft: (1) The evaluation loop. TreeWise is used both to decide which facts are kept (via proof usage/coverage on training questions) and to decide whether test answers are 'grounded.' So the +8% grounding gain partly reflects that the facts were chosen because TreeWise finds them easy to use. The full-pool F result mitigates this somewhat, since F isn't selected by TreeWise, but the distilled-Mt comparisons and the QC/PC-versus-random advantages are all within the same engine. An independent entailment engine or human verification would make the claim much stronger. (2) The abstract says 'up to +8% absolute' for QA, but that's the full fact pool F (78%), not the distilled n=1000 microtheory (73%). The distilled theories give +4%. That's still a positive result, but the framing overstates it. (3) No error bars or significance tests on the main numbers; with ~250 test questions, a few questions can move the needle. (4) The R^2=1.00 for the Hill equation in Section 6 is a red flag. They extrapolate to 8953 training questions for MedQA; that's a lot of weight on a curve fit with likely overfitting.\n\nThe paper is honest about the limitations of entailment ('lacks a fully formal definition') and the method is clearly specified. The qualitative examples show the microtheories contain sensible, core facts. This is not a case of a load-bearing flaw; it's a case of claims slightly ahead of the evidence.\n\nBottom line: This paper is for anyone working on explainable QA, model auditing, or knowledge distillation. It deserves a serious referee. I'd want the authors to address the circularity with an independent check, report variance, and separate the distilled-microtheory results from the full-pool results in the abstract.\n\nRecommendation: send it to review, with a request for revision.","headline":"Microtheories are a real, useful idea, but the headline gains come from the full fact pool and the evaluation shares an entailment engine with the selection, so the +8% claims need a grain of salt.","tokens_in":29763,"tokens_out":4489,"would_cite":true,"duration_ms":37144,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Materializing a model's topical knowledge as a small 'microtheory' of reusable sentences improves verifiable grounding by up to 8% and grounded QA accuracy by up to 8 points absolute.","keywords":["microtheory","textual entailment","question answering","knowledge distillation","grounding","language models","p-relevance","explainability"],"falsifier":"Run the full pipeline on the same ARC and MedQA splits with a different entailment engine, or with human verification of every entailment tree; if the +8% grounding gain and +8% accuracy gain disappear when the proof checker changes, the reported improvement is an artifact of the engine's own search rather than a property of microtheories.","tokens_in":28706,"feed_emoji":"🧠","tokens_out":7075,"duration_ms":58102,"temperature":0.7,"pith_summary":"The paper proposes that a language model's latent, topic-level understanding can be materialized as a microtheory: a small, inspectable set of natural-language sentences that jointly entail answers to many questions on a topic. It argues that such microtheories, extracted from the model itself and then distilled for conciseness, add topical knowledge missing from a general corpus, letting an entailment engine ground more answers (up to +8%) and answer more accurately (up to +8% absolute). The paper also claims the distilled facts are more topically critical than the raw fact pool, and introduces p-relevance to predict how much training data is needed to make a microtheory useful. If right, users get a verifiable, inspectable window into what a model knows about a topic, not just a per-question explanation.","feed_headline":"Distilled 'microtheories' ground up to 8% more model answers","feed_subtitle":"A model's own concise topical knowledge helps prove more answers and raises grounded accuracy by up to 8 points.","key_machinery":"The load-bearing object is the microtheory, a list of natural-language sentences intended to capture reusable principles (for example, 'force equals mass times acceleration') rather than question-specific details. Construction runs through four stages: per-question fact generation with a chain-of-thought-style prompt; a genericness filter; soft deduplication via SBERT cosine similarity and entailment condensation via a cross-encoder; and budgeted selection using one of three objectives, namely usage count, maximum question coverage through a linear program, or maximum partial coverage through an integer linear program. The entailment engine TreeWise supplies the definition of grounded: a hypothesis is grounded when the engine finds an entailment tree whose leaves lie in the corpus plus the microtheory. The paper's p-relevance metric fits learning curves to predict how many training questions are needed to reach a given probability of containing a fact relevant to a random test question.","core_discovery":"The central discovery is that a set of model-generated sentences selected for reuse across questions, rather than the full pool of extracted facts, can act as a concentrated theory of a topic. Starting from questions that define a topic, the method prompts the model for facts supporting correct answers, discards context-specific statements, removes paraphrases and entailed redundancies, and then chooses a budgeted set that maximizes proof coverage of training hypotheses. When these microtheories are added to Wikipedia (or Wikipedia plus textbooks), TreeWise grounds fully up to 8% more test answers than the corpus alone, and grounded QA accuracy rises by up to 8 points; the 1000-fact theories match the benefit of the hand-built WorldTree corpus at under 10% of its size. A human expert evaluation in medicine found distilled facts score significantly higher in topical relevance than random facts from the pool, and the p-relevance analysis shows how training-set size predicts topic coverage.","pith_inferences":["If the entailment engine's 'grounded' verdicts are accepted, the method effectively re-uses the model's own knowledge to verify itself; a fair test would check grounding against human judges or an independent engine, since the same step verifier selects and evaluates the facts.","The large gap between ARC and MedQA suggests microtheories will pay off most in domains with a compact shared curriculum and least in open-ended factoid settings where each question is idiosyncratic; the paper gestures at this implication but does not develop it.","p-relevance could serve as a pre-construction data-budgeting tool: estimate whether a candidate domain's questions share enough reusable knowledge to make distillation worthwhile before spending on LLM calls for fact extraction.","The appendix's Mixtral replication indicates the pipeline transfers to open models, though with lower proof coverage and relevance, so the practical ceiling of the method may depend on the generating model's knowledge quality."],"forward_implications":["Adding microtheories to a general corpus can raise the fraction of fully groundable answers by up to 8 points, with most of the proof leaves coming from the microtheory in the science domain.","Grounded QA accuracy on ARC improves by up to 8 points absolute; 1000-fact microtheories add about 4 points, on par with the hand-built WorldTree corpus.","Distilled microtheories concentrate topically critical facts: human experts rated optimized facts 0.6 to 1.2 grades above random pool facts in the medical domain, a significant difference.","p-relevance curves can predict data needs, with roughly 940 training questions required for 90% relevance on ARC versus about 8,953 on MedQA, giving practitioners a budget estimate before building a microtheory."],"supporting_citations":[{"why":"Supplies the notion of logical microtheories that the paper adapts to natural-language sentence sets.","marker":"(Blair et al., 1992)"},{"why":"Defines textual entailment, the inference mechanism used to decide whether an answer is grounded.","marker":"(Dagan et al., 2005)"},{"why":"EntailmentBank provides the few-shot exemplars and proof-tree structure used to prompt fact extraction.","marker":"(Dalvi et al., 2021)"},{"why":"TreeWise is the entailment engine that searches for and verifies the proof trees for grounding and QA.","marker":"(Weir et al., 2024)"},{"why":"ARC supplies the grade-school science questions that define the primary evaluation domain.","marker":"(Clark et al., 2018)"},{"why":"MedQA supplies the medical exam questions and associated textbooks for the second domain.","marker":"(Jin et al., 2021)"},{"why":"WorldTree is the hand-built science knowledge corpus used as a comparison baseline for microtheory quality.","marker":"(Xie et al., 2020)"},{"why":"SBERT embeddings are used for soft deduplication of facts and for retrieval candidate selection in the entailment engine.","marker":"(Reimers & Gurevych, 2019)"},{"why":"Provides the rubric used to measure whether a microtheory fact is relevant to answering a given question.","marker":"(Jansen et al., 2021)"},{"why":"Chain-of-thought prompting style is used as the basis of the fact-extraction prompt.","marker":"(Wei et al., 2022)"}],"fun_headline_variants":["Distilled microtheories lift grounded QA accuracy by 8%","Model's own concise theory grounds 8% more answers","Distilled facts add missing topical knowledge for QA","Microtheories: concise model knowledge 8% more grounding","Budgeted model facts boost grounded accuracy up to 8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the TreeWise entailment engine, with a fine-tuned ChatGPT step verifier, gives a trustworthy definition of 'grounded': the same engine selects microtheory facts, judges whether answers are grounded, and scores QA accuracy, so a systematic bias in its proof search would inflate all three measures.","fun_headline_variants_meta":{"raw":{"variants":["Distilled microtheories lift grounded QA accuracy by 8%","Model's own concise theory grounds 8% more answers","Distilled facts add missing topical knowledge for QA","Microtheories: concise model knowledge 8% more grounding","Budgeted model facts boost grounded accuracy up to 8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2730,"prompt_tokens":1097,"completion_tokens":1633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":1550}},"tokens_in":713,"tokens_out":1633,"duration_ms":12097,"temperature":1.0,"reasoning_tokens":1550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:15:33.367224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on the same ARC and MedQA splits with a different entailment engine, or with human verification of every entailment tree; if the +8% grounding gain and +8% accuracy gain disappear when the proof checker changes, the reported improvement is an artifact of the engine's own search rather than a property of microtheories.","supporting_citations":[{"cited_title":"When we retrieve facts from both the Microtheory and Wikipedia (or Wikipedia+Textbooks), we append the sets in that order","cited_arxiv_id":null,"evidence_quote":"SBERT embeddings are used for soft deduplication of facts and for retrieval candidate selection in the entailment engine."}],"review_version":1}