{"id":"2980bdca-46f0-41ae-98bd-2dd2ba55080f","arxiv_id":"2501.17840","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Continued pre-training with LoRA on simplified triple-only documents improves LLM performance on the paper's own insight-extraction benchmarks, but those benchmarks are built from the same documents used for training.","lead":"This paper tests whether continued pre-training with LoRA helps language models learn deeper insights from medical and financial documents. It reports that simplifying documents down to extracted fact triples, rather than full text, produces the largest gains on its own benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark reuses the training triples as test queries, so the near-perfect declarative scores in Table 1 measure memorization of seen triples, not insight learning.","rationale":"The reader's weakest assumption identifies exactly the load-bearing problem: the evaluation is built from the same documents used for training, and the simplified training data is literally the test triples. My independent reading of Section 2.2 and Table 1 confirms that the declarative and statistical benchmarks have direct train/test overlap, making the 'near-perfect' triple-training results an expected consequence of memorization rather than evidence of insight learning. The paper is transparent about its pipeline and the experimental details are reproducible, but transparency does not remove the validity threat. A document-level held-out split is the natural and decisive check; without it, the central claim that document simplification 'significantly enhances insight-learning capabilities' is unsupported. I therefore agree with the REJECT verdict and see no reason to adjust it, though the rejection is based on a fixable evaluation-design issue rather than on any internal inconsistency in the method itself.","tokens_in":9686,"tokens_out":3111,"duration_ms":34099,"concrete_test":"Create a document-level held-out evaluation: split each corpus 80/20 before any triple extraction, extract triples only from the held-out 20% of documents to build the declarative and statistical test sets, and continual pre-train with LoRA on triples from the remaining 80%. Re-run the triple-only condition (Table 1) with LLaMA-3.1 8B and compare declarative Exact Match and statistical Recall@10 against the current numbers. If performance on unseen-document triples collapses toward the vanilla baseline, the reported near-perfect scores are memorization artifacts; if it remains high on triples from documents never seen during training, the insight-generalization claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that simplifying documents to triples significantly enhances insight learning—rests on benchmark scores whose test items are constructed from the exact triples used in training. In Section 2.2, the authors extract triples from the Hallmarks of Cancer and Buster documents, use those triples to build the declarative, statistical, and probabilistic evaluation sets, and then define 'simplified documents' as documents reduced to those same triples rendered as sentences. Section 4 then trains on this simplified/triple-only format and reports near-perfect declarative Exact Match in Table 1. Because a declarative query is a subject-relation pair whose gold object is the object from a training triple, the model can achieve high Exact Match simply by next-token prediction on strings it has already seen. The same holds for statistical insights, where the gold set of all objects for a subject-relation pair is exactly the set of objects appearing in training triples. Thus the reported gains are consistent with memorization of training content, not with the 'insight learning' or 'beyond surface-level' capability claimed in the abstract and conclusion. The paper does not report any held-out split, withheld relations, or unseen-document evaluation that would distinguish generalization from recall. The probabilistic results are less directly contaminated because they are based on entity co-occurrence statistics, but those statistics are also computed from the same training documents. This train/eval overlap is the most load-bearing weakness: if it is not addressed, the headline empirical result does not support the stated conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether continual pre-training with LoRA can help LLMs internalize three types of domain-specific insights—declarative, statistical, and probabilistic—from two domain corpora (Hallmarks of Cancer for medicine, Buster for finance). The authors extract <subject, relation, object> triples with GPT-4o mini, manually filter and normalize them, and construct evaluation sets for each insight type from those triples. They then continually pre-train LLaMA-3.2 1B, LLaMA-3.2 3B, and LLaMA-3.1 8B with LoRA on (a) original documents, (b) simplified documents containing only the extracted triples rendered as sentences, and (c) individual triples. The reported results show marginal gains from original-document training and large gains, especially for declarative insights, when training on simplified or triple-only data. The central claim is that modifying documents to retain only essential information significantly enhances insight learning.","tokens_in":9886,"tokens_out":3155,"duration_ms":34141,"significance":"If the central claim were valid, the paper would make a practically useful point: the format of continual pre-training data strongly influences how effectively LoRA-adapted LLMs acquire relational knowledge. The authors have invested substantial effort in benchmark construction, test multiple model sizes, and release their dataset and code. However, the evaluation design is internally closed: test queries are generated from the same triples that appear in the training data for the simplified and triple-only conditions. The near-perfect declarative scores in Table 1 are therefore consistent with memorization rather than 'insight learning' or generalization beyond surface-level knowledge. The paper does not provide any held-out evaluation on unseen documents, unseen relations, or withheld triples, nor does it report variance or significance across runs. I agree with the stress-test concern: the reported improvements do not, as they stand, support the abstract's claim that simplifying documents enhances insight-learning capabilities. The paper's contribution is better framed as a study of memorization capacity under different input formats unless a proper generalization test is added.","major_comments":[{"comment":"The evaluation is circular for the central claim. In Section 2.2, the authors extract triples from the Hallmarks of Cancer and Buster documents, then build the declarative and statistical evaluation sets from those same triples. The 'simplified documents' condition is defined as retaining only those extracted triples, and the triple-only condition trains on the individual triples directly. For declarative insights, a test query is a subject-relation pair whose gold object is the object from a training triple; for statistical insights, the gold set of all objects for a subject-relation pair is exactly the set of objects seen in the training triples. Under these conditions, high Exact Match and Recall@10 can be achieved by next-token prediction on memorized strings, so Table 1's near-perfect declarative scores do not demonstrate insight learning. A held-out evaluation on triples from documents not used in training, or at least on withheld subject-relation pairs, is needed to separate memorization from generalization.","section":"Section 2.2 and Section 4, Table 1"},{"comment":"The statistical-insight results are also contaminated by the same train/test overlap. The gold answer for a statistical query is the full set of objects for a subject-relation pair, and the model is trained on exactly those triples in the simplified and triple-only conditions. Improvements in Recall@10 therefore reflect the model's ability to reproduce a seen list of objects, not the ability to aggregate across documents or recognize a distribution. The paper's claim that LLMs are 'limited in effectively aggregating information' (Section 4) is not testable with this protocol. An evaluation on unseen relations, unseen entities, or held-out document subsets is required.","section":"Section 4, statistical insights"},{"comment":"The probabilistic evaluation is less directly contaminated but still not independent. The conditional probabilities p(entity2 | entity1) are computed from co-occurrence within the very documents used for training, and in the simplified/triple conditions the model is trained on triples that expose those entity co-occurrences. The reported MAE values in Table 1 and Figure 2 therefore measure how well the model can reproduce corpus statistics it has been trained on, not whether it has learned a generalizable notion of uncertainty. An evaluation on entity pairs or documents held out from training would be necessary to support the paper's claims about probabilistic insight learning.","section":"Section 2.2 and Section 4, probabilistic insights"},{"comment":"No variance, confidence intervals, or significance tests are reported. All figures and tables appear to come from single runs, and the Appendix states only that hyperparameters were tuned on training loss via grid search. Given that several headline differences are small (e.g., Buster declarative scores of 97.2 vs. 97.4 vs. 97.8 in Table 1), the absence of repeated runs or statistical testing makes it impossible to assess whether the marginal improvements attributed to continual pre-training on original documents are meaningful. This is not merely a presentation issue; it weakens the empirical support for the paper's comparative claims.","section":"Section 3 and Figures 2 and 4"}],"minor_comments":[{"comment":"The description of the benchmark sampling says queries are sampled 'as evenly as possible,' but no random seed or exact sampling procedure is given, so the benchmark construction is not fully reproducible from the text alone.","section":"Section 2.2"},{"comment":"The paper does not provide the exact prompts or decoding settings used for declarative and statistical insight extraction (top-k, temperature, number of generated tokens), which are needed to reproduce the results.","section":"Section 3"},{"comment":"The term 'continual pre-training' is used for training on individual triples rendered as sentences; since this is highly supervised and task-specific, the authors should clarify how this differs from standard fine-tuning and whether the distinction affects the interpretation of the results.","section":"Section 4"},{"comment":"The per-relation breakdowns in Tables 4-7 are useful, but they would be easier to interpret if accompanied by the number of queries per relation and some measure of uncertainty.","section":"Appendix B"},{"comment":"The conclusion restates the central claim without acknowledging the train/test overlap in the benchmark; a limitations paragraph should be added if the paper is revised.","section":"Conclusion"}],"recommendation":"reject","confidential_remarks":"The circular evaluation is the core problem: the manuscript's main result—that simplifying documents to triples enhances insight learning—is not supported by the reported experiments because the test queries are constructed from the same triples used for training in the simplified and triple-only conditions. This is not a matter of a missing baseline or a minor robustness check; it is a load-bearing flaw in the evaluation protocol. The paper could potentially be made publishable by redoing the evaluation with held-out documents, held-out relations, or withheld triples, and by reporting variance and significance. Given the current state, however, the central claim is untested. I would be willing to review a revised version that addresses this issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. The paper is clearly written, transparent about its pipeline, and the three-way insight taxonomy plus the released benchmark are genuinely reusable assets. But the central empirical claim—that simplifying documents to extracted triples enhances LLMs' insight learning—is not supported by the experiments as designed, because the evaluation set is built from the very triples used in training.\n\nWhat is actually new: the authors define declarative, statistical, and probabilistic insights, then construct benchmarks from GPT-4o-mini-extracted triples that they filter and normalize. The benchmark construction is described in enough detail to reproduce. The per-relation breakdown is a nice touch, and the observation that full-document LoRA continual pre-training yields only marginal gains is consistent with earlier findings on LoRA's limited learning capacity.\n\nWhere it falls down: the test queries come from the same documents (and the same triples) used for training. For declarative insights, each query is a subject-relation pair whose gold object is an object from a training triple; the simplified training condition presents those exact triples as sentences. Near-perfect exact match on those items is the signature of memorized next-token prediction, not of insight generalization. Statistical insights are similarly contaminated: the gold set of objects is exactly the set seen during training. Probabilistic insights are less directly contaminated, but the entity co-occurrence statistics are computed from the same training documents. No held-out split, no withheld relations, no variance across runs, and no significance tests are reported. This is a load-bearing flaw, not a minor caveat.\n\nThe paper is still worth engaging with, because the flaw is fixable and the authors are transparent about their method. A revision that evaluates on held-out documents or withheld relations, and reports seed-level variance, could turn this into a useful result. I would not cite the current results as evidence for the headline claim.\n\nBottom line: send it out for review, but the referee should make the contamination issue the centerpiece of the revision. For a reading group, it's a good cautionary example of evaluation-loop design.","headline":"Useful benchmark and taxonomy, but the headline result measures memorization of training triples, not insight learning.","tokens_in":10457,"tokens_out":2551,"would_cite":false,"duration_ms":24698,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continual pre-training with LoRA gains little from full documents but much from documents reduced to extracted fact triples, yielding near-perfect declarative insight accuracy.","keywords":["insight learning","continual pre-training","LoRA","declarative insights","statistical insights","probabilistic insights","triple extraction","domain adaptation"],"falsifier":"Hold out a random subset of documents, extract triples only from those held-out documents, train on simplified triples from the remaining documents, and evaluate on the held-out triple queries; if the declarative exact-match gains collapse toward the vanilla baseline, the claim that simplification enhances insight learning is refuted in favor of a memorization explanation.","tokens_in":9446,"feed_emoji":"🧠","tokens_out":5459,"duration_ms":45125,"temperature":0.7,"pith_summary":"This paper asks whether continual pre-training with low-rank adapters (LoRA) can teach large language models to internalize 'insights' from domain documents, and whether the format of the training text matters. It claims that training on original documents produces only marginal gains in extracting declarative, statistical, and probabilistic insights, but that rewriting documents so they contain only the essential information — extracted fact triples rendered as sentences — sharply improves declarative and statistical insight extraction. The strongest version of the claim, reported for training on individual triples, is that models reach near-perfect accuracy of roughly 97–99% exact match on declarative insight queries in both medicine and finance domains. The paper matters because it isolates input format as a major lever in domain adaptation, separate from model scale and adapter capacity.","feed_headline":"Stripping documents to triples boosts LLM insight gains","feed_subtitle":"Training on triple-form sentences, not full documents, lifts declarative and statistical insight scores sharply.","key_machinery":"The central machinery is the three-way insight taxonomy plus the triple-based document simplification. Declarative insights are single-object facts, statistical insights are multi-object aggregations, and probabilistic insights are conditional entity probabilities. The pipeline uses a strong LLM to extract <subject, relation, object> triples from documents, manually filters and normalizes the relations, then constructs three training formats: original documents, simplified documents containing only the triples rendered as sentences, and individual triples as separate inputs. LoRA adapters on the LLaMA-3.2 1B, LLaMA-3.2 3B, and LLaMA-3.1 8B models carry the continual pre-training, and the comparison across formats is what reveals that removing non-essential document content lets the adapter store relational knowledge, whereas full documents dilute it.","core_discovery":"The central discovery is that the format of continual pre-training data, not the amount of text, governs whether low-rank adapters internalize relational knowledge. On original documents, continued pre-training with LoRA over 30 epochs improves declarative and statistical insight scores by only a few points and leaves probabilistic insight estimates essentially unchanged. When the same documents are reduced to extracted triples rendered as sentences, all evaluated models improve substantially on declarative and statistical queries, and when the model is trained on individual triples as separate inputs, declarative exact match reaches 97–99% across both domains. The authors interpret this as evidence that LoRA's capacity is sufficient to store structured facts, that input format is the limiting factor, and that statistical aggregation and probabilistic reasoning remain harder even under the best tested format.","pith_inferences":["The near-perfect declarative scores likely measure recall of the training triples themselves; a held-out split using triples from unseen documents would test whether the model can answer new subject-relation queries it never saw during training.","The large per-relation variation, such as the 'acquired' relation jumping from near zero to roughly 78% while 'decreased' stays flat, suggests that the base model's prior relational knowledge gates what LoRA can absorb; knowledge editing could target relations where the base model is weak.","The same simplification idea could be tested on multi-hop or numerical reasoning by extracting n-ary facts or aggregate statistics rather than binary triples, and on other adapter methods to see whether the format effect is specific to LoRA.","The pipeline depends on a strong teacher model to extract clean triples, so the findings reflect a distillation setup; a natural extension would test cheaper or weaker extractors and measure how quickly the insight gains degrade."],"forward_implications":["Domains with fact-dense documents can be pre-processed into triple-form text to make LoRA continual pre-training far more effective for factual question answering.","Statistical insight extraction, predicting all objects for a subject-relation pair, improves with simplification but still falls well short of declarative performance, indicating that aggregation remains hard for low-rank adapters.","Probabilistic insights, estimating the conditional probability of one entity given another, barely improve under any training format, so LoRA continual pre-training is not sufficient for learning uncertainty estimates from text.","Larger LLaMA models benefit more from simplified training data, suggesting that model capacity bounds insight learning under LoRA.","The input format, not just the amount of training, determines whether knowledge is internalized during continual pre-training."],"supporting_citations":[{"why":"Supplies the LoRA adaptation method used for all continual pre-training runs.","marker":"Hu et al., 2021"},{"why":"Motivates continual pre-training as the domain-adaptation paradigm the paper follows.","marker":"Gururangan et al., 2020"},{"why":"Provides the Hallmarks of Cancer corpus used as the medicine-domain dataset.","marker":"Baker et al., 2016"},{"why":"Provides the Buster corpus used as the finance-domain dataset.","marker":"Zugarini et al., 2024"},{"why":"Provides GPT-4o mini, the model used to extract the triples that ground both the benchmarks and the simplified training data.","marker":"Hurst et al., 2024"},{"why":"Documents LoRA's limited knowledge acquisition, cited to explain the marginal gains observed on original documents.","marker":"Biderman et al., 2024"},{"why":"Supplies self-consistency sampling, the method used to estimate probabilistic insight probabilities from generated outputs.","marker":"Wang et al., 2022"}],"fun_headline_variants":["Triple-form text beats full docs for LLM insight gains","LoRA insight learning: format trumps volume in pre-training","Stripped triples boost declarative and statistical insight","For LLM insight, triple sentences outdo full documents","Input format gates LoRA insight: triples win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation queries are built from triples extracted from the same documents used for training, so the measured gains may reflect memorization of training triples rather than generalization to insights from unseen documents.","fun_headline_variants_meta":{"raw":{"variants":["Triple-form text beats full docs for LLM insight gains","LoRA insight learning: format trumps volume in pre-training","Stripped triples boost declarative and statistical insight","For LLM insight, triple sentences outdo full documents","Input format gates LoRA insight: triples win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1219,"prompt_tokens":842,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":458,"tokens_out":377,"duration_ms":4557,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:31:36.018810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a random subset of documents, extract triples only from those held-out documents, train on simplified triples from the remaining documents, and evaluate on the held-out triple queries; if the declarative exact-match gains collapse toward the vanilla baseline, the claim that simplification enhances insight learning is refuted in favor of a memorization explanation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Hallmarks of Cancer corpus used as the medicine-domain dataset."},{"cited_title":"BUSTER: a \"BUSiness Transaction Entity Recognition\" dataset","cited_arxiv_id":"2402.09916","evidence_quote":"Provides the Buster corpus used as the finance-domain dataset."}],"review_version":1}