{"id":"ae2f9722-943b-4e94-aff1-38b2cc8d9098","arxiv_id":"2505.24302","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"ScienceMeter evaluates language model knowledge updates across three axes, preservation of old scientific claims, acquisition of new claims, and projection to future findings, and finds all current methods fall short.","lead":"This paper introduces ScienceMeter, a benchmark for testing whether language models can update their scientific knowledge without forgetting old knowledge. It measures three abilities: remembering old science, absorbing new papers, and predicting future findings, and finds that current methods are far from adequate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Projection metric labels claims by a single unseen paper's stance while prompting 'Is the claim correct?', so the 37.7% figure conflates truth with paper stance.","rationale":"The reader's weakest assumption identifies exactly this construct-validity flaw, and the full text confirms it. The projection task is the most novel axis of ScienceMeter, and the headline conclusion—that no knowledge update method can simultaneously preserve, acquire, and project—depends on that axis being meaningful. The prompt 'Is the claim correct?' is paired with labels derived from whether one specific future paper supports or refutes the claim, and REFUTE claims are explicitly defined as 'relevant but non-supporting' rather than false. This is not a subtle philosophical disagreement: it is an internal mismatch between the operationalization and the construct. The paper does have real strengths: a large multi-domain dataset, expert validation of synthetic claims, a consistency check with author-annotated claims, and public code/data. The preservation and acquisition axes are more defensible, and those parts of the benchmark may be reusable. But the central empirical claim bundles all three axes, and the projection number that anchors the paper's main finding is not measuring 'correctness' of future scientific knowledge. Recomputing with truth-based labels or with a prompt aligned to the label would be required to know whether the three-way impossibility result survives. The reader's REJECT verdict is therefore appropriate, though the manuscript could become a valuable benchmark if the projection task is redefined or repositioned.","tokens_in":19053,"tokens_out":5922,"duration_ms":79776,"concrete_test":"Take 200 future-knowledge items (balanced SUPPORT/REFUTE) from the Appendix D setup. Have two domain experts, blinded to the future paper's stance, independently label each claim's real-world truth (true/false/unknown) using the claim text and independent evidence. Compute projection accuracy two ways: (1) model prediction vs future-paper stance as in the paper; (2) model prediction vs expert truth, with a prediction counted correct only when the model's yes/no answer matches the claim's actual truth value. If the two accuracies differ materially (e.g., by more than 10 percentage points), or if a nontrivial fraction of REFUTE claims are judged true, the 37.7% projection figure is an artifact of stance-matching rather than knowledge projection.","verdict_should_be":"REJECT","load_bearing_attack":"The central three-axis claim rests on the projection metric, which is not measuring what it is named. In Section 2.2, the future-knowledge judgment task gives the model a claim c associated with a future paper and asks it to predict y(c) ∈ {SUPPORT, REFUTE}, where y(c) indicates whether that specific future paper supports or refutes c. But the prompt in Appendix D (Table 8) is 'Is the claim correct?' with no reference to the future paper. The labels are not truth labels: REFUTE claims are generated as 'relevant but non-supporting' (Section 2.1), so a claim can be true and still be labeled REFUTE if that particular future paper does not support it; conversely a claim can be false but supported by the future paper and labeled SUPPORT. Therefore the projection score (37.7%) measures how often the model guesses which of two paper-specific claims a future paper will make, not how well the model anticipates correct scientific knowledge. Since the headline 'no method achieves all three objectives' requires the projection axis to be valid, this is a load-bearing construct-validity problem. The paper's own note that 'the true magnitude of Knowledge Projection is likely higher' (Section 2.3) acknowledges label unreliability but does not fix the mismatch between prompt, label, and construct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ScienceMeter proposes an evaluation framework for scientific knowledge updates in LLMs, defining three metrics: knowledge preservation (retention of prior claims), knowledge acquisition (uptake of new claims), and knowledge projection (anticipation of future claims). The authors construct a dataset of 15,444 papers and 30,888 synthetic claims across 10 domains, arrange papers into prior/new/future trajectories by citation and publication dates, and evaluate five update methods on LLaMA-3.1-8B and OLMo2-32B using claim judgment and claim generation tasks. The main empirical claim is that no knowledge update method achieves all three objectives, with the best observed values being 85.9% preservation, 71.7% acquisition, and 37.7% projection. The paper also reports cross-domain correlations with citation counts and pretraining occurrence, and evaluates a domain-adapted scientific LLM in Materials Science.","tokens_in":19289,"tokens_out":6034,"duration_ms":79577,"significance":"If the framework is valid, this would be a useful and much-needed benchmark for evaluating scientific knowledge updates in LLMs, an area where most existing benchmarks focus on factoid or Wikipedia-style knowledge. The paper has clear strengths: code and data are linked; the dataset is large and multi-domain; synthetic claims receive expert validation; a comparison with author-annotated claims is reported; metric formulas are explicit; and a specialized scientific LLM is included in the evaluation. These are genuine contributions. The main obstacle is construct validity of the projection metric, which directly underpins the 37.7% headline number; until that is repaired or relabeled, the central claim is not supported as stated.","major_comments":[{"comment":"The projection metric does not measure what it is named, and this issue is load-bearing for the headline 37.7% figure. In Section 2.2, the future-knowledge task defines the ground-truth label y(c) as whether the associated future paper supports or refutes the claim, with no title or paper provided to the model. Appendix D, Table 8, however, prompts the model with 'Is the claim correct?' rather than 'Would the future paper support this claim?' Meanwhile, Section 2.1 defines REFUTE claims as 'relevant but non-supporting,' not as false. A claim can therefore be true and still labeled REFUTE if one particular future paper does not support it, and a claim can be false yet labeled SUPPORT if that paper endorses it. The resulting score measures how often the model predicts a single paper's stance, not how well it anticipates correct scientific knowledge. The paper's own caveat in Section 2.3 that 'the true magnitude of Knowledge Projection is likely higher' acknowledges label unreliability but does not fix the mismatch between the prompt, the label, and the construct. A concrete fix is to (i) revise the prompt and label so that the task is explicitly to predict whether the named future paper would support the claim, and rename the metric accordingly, or (ii) build future labels from genuinely confirmed or refuted claims using later consensus or human verification.","section":"2.2, Appendix D, Table 8"},{"comment":"The claim-generation results in Table 2 rely on GPT-4o as both claim generator and factual-accuracy judge, but the paper does not report human validation of this judge. The synthetic-claim generation is validated by experts (Table 6) and the linguistic-confidence judge is validated against human raters (75.9%), but the statement in Section 3.2 that 'we assess the factual accuracy of the generated claim by inviting GPT-4o to determine whether y(ĉ,t)=SUPPORT' has no such check. Because the reference claims were also produced by GPT-4o, a systematic bias in the judge would directly affect the generation-task columns of Table 2, which are averaged into the headline preservation, acquisition, and projection numbers. Please add a human sample evaluation of the GPT-4o judge on generated claims, or report agreement statistics and failure cases.","section":"3.2"},{"comment":"Knowledge Acquisition and Knowledge Projection are conditional on the claim being unknown to the model before the update. The formulas in Table 10 are explicit about this, but the abstract and Section 3.4 state that the best methods 'acquire 71.7% of new knowledge' and 'project 37.7% of future knowledge' without the conditional qualifier. Because the denominator excludes claims the model already answered correctly, a method that already knows many of the new or future claims can appear to have low acquisition or projection even if it updates correctly. The percentages are therefore not rates over all new or future claims, and the headline statement is potentially misleading. Please state the conditional interpretation prominently in the abstract and Section 3.4, or additionally report unconditional rates.","section":"2.3, Table 10"}],"minor_comments":[{"comment":"The prompt templates contain grammatical errors, for example 'a atomic verifiable statements' in Tables 5, 8, and 9; these should be corrected before publication.","section":"Appendix C.2 and D"},{"comment":"The column headers 'PresDist LossAcquDist LossProjLoss' are compressed and hard to parse; the color-coding described in the text is not visible in a monochrome printout. Consider using separate labeled subcolumns or a clearer grouping.","section":"Table 2"},{"comment":"The cross-domain correlation analysis is based on only 10 domains, but the paper reports Pearson r values without p-values or confidence intervals. Adding these, or a nonparametric alternative, would make the 'strong relationship' claim more credible.","section":"4.1, Figure 4"},{"comment":"The notation in the INFER method description, 'g(LMf(Ptestnew), p)=g(LM|pnew,p)', is confusing and should be rewritten with a cleaner definition of how the new paper is inserted into the prompt.","section":"3.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical infrastructure is solid and the benchmark has clear value for the NLP community, but the projection metric is the deciding issue. If the authors re-scope the metric as a future-paper-stance prediction task or construct genuinely refuted future claims, I would be supportive after major revision. I would also ask the authors to include human validation of the GPT-4o judge used in the generation task. The current framing overstates what the data measure, especially in the abstract and Section 3.4."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ScienceMeter is a serious attempt to benchmark scientific knowledge updates, and the core idea – tracking claims across prior, new, and future papers and measuring preservation, acquisition, and projection – is genuinely new. The dataset is substantial (5,148 paper triplets, 30,888 claims across ten domains), and the authors do real validation work on their synthetic claims: expert evaluation, and an author-annotated comparison showing no significant difference. The preservation and acquisition metrics, based on claim verification against a paper title, are defensible and useful.\n\nThe problem is the projection axis. As the stress-test note says, the prompt for future-knowledge judgment is 'Is the claim correct?' but the gold label is whether the associated future paper happens to support or refute the claim. Those are not the same. REFUTE claims are generated as 'relevant but non-supporting,' so a true claim can be labeled REFUTE if the future paper doesn't make that claim, and a false claim could be labeled SUPPORT if the paper does. The model is essentially being asked to guess paper content, not to anticipate correct science. The paper even acknowledges this in Section 2.3 ('the true magnitude of Knowledge Projection is likely higher'), but that doesn't fix the mismatch between prompt, label, and construct. This is load-bearing because the headline 'no method achieves all three objectives' rests on the 37.7% projection number.\n\nOther soft spots are less severe. There are no error bars and no no-update baseline, which makes the comparative claims about methods weaker than they look. The reliance on GPT-4o for claim generation, generation judging, and confidence assessment is a mild circularity, though the human validation on claims and confidence helps.\n\nIf the authors reframe projection as 'predicting future-paper stance' or, better, relabel the task and adjust the prompt to match the construct, the framework becomes much stronger. As is, the preservation and acquisition results are probably fine, and the dataset is a real resource. I would send this to peer review with a clear request to fix the projection metric; it deserves referee time, not because it's right in its current form, but because the framework is novel and the flaw is fixable. I'd bring it to a reading group precisely to discuss that flaw.\n\nRecommendation: engage with it, but don't take the projection number seriously until the task is redefined.","headline":"ScienceMeter has a valuable dataset and a sensible preservation/acquisition framework, but its headline projection metric asks models to judge claim truth while scoring them on paper stance, so the 37.7% projection figure should not be taken at face value.","tokens_in":19824,"tokens_out":3426,"would_cite":false,"duration_ms":41376,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No knowledge-update method can simultaneously preserve past scientific claims, acquire new ones, and project future ones; the best scores only 85.9%, 71.7%, and 37.7% on the three axes.","keywords":["knowledge update","scientific claims","knowledge preservation","knowledge acquisition","knowledge projection","evaluation framework","domain volatility","large language models"],"falsifier":"Have expert scientists independently judge the truth of a sample of future claims without knowing which paper cites them, recompute the 37.7% projection figure with those expert labels, and see whether it moves materially; if it does, the projection metric tracks citation stance rather than future knowledge.","tokens_in":18794,"feed_emoji":"📊","tokens_out":10113,"duration_ms":102788,"temperature":0.7,"pith_summary":"The paper proposes that a useful scientific knowledge update for an LLM must do three things at once: keep previously learned claims intact, absorb claims from newly added papers, and anticipate claims that will appear only in future papers. To test this, the authors build ScienceMeter, a ten-domain dataset of 15,444 papers and 30,888 atomic, verifiable claims organized into prior, new, and future literature by publication date, and define three metrics—knowledge preservation, acquisition, and projection. Evaluating five update methods on mid-size and frontier models, the paper finds that the best method reaches 85.9% preservation, 71.7% acquisition, and 37.7% projection, and that no method achieves all three objectives. The finding matters because it identifies future-knowledge projection as the unmet bottleneck for language models in scientific work, with consequences for their use in hypothesis generation and staying current.","feed_headline":"Best LLM knowledge update preserves 85.9%, projects 37.7%","feed_subtitle":"Even the best update methods fail to keep old claims, learn new ones, and foresee future ones at once.","key_machinery":"ScienceMeter is the load-bearing object: an evaluation framework that treats scientific knowledge as atomic, verifiable claims tied to papers and measures an update along three axes—preservation of prior claims, acquisition of new claims, and projection of future claims. Each claim is scored correct, incorrect, or unknown by combining factual accuracy with an estimated confidence, which yields a two-error decomposition into distortion (confident and wrong) and loss (low confidence). The dataset is built as citation-linked triplets of prior, new, and future papers across ten domains, and the headline numbers are the best scores any tested update method achieves on the three axes averaged over tasks and models.","core_discovery":"The central claim is an empirical ceiling: today's knowledge-update methods cannot satisfy preservation, acquisition, and projection simultaneously. Based on claim judgment and generation over the ScienceMeter corpus, the best-performing method averaged across tasks and models preserves 85.9% of prior correct claims, acquires 71.7% of new claims, and projects 37.7% of future claims, and no method meets all three. The paper also reports that inference-time update methods work for a 32B-parameter model but not for an 8B model, which needs training-based updates; that cross-domain scores correlate with domain volatility rather than pretraining exposure; and that a domain-adapted scientific model still falls short. The authors conclude that projection, not acquisition, is the open challenge for keeping LLMs scientifically current.","pith_inferences":["The projection metric likely understates true projection ability: it scores a claim as 'projected' only if it matches the stance of one specific future paper, even though a true claim could be missing from that paper and a false claim could appear in it; the paper itself notes the true magnitude may be higher.","The temporal-triplet design can serve as a rolling benchmark: as today's 'future' papers become published, the same claims become verifiable, allowing head-to-head comparison of update methods over time without re-annotation.","The distortion-loss decomposition points to a concrete design goal for the next generation of update methods: add an abstention mechanism that refuses to modify the model on low-confidence content, which could raise preservation without hurting acquisition.","The scale-dependent result suggests a practical deployment rule: retrieval-based updating is a premium feature of large models, not a universal replacement for training in smaller deployed systems."],"forward_implications":["No robust scientific knowledge update method exists: any current approach sacrifices at least one of preservation, acquisition, or projection.","Projection is the hardest objective, so today's LLMs are not yet reliable for anticipating future findings, limiting their use in hypothesis generation and research idea formulation.","Model scale determines the right update strategy: inference-time injection works for large models, while small models require training-based updates to absorb new claims.","Domain volatility, not pretraining data availability, drives preservation and projection; fast-moving fields such as materials science and environmental science are where updates fail most.","Even domain-specialized scientific LLMs do not collectively satisfy the three objectives, so the problem is not solved by better pretraining alone."],"supporting_citations":[{"why":"Supplies the atomic scientific claim framework and claim-verification task that ScienceMeter adapts for its knowledge unit.","marker":"[54]"},{"why":"The literature-graph API provides the paper corpus and citation links used to build the prior, new, and future paper triplets.","marker":"[4]"},{"why":"GPT-4o generates synthetic support/refute claims and serves as judge for factual accuracy and linguistic confidence.","marker":"[40]"},{"why":"LLaMA-3.1-8B-Instruct is the mid-size model whose behavior under the five update methods is measured.","marker":"[13]"},{"why":"OLMo2-32B-Instruct is the frontier-size model used to test whether inference-based updates scale with model size.","marker":"[39]"},{"why":"LoRA is the parameter-efficient training adapter used for every training-based update baseline.","marker":"[18]"},{"why":"Continual pre-training is one of the evaluated training-based update methods and a key baseline in the comparison.","marker":"[14]"},{"why":"Instruction-tuning is the other main training-based baseline and the foundation of the combined training-plus-inference method.","marker":"[59]"},{"why":"HoneyBee, a materials-science-adapted model, is used to test whether domain-specialized models escape the three-objective ceiling.","marker":"[49]"}],"fun_headline_variants":["LLM update ceiling: preserve 85.9%, acquire 71.7%, project 37.7%","Projection is the open challenge: only 37.7% future knowledge after updates","Best LLM update keeps old (85.9%), learns new (71.7%), but foresees little (37.7%)","No single update method masters all three knowledge dimensions","For 8B models, training-based updates beat inference-time methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The projection score assumes that a future paper's support or refutation of a claim equals the claim's truth, even though a claim can be true without appearing in that paper or false while being supported by it.","fun_headline_variants_meta":{"raw":{"variants":["LLM update ceiling: preserve 85.9%, acquire 71.7%, project 37.7%","Projection is the open challenge: only 37.7% future knowledge after updates","Best LLM update keeps old (85.9%), learns new (71.7%), but foresees little (37.7%)","No single update method masters all three knowledge dimensions","For 8B models, training-based updates beat inference-time methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3371,"prompt_tokens":884,"completion_tokens":2487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2372}},"tokens_in":500,"tokens_out":2487,"duration_ms":20764,"temperature":1.0,"reasoning_tokens":2372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:26:42.474882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert scientists independently judge the truth of a sample of future claims without knowing which paper cites them, recompute the 37.7% projection figure with those expert labels, and see whether it moves materially; if it does, the projection metric tracks citation stance rather than future knowledge.","supporting_citations":[],"review_version":1}