{"id":"a29f8bb5-2549-4195-a169-aafe8fffa669","arxiv_id":"2505.24615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors propose an LLM-distilled retriever trained on rephrased, partial, and incremental idea variants, and show it improves retrieval and novelty detection on two new closed-domain datasets in marketing and NLP.","lead":"This paper builds two new benchmark datasets for detecting whether a research idea is novel, using LLMs to generate non-novel variants of real paper ideas, and trains a retriever to find conceptually similar ideas for LLM-based novelty checks. A generalist reader might care because automated novelty screening could help researchers and reviewers cope with the flood of new papers, if the approach generalizes beyond its synthetic test setup.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-novel labels are defined by LLM generation, but Prompt 2(c) fuses two anchor ideas into 'incremental' negatives that the paper's own Prompt 3 rubric would rate moderately or highly novel; benchmark validity is unestablished.","rationale":"The reader identified the benchmark's synthetic non-novel labels as the weakest assumption; I agree and sharpen it. The issue is not merely that LLM paraphrases may not represent real non-novel papers: Prompt 2(c) explicitly fuses two anchor ideas into an 'incremental' negative, while Prompt 3 treats combinations and recontextualizations as 0.5-0.7 novelty, so the data-generation process is internally inconsistent with the evaluation rubric. The positive-label assumption (Section 3) is also unverified. The proposed contrastive distillation itself is coherent and the code/data release is a plus, so I would not reject the paper. The verdict remains conditional on expert validation of a label sample and on re-running the headline comparisons on a cleaned subset. This is the same verdict as the reader, hence UNCHANGED; agreement is partial because the internal Prompt 2(c)-Prompt 3 contradiction is a distinct, narrower mechanism than the reader's broader representativeness concern.","tokens_in":16581,"tokens_out":11068,"duration_ms":139975,"concrete_test":"Have three domain experts independently rate a stratified sample (30 per type per domain) of synthesized negatives plus 30 seed positives against the closure corpus using the exact Prompt 3 rubric, taking majority vote. If more than 20% of labeled non-novel incremental/partial ideas receive novelty >= 0.5, or any seed positive is rated <= 0.3, the ground truth contradicts the paper's own definition; then re-run the Table 3 comparisons on the expert-cleaned subset and check whether RAG-KD's F1 advantage over RAG-Vanilla and the best baseline persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sections 5.2-5.3: LLM-KD retriever and RAG-KD consistently outperform baselines) rests on benchmark labels constructed in Section 4.2. Section 3 assumes every synthesized idea in G_S is non-novel merely because it was generated from an anchor in G_N, and that all seed ideas in G_N are novel and mutually distinct. Neither assumption is verified. Prompt 2(c) builds 'incremental ideas' by extracting subsets from two anchor ideas (e.g., idea_A1 + idea_B2 -> new abstract) and deliberately targets low BERT similarity and conceptual fusion. But Prompt 3's own rubric gives 0.5 to a 'different combination of claims' and 0.7 to 'applies known ideas in a new context.' By that rubric, many incremental (and some partial) synthesized ideas are moderately or highly novel, yet every one is labeled non-novel. Likewise, no expert check is reported for the claim that seed papers are novel and pairwise distinct. Since both label classes are defined by an unvalidated GPT-4o-mini generation loop, the F1 improvements in Table 3 (e.g., 0.7344 vs 0.7180 for RAG-Vanilla in Marketing) may measure the ability to reverse-engineer the LLM's paraphrase/recombination distribution, not to detect real scientific non-novelty. The Limitation section concedes LLM-generated ideas and novelty scores are not guaranteed accurate, but the paper does not quantify the effect on the benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two novelty-detection benchmark datasets (Marketing and NLP) constructed by selecting seed papers, crawling their reference closure sets, and compressing each paper into an LLM-generated 'idea' summary. It then proposes an LLM-based knowledge distillation framework that fine-tunes a retriever on contrastive pairs of anchor ideas and LLM-synthesized non-novel variants (rephrased, partial, and incremental), followed by a RAG-based novelty-detection stage that scores retrieved candidates with a five-level LLM rubric and learns a decision tree on the resulting score vectors. Experiments report consistent improvements over baselines for both idea retrieval and novelty detection, with the largest gains on incremental synthetic ideas.","tokens_in":16906,"tokens_out":3237,"duration_ms":40343,"significance":"If the benchmark and the methodology are sound, the paper would provide a useful resource and a practical recipe for bridging textual and conceptual similarity in novelty detection. The strengths are the release of two closed corpora with compact idea summaries, the contrastive distillation idea, the systematic comparison across six retriever backbones, and the group analysis separating rephrased, partial, and incremental ideas. The core reservation is that the benchmark's ground truth is defined entirely by LLM-generated variants of the same seed ideas that the retriever is trained on, so the reported gains may partly measure the retriever's ability to reverse-engineer the generation distribution rather than to identify real-world non-novelty. The paper's own Limitation section concedes that the pseudo-labels are not guaranteed accurate, but does not quantify how this affects the headline results.","major_comments":[{"comment":"The benchmark's non-novel ground truth is asserted, not verified, and appears inconsistent with the paper's own novelty rubric. Section 3 assumes every synthesized idea in G_S is non-novel because it was generated from an anchor in G_N, but Prompt 2(c) intentionally fuses fragments of two anchor ideas and targets low BERT similarity, and Prompt 3 assigns 0.5 to a 'different combination of claims' and 0.7 to 'applies known ideas in a new context.' By that rubric, many incremental ideas (and some partial ideas) would be scored moderately or highly novel, yet every one is labeled non-novel. The paper reports expert voting only for idea-extraction alignment on 50 papers, not for the novelty labels of the synthesized variants. This makes the F1 and retrieval gains in Tables 1 and 3 hard to interpret as measuring scientific non-novelty.","section":"3, 4.2 (Prompt 2(c), Prompt 3)"},{"comment":"The evaluation loop is self-referential: the retriever is contrastively trained on anchor–synthesized pairs produced by Prompt 2, and the same type of LLM-generated variants constitutes the test queries. The consistent improvement of LLM-KD over Vanilla may therefore reflect learning the paraphrase/recombination distribution of GPT-4o-mini rather than a generalizable idea-level similarity. The paper does not test on human-authored non-novel ideas, such as close incremental papers from the reference closure, or provide any evidence that the synthetic variants are representative of real non-novelty. The Limitation section acknowledges noise in pseudo-labels but does not quantify the effect on the benchmark or on the reported improvements.","section":"4.2, 5.2"},{"comment":"The assumption that all seed ideas in G_N are novel and mutually distinct is stated without verification. Because seed papers are drawn from a small set of venues and the corpus is closed by references, the paper does not rule out the possibility that two seed ideas are themselves highly similar or that a seed idea closely matches an older reference paper in the closure set. If such cases exist, the training and evaluation labels in Section 4.2's contrastive objective and in the ND task are inconsistent. An expert audit of seed summaries, or a report of nearest-neighbor idea overlap among seed ideas, is needed to support the benchmark's validity.","section":"3, 4.1"},{"comment":"The ND evaluation uses only 100 training and 100 test samples per domain with a 1:1 novel/non-novel split, and no confidence intervals or multiple runs are reported. Given this sample size, the Marketing margin between RAG-KD and RAG-Vanilla (F1 0.7344 vs 0.7180) is small and may fall within sampling noise. The claim that RAG-KD 'consistently outperforms' baselines should be supported by repeated runs, standard deviations, or a statistical test.","section":"5.3, Table 3"}],"minor_comments":[{"comment":"The heading 'Experiments on idea retrieval tasks (RQ2)' appears to be a typo; RQ2 is the novelty-detection question, and the section describes the ND task.","section":"5.3"},{"comment":"The datasets are described in Section 4.1, not Section 3.1 as stated in the experimental setup.","section":"5.1"},{"comment":"The phrase 'topologicalclosurein topological andcompactnessfor ND' is garbled and should be rewritten.","section":"1"},{"comment":"There is a typo in 'LAGRELY different from these two papers'; it should be 'LARGELY.'","section":"Prompt 2(c)"},{"comment":"DeepSeek-reasoner is used as an LLM backbone but is not cited in the references; please add a proper citation.","section":"5.4"},{"comment":"The group analysis is reported only for the NLP dataset; please clarify why Marketing is omitted or add the corresponding table.","section":"Table 2"},{"comment":"The reported 'average improvements' of 5.40% and 15.19% are not defined in Table 1; state explicitly over which metrics and baselines the averages are computed.","section":"5.2"}],"recommendation":"major_revision","confidential_remarks":"The core methodological risk is that the benchmark and method share a single source of labels: GPT-4o-mini-generated variants of the seed ideas. I would encourage the editor to ask for validation with human-annotated non-novel examples or, at minimum, a quantification of label noise and its effect on the reported gains. The anonymous data/code release is a positive feature and should be preserved during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper with a genuine new idea—distilling three types of LLM-generated idea transformations into a contrastive retriever—and two reusable closed-domain benchmark datasets. I want to like it. The experiments are systematic: six retriever backbones, consistent gains, and a nice breakdown showing the largest improvement on incremental ideas, which are exactly the ones with low textual similarity.\n\nThe soft spot is the label construction. Section 4.2 assumes every synthesized idea in G_S is non-novel because it came from an anchor, and every seed in G_N is novel and pairwise distinct. Neither assumption is checked. More concretely, Prompt 2(c) builds \"incremental\" ideas by fusing fragments of two anchors and deliberately targets low BERT similarity. Under the paper's own Prompt 3 rubric, a \"different combination of claims\" is a 0.5 and \"applies known ideas in a new context\" is a 0.7—so many of the negatives would be scored as moderate or high novelty if judged by the same rubric used for evaluation. That makes the benchmark largely self-referential: the retriever is trained to match LLM-generated recombinations, and then scored on how well it retrieves those same kinds of recombinations. The F1 gap in Table 3 (0.7344 vs 0.7180 in Marketing) could easily be a measure of how well the retriever internalizes GPT-4o-mini's paraphrase and fusion tendencies, not how well it detects real scientific non-novelty.\n\nThe Limitation section does concede that LLM-generated ideas and novelty scores are not guaranteed accurate. That is honest, but it doesn't address the structural issue: the benchmark's ground truth is defined by the very model family under test. A human evaluation of the non-novel labels, or a holdout set of genuinely incremental/paraphrased published papers, would be the natural fix. The reference-closure dataset construction is a real contribution, and the decision tree on novelty scores is a small but sensible choice, even if it inherits the same label bias.\n\nWho should read this: people building RAG pipelines for scientific text or studying idea-level similarity. They'll get a concrete training recipe and two benchmark resources. They should not treat the reported F1 as evidence about novelty detection in the wild.\n\nMy recommendation: send it to serious peer review. The method is novel, the writing is clear, the experiments are reproducible in structure (code and data are promised), and the flaws are fixable—but the authors should be pushed to validate the benchmark against human judgments or real papers before it is cited as a general tool.","headline":"A plausible retriever-distillation framework with two new benchmarks, but the benchmark labels are LLM-generated from the same family as the scorers, so the headline gains likely measure how well the retriever learns to reverse-engineer LLM paraphrase distribution rather than real scientific non-novelty.","tokens_in":17419,"tokens_out":2838,"would_cite":true,"duration_ms":33782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper trains a lightweight retriever on LLM-generated rephrased, partial, and incremental variants of real ideas so that novelty detection matches conceptual similarity rather than surface wording.","keywords":["scientific novelty detection","idea-level retrieval","knowledge distillation","retrieval-augmented generation","LLM-generated synthetic ideas","contrastive learning","textual vs conceptual similarity","benchmark datasets"],"falsifier":"Take a held-out set of real papers widely recognized as incremental extensions of prior work (for example, follow-up papers in the same line of research) and run the trained RAG-KD system; if it labels most of them novel even though human experts judge them non-novel, then the synthetic ground truth has not transferred to real non-novelty.","tokens_in":16397,"feed_emoji":"🔍","tokens_out":6993,"duration_ms":68298,"temperature":0.7,"pith_summary":"The paper argues that scientific novelty detection fails when systems compare texts by surface wording, because a paraphrased or extended idea can look textually distinct while being conceptually the same. It proposes to close this gap by having an LLM generate three kinds of non-novel variants of real paper ideas — rephrases, partial restatements, and incremental extensions — and then distilling these synthetic pairs into a lightweight retriever that aligns ideas by conception. On two new benchmark corpora built from closed sets of seed papers plus their references (marketing and NLP), the distilled retriever consistently outperforms standard retrievers at idea retrieval, and a retrieval-augmented novelty detection pipeline using it beats the compared baselines. If correct, the result is a practical way to audit whether a new idea is genuinely new relative to a field's prior work, bridging the gap between textual similarity and idea conception.","feed_headline":"Distilled retriever learns ideas, not words, to catch non-novel papers","feed_subtitle":"Rephrased, partial, and incremental LLM variants teach a retriever to beat text-similarity baselines on two benchmarks.","key_machinery":"The load-bearing object is the anchor–synthesized idea pair set built from LLM-generated rephrased, partial, and incremental variants of each seed idea, together with a contrastive learning loss that pulls the retriever's embedding of each synthesized variant toward its anchor and away from unrelated seed ideas. The three-way taxonomy — information equivalence (rephrase), information reduction (partial), and information addition (incremental) — is what makes the synthetic non-novel corpus cover distinct failure modes: rephrasings test vocabulary invariance, partials test subset detection, and incrementals test the hardest case of small textual overlap with large conceptual overlap. A decision tree trained on the LLM's five-level novelty scores for the retrieved candidates supplies the final novel/non-novel decision without a hand-set threshold.","core_discovery":"The central discovery is that the gap between textual similarity and idea conception can be bridged by a knowledge-distillation procedure: an LLM generates anchored non-novel variants of each seed idea — information-equivalent rephrasings, information-reducing partial statements, and information-adding incremental fusions — and the retriever is fine-tuned with a contrastive objective so that each variant's embedding is pulled toward its anchor and pushed away from other seed ideas. The resulting retriever captures conceptual overlap that standard encoders such as GTE, E5, SimCSE, Sentence-BERT, NLI, and BGE miss, especially for incremental ideas that have the lowest textual similarity to their anchors. Feeding the retriever's top candidates to an LLM with a five-level novelty rubric, then learning a decision tree on the resulting score vector, yields about 0.745 accuracy on the marketing test set and 0.747 on the NLP test set, outperforming all compared baselines including reference-alignment variants on both retrieval and classification.","pith_inferences":["The same distillation recipe should transfer to other closed-corpus sciences, such as biomedicine or computer vision, wherever seed papers and their references can be assembled into a closure set.","Because the benchmark's non-novel ground truth is generated entirely by LLMs, reported gains could shrink on real incremental papers, which typically mix paraphrase, narrowing, and extension less cleanly than synthetic variants; a human-annotated holdout of genuine incremental papers would test transfer.","The contrastive objective assumes seed ideas are mutually novel, which may be too strong in dense research areas; relaxing it with soft negatives could yield better-calibrated novelty scores.","Since the LLM only sees the top-K retrieved candidates, the system's ceiling is set by retrieval recall; retrieving diverse candidates rather than only the most similar ones could further improve novelty detection."],"forward_implications":["Automated novelty screening for submissions or grant proposals could flag rephrased or incremental contributions against a closed corpus of the field's prior work.","The three-way synthesis taxonomy gives a precise vocabulary for why an idea is not new: it restates an existing conception, covers only part of one, or makes a small extension of known ideas.","Because the retriever is lightweight and the LLM only scores a handful of retrieved candidates, the pipeline is cheap enough to run over whole corpora rather than single queries.","The decision-tree layer on LLM novelty scores keeps the final verdict interpretable: a low novelty label can be traced back to which retrieved idea most influenced it."],"supporting_citations":[{"why":"Provides the URPC baseline that measures novelty by atypical journal-pair combinations in a paper's references.","marker":"Uzzi et al., 2013"},{"why":"Provides the PES baseline that detects novelty through semantically distant biological entity pairs.","marker":"Liu et al., 2022"},{"why":"Provides the CD baseline that scores novelty from cosine distances among word embeddings of cited references.","marker":"Shibayama et al., 2021"},{"why":"Provides the SciMON baseline that judges novelty by thresholding cosine similarity against existing ideas.","marker":"Wang et al., 2024"},{"why":"Supplies the BGE retriever backbone used for the vanilla, reference-alignment, and LLM-KD comparisons.","marker":"Xiao et al., 2024"},{"why":"Supplies the Sentence-BERT paraphrase backbone used as one of the six retriever baselines.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies the SimCSE backbone and motivates the contrastive learning objective used to train the distilled retriever.","marker":"Gao et al., 2021"},{"why":"Motivates the retrieval-augmented generation strategy the paper applies to novelty detection.","marker":"Gao et al., 2023"}],"fun_headline_variants":["LLM-distilled retriever spots non-novel ideas by concept, not text","Train a retriever to think like an LLM for novelty detection","Idea-level distillation beats text similarity for novelty detection","LLM teaches retriever to catch rephrased, partial, incremental ideas"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that LLM-generated rephrased, partial, and incremental variants are a faithful stand-in for real non-novel scientific ideas; if real incremental papers do not resemble these synthetic variants, the benchmark may overstate real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["LLM-distilled retriever spots non-novel ideas by concept, not text","Train a retriever to think like an LLM for novelty detection","Idea-level distillation beats text similarity for novelty detection","LLM teaches retriever to catch rephrased, partial, incremental ideas"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1355,"prompt_tokens":962,"completion_tokens":393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":578,"tokens_out":393,"duration_ms":4560,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:17:04.523588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of real papers widely recognized as incremental extensions of prior work (for example, follow-up papers in the same line of research) and run the trained RAG-KD system; if it labels most of them novel even though human experts judge them non-novel, then the synthetic ground truth has not transferred to real non-novelty.","supporting_citations":[],"review_version":1}