{"id":"29070dc6-14ca-4bec-bb45-af837e7ca00c","arxiv_id":"2506.00249","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new dataset and MAG-guided triplet-loss fine-tuning, plus LLM reranking, improves retrieval of methodologically inspirational papers for research proposals by several points over strong baselines.","lead":"The paper introduces a new retrieval task, Methodology Inspiration Retrieval, where a system given a research problem and its motivation must retrieve papers whose methods could inspire a solution. The authors build a dataset from citation-intent labels, fine-tune dense retrievers with a Methodology Adjacency Graph, and report gains of +5.4 Recall@3 and +7.8 mAP over strong baselines, with further gains from LLM reranking.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on the citation-intent proxy for 'methodological inspiration'; the paper's own error analysis and Limitations section show the test labels are noisy, so the +5.4/+7.8 gains may partly reflect learning citation-intent annotation patterns rather than true inspiration.","rationale":"The central claim — that a retriever fine-tuned on citation-intent-derived graphs substantially improves methodological-inspiration retrieval — is only as strong as the ground truth that defines 'inspiration'. The paper operationalizes this solely through MultiCite's Uses/Extends intents, and both the training triplets and the test metrics use the same noisy labels. The paper's own error analysis and Limitations section publicly document missing and incorrect annotations, trivial references, and subjectivity, so the proxy is acknowledged to be imperfect. In my reading, this is the most load-bearing assumption: if the labels do not track true methodological inspiration, the numerical gains may reflect learning the annotation pattern rather than a practically useful signal. I do not think this warrants rejection, because the dataset and task definition are valuable regardless of exact numbers, and the authors transparently disclose the limitations. The reader's CONDITIONAL verdict with high confidence is appropriate. My proposed re-annotation experiment is a concrete way to settle whether the proxy is adequate and whether the reported improvements survive a cleaner ground truth. The paper deserves credit for releasing the dataset, conducting detailed ablations, and including an honest error analysis; those strengths are compatible with a conditional acceptance.","tokens_in":37159,"tokens_out":6586,"duration_ms":66178,"concrete_test":"Re-annotate the test labels for all 139 test proposals: have at least three independent NLP researchers label each cited paper as 'methodological inspiration' or not, following the paper's definition but blinded to the MultiCite intents. Compute inter-annotator agreement (e.g., Cohen's kappa) with the MultiCite-derived labels. Then re-run the main comparison (Stella_1.5B base vs fine-tuned, and the best LLM reranker) on (i) the subset of proposals where all annotators agree with the original labels, and (ii) the full set with majority-vote corrected labels. If the +5.4 R@3 / +7.8 mAP gains do not persist (e.g., shrink by more than half or change sign), the central claim depends on the noisy citation-intent proxy rather than on genuine methodological inspiration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that fine-tuning on the Methodology Adjacency Graph improves retrieval of methodologically inspirational papers. The graph's edges and the test-set labels are both defined by MultiCite citation intents: Section 4.1 states 'We view papers cited with citation intents (Uses or Extends) as potential inspirations.' This is the only operationalization of 'methodological inspiration' used anywhere in training or evaluation. The concern is construct validity: if the Uses/Extends proxy is inaccurate or incomplete, the reported gains measure citation-intent prediction, not inspiration retrieval. The paper itself supplies evidence of this risk. The Limitations section admits that MultiCite annotations 'occasionally missed inspirational citations, while simultaneously including some trivial references (e.g., mere use of GPT-4).' Appendix D's error analysis of 15 proposals with 39 relevant papers reports 17 incorrect/missing annotations and 15 misclassifications — in that error-analysis sample, over 80% of relevant papers have some labeling problem. Because the same noisy labels generate the MAG triplets (training) and the gold standard (test), the model may be fitting annotation noise rather than a genuine inspiration signal. A further consequence is metric bias: mAP is computed per relevant paper, and missing positives or extra negatives change denominators unequally across systems, so the +7.8 mAP gain could be an artifact of label incompleteness rather than ranking quality. The paper reports no human evaluation of whether retrieved papers are actually inspirational, and no inter-annotator agreement on the test labels. This makes the central claim conditional on a proxy whose quality is acknowledged to be imperfect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Methodology Inspiration Retrieval (MIR), a task in which a system ranks papers from a literature corpus that could inspire a solution methodology for a given research proposal containing only a problem and motivation. The authors construct MIR-MultiCite by adapting the MultiCite citation-intent dataset: papers cited with 'Uses' or 'Extends' intents are treated as methodological inspirations, and this labeling is used both to build a Methodology Adjacency Graph (MAG) for fine-tuning dense retrievers and to create test ground truth. They propose a joint triplet-loss fine-tuning strategy over MAG-derived triplets, an arXiv-based weak-label augmentation scheme, and LLM-based re-ranking strategies, including a MIR-Agent that decomposes proposals before judging relevance. On their test set they report substantial improvements over strong baselines (e.g., +5.4 Recall@3 and +7.8 mAP on the extended corpus), further gains from LLM re-ranking, and downstream benefits in LLM-as-a-judge idea-generation evaluations. The paper includes ablations, a two-corpus evaluation design, qualitative analyses, and an explicit error analysis and limitations section.","tokens_in":37424,"tokens_out":4729,"duration_ms":49967,"significance":"If the reported effects are robust, the paper makes a useful contribution: it defines a realistic retrieval task that goes beyond surface semantic similarity, releases a dataset, and demonstrates that citation-intent-derived graph structure can be distilled into dense retrievers through a principled triplet formulation. The ablation study in Table 5 and the training-data scaling results in Table 7 provide internally consistent evidence for the method's components, and the authors are unusually transparent about timeline contamination, label noise, and subjectivity in their ground truth. The main limitation is construct validity: the training signal and the evaluation labels come from the same MultiCite citation-intent construct, and the paper's own error analysis shows that this construct is noisy. The downstream idea-generation evaluation is a step toward external validation, but it is still LLM-judged. With a reframed central claim or an independent annotation check, the paper's core technical contribution would be solid; as written, the strength of the conclusion exceeds what the evaluation can support.","major_comments":[{"comment":"The task's operationalization of 'methodological inspiration' is the MultiCite 'Uses'/'Extends' citation intent, and the same intent labels are used to build the MAG training triplets and the test gold standard. Section 4.1 states: 'We view papers cited with citation intents (Uses or Extends) as potential inspirations.' Consequently, the reported +5.4/+7.8 gains may partly reflect learning to predict the citation-intent annotation pattern rather than retrieving papers that an independent expert would consider inspirational. This concern is reinforced by the paper's own evidence: Appendix D reports 17 incorrect or missing annotations and 15 misclassifications among 15 analyzed proposals, and the Limitations section concedes that MultiCite annotations 'occasionally missed inspirational citations, while simultaneously including some trivial references (e.g., mere use of GPT-4).' I do not regard this as fatal, because the paper explicitly frames the dataset around citation intents, but the abstract and conclusion overstate the claim by speaking of 'implicit domain-intuition for prioritizing inspirational literature.' Please either reframe the central claim as retrieving methodologically cited papers, or add a human or independently annotated evaluation on a sample that validates the learned ranking against genuine methodological inspiration.","section":"§4.1, §5.1.2, Limitations, Appendix D"},{"comment":"The headline numbers in the abstract and conclusion (+5.4 Recall@3, +7.8 mAP) are the Extended Corpus results from Table 2; on the Restricted Corpus the corresponding gains over the strongest baseline are only +2.76 Recall@3 and +1.16 mAP. Meanwhile, Section 4.2 admits that the Restricted Corpus 'introduces potential timeline contamination risks, as there is no guarantee that the corpus papers predate the test proposal.' The evaluation also rests on only 139 test proposals and 193 relevant papers, yet no confidence intervals, bootstrap estimates, or per-proposal variance statistics are reported anywhere. Please state in the abstract which corpus setting the headline numbers refer to, and report significance or variance estimates so the reader can assess the stability of the claimed gains.","section":"§4.2, Table 2, §7"},{"comment":"The full-paper re-ranking experiments, which support the Conclusion's claim of '+5.7 in Recall@3 and +5.8 in mAP,' are evaluated on only 75 of the 139 test proposals, selected by full-text availability. Section 5.2.2 says: 'For this analysis, we identify a sub-set of the test-set proposals for which the full text of top-k (k=75) retrieved papers is available.' No analysis is provided of how these 75 proposals differ from the remaining 64, so selection bias cannot be ruled out. Please report the full-paper re-ranking results on the full test set, or justify that the subset is representative and treat the subset explicitly when citing these numbers.","section":"§5.2.2, Table 6, §7"},{"comment":"The downstream validation in RQ3 is weaker than the retrieval evaluation and does not fully resolve the construct-validity concern. The IA-Score in Section B.5.2 measures alignment of generated methodologies with 'the ground truth methodology (extracted from the original seed paper abstract),' but Section 3 defines the MIR query as containing no methodology; measuring alignment against the seed paper's own methodology evaluates a related but different objective. The IdeaArena and IA-Score judgments are both produced by LLMs with no human validation. Please present these results as exploratory evidence and, if possible, add a small human study on a sample of proposals to validate the downstream benefit.","section":"§6.4, §B.5.2"}],"minor_comments":[{"comment":"The abstract reports LLM re-ranking improvements of +4.5 in Recall@3 and +4.8 in mAP, while the Conclusion reports +5.7 and +5.8 for the full-paper few-shot MIR-Agent setting; please make the setting for each set of numbers explicit.","section":"Abstract vs. §5.2/§7"},{"comment":"There is a typo in RQ1: 'SPERCTER2' should be 'SPECTER2'.","section":"§6.4"},{"comment":"'respecitvely' should be 'respectively'; footnote 7's 'Methodologysection' should be 'Methodology section'; Appendix B.4 contains 'o8ther' instead of 'other'.","section":"§6.1"},{"comment":"The paper says listwise and pairwise re-ranking 'showed no improvement' but does not report their numerical results; a sentence summarizing their performance would make the decision to exclude them more verifiable.","section":"§5.2/B.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the authors are transparent about limitations, which is commendable. The main risk is that the central claim is framed more strongly than the evaluation supports because the training and test labels come from the same noisy citation-intent proxy. A revision that reframes the claim, adds significance/variance estimates, and ideally includes a small independent human evaluation would make the contribution publishable. No concerns about novelty or ethical compliance beyond those stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it gives the community something usable: a new task definition (MIR), a released dataset (MIR-MultiCite), and a fine-tuning recipe that consistently beats strong retrieval baselines. Second, the headline numbers are real but they are measured against a noisy, self-referential gold standard, so the central claim about \"inspiration\" is less solid than the abstract suggests.\n\nWhat's new: framing retrieval of methodologically inspirational papers as a distinct task, building a graph from MultiCite citation intents (Uses/Extends), and a joint triplet loss sampling from that graph. The ablations are honest and the dataset release is a concrete contribution. The paper also differentiates itself well from ReasonIR and ResearchBench.\n\nThe soft spots. The stress-test concern lands: both the MAG edges and the test labels come from the same citation-intent annotation. The retriever is essentially trained to predict which papers would be cited with \"uses\" or \"extends,\" and evaluated on the same. The paper's own error analysis finds 17 missing/mislabeled and 15 misclassified among 39 relevant papers in 15 proposals—so the ground truth is quite noisy. That doesn't kill the paper, but it means the +7.8 mAP gain could partly be the model learning annotation patterns rather than discovering true inspiration. The test set is only 139 proposals, with no variance or significance tests, and the restricted corpus has admitted timeline contamination; the extended corpus mitigates that but is still small. The downstream LLM-as-judge evaluation is suggestive at best: small ELO differences, no human judges.\n\nWhat holds up: the internal logic of the method is coherent, the ablations support the design choices, and the qualitative examples do show cases where the fine-tuned retriever surfaces methodologically relevant papers that naive retrieval misses. If you accept citation intent as a proxy for inspiration, the paper does what it claims.\n\nVerdict: this deserves a serious referee. It's a solid first step, not a breakthrough. I'd ask for human evaluation of retrieved inspirations, bootstrap CIs across models, and a clearer statement that the task is citation-intent prediction until a human eval says otherwise.\n\nWho it's for: people working on retrieval for scientific discovery, and anyone building idea-generation agents.\n\nRecommendation: engage, with conditional acceptance in mind.","headline":"Useful new task definition and dataset, but the headline inspiration gains are measured against a noisy, self-referential citation-intent gold standard.","tokens_in":38041,"tokens_out":2710,"would_cite":true,"duration_ms":26687,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning dense retrievers on a citation-intent-annotated graph (the Methodology Adjacency Graph) substantially improves retrieval of methodologically inspirational papers, with LLM re-ranking adding further gains.","keywords":["methodology inspiration retrieval","citation intent","dense retrieval","triplet loss","LLM re-ranking","automated scientific discovery","research proposal","literature-based discovery"],"falsifier":"Re-annotate the MIR-MultiCite test proposals with an independent operational definition of 'methodological inspiration'—for example, ask expert researchers to mark, for each proposal, which papers in the corpus could genuinely inspire a solution, without seeing the original citation intents—then re-run the fine-tuned retrievers and baselines on that relabeled ground truth. If the MAG-finetuned retriever does not maintain its lead over the strong semantic baseline on the relabeled set, or if the disagreement between the two label sets is large, the central claim that citation-intent-derived graphs teach retrievers to identify inspirational literature is refuted.","tokens_in":36952,"feed_emoji":"💡","tokens_out":9476,"duration_ms":84492,"temperature":0.7,"pith_summary":"Scientific discovery systems that ground idea generation in literature retrieve papers by semantic similarity, which surfaces keyword overlaps rather than papers whose methods could actually solve a stated problem. This paper defines Methodology Inspiration Retrieval (MIR) as the task of ranking papers by their potential to inspire a methodology for a given research proposal, using only the proposal's problem and motivation as the query. It builds a Methodology Adjacency Graph from citation-intent-annotated citations, and fine-tunes dense retrievers with a joint triplet loss so that proposals, their methodological citation contexts, and the cited papers are pulled close together while non-methodological neighbors are pushed away. On the resulting MIR-MultiCite test set this lifts Recall@3 by 5.4 and mean Average Precision by 7.8 over strong baselines, and an LLM-based re-ranker (MIR-Agent) adds further gains of 4.5 in Recall@3 and 4.8 in mAP. The paper further shows that ideas generated from MIR-grounded retrieval score higher on LLM-as-judge novelty, feasibility, and alignment with ground-truth methodology.","feed_headline":"Citation graph lifts inspirational-paper retrieval by 7.8 mAP","feed_subtitle":"Fine-tuning retrievers on citation-intent graphs beats semantic search, raising Recall@3 by 5.4 and mAP by 7.8.","key_machinery":"The central object is the Methodology Adjacency Graph (MAG): a directed graph whose nodes are research proposals, whose outgoing edges point to cited papers, and whose edges are annotated with citation intents (methodology: 'uses'/'extends'; non-methodology: everything else) and the citation contexts that describe how the cited paper was used. The MAG feeds a joint triplet loss with three triplets per sample—anchor proposal vs. positive cited paper, anchor citation context vs. positive proposal, and anchor citation context vs. positive cited paper—so that the retriever learns the relationship between a problem, the textual description of how a cited paper contributes, and the paper itself. Negative sampling is MAG-guided: hard negatives are papers in the same scientific domain (computed from SciBERT representations of extracted named entities) that are not methodologically relevant, pushing the retriever to separate semantic neighbors from methodological allies. A second mechanism is the MIR-Agent LLM re-ranker, which decomposes a proposal into sub-problems and a generic action plan, evaluates each retrieved paper's applicability to those sub-problems, and issues a binary relevance judgment, optionally with few-shot exemplars drawn from the MAG and with full-paper context.","core_discovery":"The paper's central claim is that the methodological lineage encoded in citation graphs, distilled into a dense retriever by a joint triplet loss, gives the retriever an implicit 'domain intuition' for inspirational literature that purely semantic similarity lacks. The authors operationalize methodological inspiration through the MultiCite citation intents 'uses' and 'extends', treat proposals as graph nodes whose outgoing edges point to methodologically relevant cited papers, and train with three interlocking triplets: proposal-to-cited-paper, citation-context-to-proposal, and citation-context-to-cited-paper, using hard negatives selected by domain similarity. They report that fine-tuning SPECTER2 and Stella models this way yields consistent improvements of up to +5.4 Recall@3 and +7.8 mAP over strong zero-shot baselines, and that a few-shot LLM agent that decomposes the proposal into sub-problems before judging methodological relevance adds further improvements of up to +5.7 Recall@3 and +5.8 mAP when it reads full papers. They also report that grounding idea generation in MIR retrieval raises LLM-judged novelty, significance, feasibility, clarity, and effectiveness scores over naive retrieval, and improves alignment with the ground-truth methodology from 0.142 to 0.170.","pith_inferences":["The two intents the paper merges as 'methodology'—'uses' and 'extends'—plausibly mark distinct relationships: direct reuse versus conceptual extension. Splitting them into separate positives would test whether the MAG fine-tuning learns one or two notions of inspiration, and could yield a more informative ranking for downstream discovery.","The MIR-Agent's gains come from decomposing a proposal into sub-problems before judging relevance; this is a domain-specific instantiation of chain-of-thought retrieval. A testable extension is whether the same decomposition step helps on other reasoning-intensive retrieval benchmarks or whether the sub-problem decomposition only helps because 'methodological relevance' is naturally sub-problem-st","Because the paper's error analysis finds that ground-truth annotation errors are a major error source (17 of 39 papers in the analyzed sample), the true ceiling of the task may be higher than reported, and a cleaner label set—for instance, LLM-driven synthetic citation-intent labeling that the authors suggest—could change the relative ranking of the embedding retriever and the LLM re-ranker.","The 0.170 vs 0.142 IA-Score alignment is modest in absolute terms (about 1 in 6 generated ideas matching ground-truth methodology), so the downstream benefit of MIR is real but limited; a more direct test would measure whether MIR-grounded ideas are judged by human experts as more novel and implementable, not just more aligned with a known methodology."],"forward_implications":["Retrievers fine-tuned on citation-intent graphs can replace generic semantic search in the grounding stage of automated idea generation, since the paper shows MIR-grounded generations score higher on LLM-as-judge novelty, significance, feasibility, clarity, and effectiveness than naive-retrieval-grounded generations.","MIR enables a realistic retrieval setting for hypothesis-generation systems: the query is only the research problem and motivation, with no assumption that citation neighbors or ground-truth inspirations of a seed paper are already known, so novel methodology generation is not constrained to resemble a seed paper's own approach.","LLM re-ranking of a small top-k list (k around 10 to 20) is a practical complement to embedding retrieval: the paper's cost analysis puts Gemini re-ranking at roughly a cent to a few cents per proposal, and the best configurations use few-shot exemplars sampled from the citation graph plus full-paper context.","The joint triplet-loss training recipe with weakly labeled, domain-distributed augmented data scales: training on weakly labeled arXiv data alone already lifts Recall@3 and mAP over the base model, indicating the recipe is robust to label noise and transferable to other domains with available citation corpora."],"supporting_citations":[{"why":"Supplies the MultiCite dataset whose eight citation intents define the ground truth for MIR-MultiCite and the Methodology Adjacency Graph; the papers cited with 'uses' or 'extends' are the positives the retriever must surface.","marker":"(Lauscher et al., 2022)"},{"why":"SPECTER is the citation-informed baseline the authors fine-tune, and its citation-aware document representations motivate using citation relationships to capture methodological lineage.","marker":"(Cohan et al., 2020)"},{"why":"SPECTER2 is a strong scientific baseline in the comparison; fine-tuning SPECTER2 isolates the contribution of MAG-guided sampling on a citation-trained model.","marker":"(Singh et al., 2022)"},{"why":"ReasonIR is the closest parallel approach (contrastive training for reasoning-intensive retrieval) and a strong competing baseline; the paper differentiates its objective from ReasonIR and must beat it.","marker":"(Shao et al., 2025)"},{"why":"SciBERT computes the domain representations used for hard-negative selection and supplies the weak-label citation-intent classifier for arXiv data augmentation; it is also a baseline.","marker":"(Beltagy et al., 2019)"},{"why":"Provides the LLM-as-re-ranker paradigm that the paper adapts into pointwise and MIR-Agent re-ranking strategies.","marker":"(Sun et al., 2023)"},{"why":"IdeaArena's pairwise ELO evaluation is used to show that MIR-grounded idea generation outperforms naive-retrieval-grounded generation on quality criteria.","marker":"(Li et al., 2024b)"},{"why":"IA-Score measures alignment of generated ideas with ground-truth methodology, used as the second downstream evaluation of MIR benefit.","marker":"(Kumar et al., 2024)"},{"why":"Independent argument that inspiration retrieval is a key bottleneck for automated scientific discovery; motivates the task definition.","marker":"(Liu et al., 2025)"}],"fun_headline_variants":["Citation graph lifts inspirational-paper retrieval: +7.8 mAP","Methodology inspiration: citation graph beats semantic search","Fine-tuning on citation intents improves paper retrieval by 7.8 mAP","Inspiration retrieval from papers: citation graph adds 5.4 Recall@3","Citation lineages reveal methodological inspiration for scientific discovery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a cited paper counts as a methodological inspiration exactly when its citation intent is 'uses' or 'extends'; if real-world inspiration is not captured by those two labels, the benchmark measures citation-intent prediction rather than inspiration, and the ground truth carries the original annotations' noise (17 incorrect or missing and 15 misclassified labels in a 39-paper error-analysis sample).","fun_headline_variants_meta":{"raw":{"variants":["Citation graph lifts inspirational-paper retrieval: +7.8 mAP","Methodology inspiration: citation graph beats semantic search","Fine-tuning on citation intents improves paper retrieval by 7.8 mAP","Inspiration retrieval from papers: citation graph adds 5.4 Recall@3","Citation lineages reveal methodological inspiration for scientific discovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1404,"prompt_tokens":1033,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":649,"tokens_out":371,"duration_ms":4213,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:09:15.719918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the MIR-MultiCite test proposals with an independent operational definition of 'methodological inspiration'—for example, ask expert researchers to mark, for each proposal, which papers in the corpus could genuinely inspire a solution, without seeing the original citation intents—then re-run the fine-tuned retrievers and baselines on that relabeled ground truth. If the MAG-finetuned retriever does not maintain its lead over the strong semantic baseline on the relabeled set, or if the disagreement between the two label sets is large, the central claim that citation-intent-derived graphs teach retrievers to identify inspirational literature is refuted.","supporting_citations":[],"review_version":1}