{"id":"b16f02f3-86c5-4c3a-b89a-e9ce8ea440c4","arxiv_id":"2506.18316","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A two-stage citation discovery system, using LLM relation extraction plus top-k retrieval and LLM re-ranking, achieves an F1 of 0.344 in retrieval-only mode on a 1,000-query SCIDOCA sample.","lead":"This shared-task paper describes a two-stage citation search system: an AI model first pulls out relationship triples from a paragraph, then a large language model picks the best matching paper abstract from a shortlist. The system reports moderate F1 scores on a 1,000-query sample, but the reported numbers do not clearly support the paper's claim that the LLM re-ranking step helps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The effectiveness claim is unsupported: the final LLM pipeline scores F1=0.2912 vs 0.344 for relation-only retrieval, and the precision/recall values in Tables 1 and 2 imply different, undefined F1 protocols.","rationale":"The reader identified the same load-bearing weakness: the F1 scores in Tables 1 and 2 are not comparable because the metric is undefined and the precision-recall pairs in Table 2 cannot arise from a single-output-per-query prediction task. My stress-test confirms this and adds a sharper internal contradiction: the paper claims LLM inference enhances recall, yet Table 2 recall values are less than half of the corresponding retrieval-only recall values in Table 1, and the final pipeline F1 (0.2912) is below the relation-only retrieval F1 (0.344). This is not merely a missing detail; it breaks the central claim as stated. I also note the evaluation uses a random 1,000-query subset of the training set rather than the official test set, with no significance testing, so the small margin of relation-based-10 (0.344) over dense retrieval-10 (0.3354) is weak evidence even if the protocol were consistent. I credit the paper for clearly stating the subset size and model names, and for reporting precision, recall, and F1 for all configurations, but the missing metric definition prevents reinterpretation. The reader's REJECT verdict is appropriate; my analysis does not move it.","tokens_in":4730,"tokens_out":3607,"duration_ms":35789,"concrete_test":"Obtain the per-query predictions and candidate gold annotations for the same 1,000-query subset, then recompute Table 2 under an explicitly defined F1 protocol. If each query has one gold citation and the system returns one ID, precision must equal recall; the current P/R split proves a different protocol is in use. Recomputing Table 1 under that same protocol would settle whether the 0.344 vs 0.2912 gap is real. As a secondary check, run the full system on the official SCIDOCA test set and compare against relation-only retrieval; if the final pipeline does not beat 0.344, the Section 5 conclusion should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 1 and Section 5 claim that combining relation extraction, retrieval, and LLM inference is effective. The only evidence is Tables 1 and 2. For that claim to hold, the F1 values must be comparable and the final system should at least not be worse. Neither condition is met. Table 1 lists recall-precision pairs for top-k retrieval; Table 2 lists recall-precision pairs for LLM inference that outputs one citation per query. In a single-answer task, per-query precision and recall are identical (both equal accuracy), so the P != R rows in Table 2 cannot be standard query-level classification metrics. The paper never defines the F1 computation or the pooling/protocol used. Moreover, the reported numbers are internally contradictory: Section 4 says LLM inference 'significantly enhances recall compared to raw retrieval scores,' but Table 2's relation-based recall is 0.4626, far below the 0.8506 recall of relation-based-10 in Table 1. The full pipeline F1 (0.2912) is also below the relation-only retrieval F1 (0.344) and below LLM+TF-IDF (0.2980). Therefore the central effectiveness claim is not supported by the reported data; either the metrics are incomparable or the final system is worse than its first stage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes Team LA's system for Task 1 of the SCIDOCA 2025 shared task on citation discovery. The proposed pipeline first uses Mistral-7B-Instruct to extract relation triples from a query paragraph, retrieves the top-k candidate abstracts using those relational features, and then prompts the same or a second LLM to choose the final citation. Experiments on 1,000 randomly selected queries compare TF-IDF, dense vector retrieval, and relation-based retrieval at top-k = 10, 15, 20, plus LLM-based final selection with each retriever, reporting precision, recall, and F1. The authors conclude that integrating relation extraction with LLM-based citation retrieval is effective.","tokens_in":4917,"tokens_out":15553,"duration_ms":146321,"significance":"If the reported numbers were valid and properly defined, the paper would provide a modest practical baseline for citation discovery in long-abstract settings: relation-based top-10 retrieval reaches F1 0.344, exceeding TF-IDF (0.3297) and dense retrieval (0.3354), and the relation-extraction idea is easy to transfer. The authors are transparent about evaluating on 1,000 queries and about using standard baselines. However, the evaluation protocol is not defined, the central claim about LLM integration is not clearly supported by the tables as written, and the experiments are run on a training sample rather than a held-out test set. Because the reported F1 values cannot be interpreted or compared as written, the positive retrieval-stage result cannot be credited without further details.","major_comments":[{"comment":"The paper never defines how precision, recall, and F1 are computed, and the reported pairs cannot be reconciled with a standard evaluation. For a task with one correct citation per query, a system that emits a single prediction has equal per-query precision and recall (both are 0 or 1), yet every row of Table 2 reports different values (e.g., LLM+TF-IDF: recall 0.4079, precision 0.2347). Even for ranked top-k retrieval, if exactly one abstract per query is relevant, precision should equal recall/k; for relation-based-10 this would give 0.8506/10 = 0.0851, not the reported 0.2156. If multiple relevant citations or multi-output predictions are allowed, that protocol must be stated so that the F1 scores across Tables 1 and 2 are comparable. As written, the table values cannot be used to rank the systems.","section":"Section 4, Tables 1 and 2"},{"comment":"The manuscript does not specify the candidate-set size used for the LLM stage: Section 4 says the first retrieval step collects the top 20 documents, but Table 2 is not labeled with a top-k and the text later discusses Table 1's top-10 results as the main retrieval outcome. This ambiguity is load-bearing. If the LLM operates on top-20 candidates, LLM with relation-based retrieval (F1 0.2912) improves over relation-based-20 retrieval alone (F1 0.2711); if the comparison is against the best retrieval row, relation-based-10 (F1 0.344), the LLM stage degrades. The paper must state precisely which retrieval results the LLM is applied to and which comparison supports the effectiveness claim.","section":"Section 4, top-k setting; Sections 1 and 5"},{"comment":"Several statements in the analysis are directly contradicted by the tables or misread the table entries. The claim that LLM inference 'significantly enhances recall compared to raw retrieval scores' is not supported under any reading: all LLM recalls in Table 2 (0.4079, 0.3253, 0.4626) are far below all retrieval recalls in Table 1, including at top-20 (e.g., relation-based-20 recall 0.967). The claim that relation-based retrieval shows a 'substantial improvement in precision (e.g., 0.8506 for top-10)' mistakes recall for precision; the precision of relation-based-10 is 0.2156, only slightly higher than dense retrieval-10 (0.2095). Also, because LLM with TF-IDF (0.2980) outperforms LLM with relation-based retrieval (0.2912), the conclusion that relation-based candidate refinement is 'important' before passing candidates to the LLM is not supported by the reported best configuration.","section":"Section 4, Tables 1 and 2"},{"comment":"The abstract states that the framework is evaluated on the 'training dataset provided by the SCIDOCA 2025 organizers,' and Section 4 says experiments use '1,000 randomly selected queries from the dataset' without stating whether these are held out or part of the training set. Reporting results on a training sample does not estimate performance on unseen paragraphs, and the selection of top-k on the same sample (with k=10 reported as best) adds a selection effect. The paper should either evaluate on the official held-out/test split or clearly limit the claims to the training-sample results.","section":"Abstract and Section 4, evaluation sample"},{"comment":"The proposed method is described at a level that prevents reproduction and attribution of the reported gain to relation extraction. The paper does not specify how the extracted relational triples are converted into a retrieval query, which similarity function is used for relation-based retrieval, how the TF-IDF and dense retrieval baselines are implemented (e.g., the embedding model for dense retrieval), or the exact prompts used for relation extraction and final citation selection. Without these details, the reader cannot determine whether the improvement comes from the relation-based query representation or from another component of the pipeline.","section":"Section 3, Methods"}],"minor_comments":[{"comment":"The keyword 'Text retrival' is misspelled; it should be 'Text retrieval'.","section":"Keywords"},{"comment":"The reference for Gated Transformer Retrieval (GTR) appears as a placeholder '[?]' and must be completed.","section":"Section 2.1"},{"comment":"There is a stray space in 'T raditional' at the start of the subsection.","section":"Section 2.2"},{"comment":"The row label 'LLM inference with Dense retrieval0.3253' is missing a space before the number, and 'relation-base' should be 'relation-based'.","section":"Section 4, Table 2"},{"comment":"The caption 'Each method is implemented with the most relevant documents from the different top-k' is unclear; it should say 'Each method is evaluated at different top-k values.'","section":"Table 1 caption"},{"comment":"The paper introduces the acronym DVR (Dense Vector Retrieval) but then uses only 'dense retrieval'; please use the notation consistently or remove the acronym.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"This is a short shared-task system description with an interesting retrieval idea, but the evaluation reporting is not at the level expected for a journal. The main load-bearing issues are the undefined F1 protocol, the ambiguous top-k comparison for the LLM stage, and the evaluation on an unlabeled training sample. These are fixable in principle: the authors should define the metric precisely, state which candidate set each table uses, correct the recall/precision misstatements, and temper the effectiveness claims. If a revision does not resolve these points, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a shared-task system description. The only genuinely new thing is the specific F1 numbers on a 1,000-query slice of SCIDOCA training data. The pipeline — prompt an LLM to extract relation triples from the query paragraph, use those triples for top-k retrieval, then let a second LLM choose one candidate — is a sensible composition of existing parts, and the paper honestly reports baselines. That deserves credit: the retrieval-stage numbers (relation-based-10 F1 0.344 vs dense-10 0.3354) suggest the relation-focused query rewrite helps retrieval, even if the gain is small and the sample is one random subset.\n\nThe soft spot is the central effectiveness claim, and it is not minor. The abstract and conclusion say the full two-stage pipeline is effective, but Table 2 shows the LLM stage with relation-based retrieval at F1 0.2912, below the retrieval-only 0.344 and below TF-IDF+LLM 0.2980. Section 4 says LLM inference significantly enhances recall, yet relation-based recall drops from 0.8506 to 0.4626. Either the two tables use different evaluation protocols, or the final system is worse than its first stage. The paper never defines the F1 computation. In a task where the system outputs one candidate per query, per-query precision and recall are identical; Table 2's differing P/R rows show something else is being measured, but the reader cannot tell what. The evaluation on a random 1,000-query slice of the training set is also a limitation, and top-k is selected on that same slice. There are no error bars or significance tests, so the few-point differences are not interpretable.\n\nI agree with the stress-test note: the load-bearing contradiction stands. The missing metric definition alone would be fixable, but the contradiction between the text and Tables 1–2 needs a rewrite, not a copy edit.\n\nWho is this for? People tracking shared-task baselines, and instructors who want a compact example of how undefined metrics can sink a claim. It does not deserve a cite in my own work. For a serious IR venue I would not accept it in current form; it needs either corrected numbers, a defined protocol, or a much more careful statement of what was measured.","headline":"A shared-task system report with a useful retrieval trick, undone by an undefined F1 metric and a central claim its own tables contradict.","tokens_in":5501,"tokens_out":4065,"would_cite":false,"duration_ms":37327,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that extracting relation triples with an LLM and using them to retrieve top-k candidates, followed by a second LLM citation selection, is an effective zero-shot approach to citation discovery.","keywords":["citation discovery","relation extraction","zero-shot retrieval","large language models","scientific document retrieval","text retrieval","dense retrieval","SCIDOCA 2025"],"falsifier":"Recompute precision and recall on the same 1,000 queries under a written-out protocol with one predicted citation per query judged by exact abstract ID, and check whether relation-based-10 still beats dense retrieval-10; if the ordering flips or the score differs materially, the central claim would not survive.","tokens_in":4456,"feed_emoji":"🔎","tokens_out":5216,"duration_ms":46091,"temperature":0.7,"pith_summary":"This paper tries to establish that citation discovery in long, near-duplicate scientific abstracts can be handled by first extracting structured relation triples from the query paragraph with a large language model, retrieving the top-$k$ candidate abstracts against those triples, and then letting a second LLM pass select the final citation. The claim matters because candidate sets in the task are deliberately seeded with distracting abstracts, so surface-level keyword or dense matching alone tends to trade precision against recall. On a 1,000-query sample, the relation-based top-10 retrieval reports $F_1=0.344$, above TF-IDF ($0.3297$) and dense retrieval ($0.3354$). The paper concludes that combining relation extraction with LLM-based retrieval is an effective zero-shot recipe for citation prediction.","feed_headline":"Relation-based retrieval beats TF-IDF and dense search in citation task","feed_subtitle":"LLM-extracted relation triples filter candidates before a second LLM picks the citation, reaching top-10 F1 of 0.344.","key_machinery":"The central object is relation-based retrieval: an LLM is prompted to convert the query paragraph into structured triples or key concepts, these relational features are used to score and filter the candidate pool to the top 20, and a second LLM compares the surviving abstracts against the paragraph to output the citation ID. The mechanism does the work of concentrating the candidate set before the expensive LLM decision, assuming the extracted triples capture the semantic connection the citation is meant to support.","core_discovery":"The paper's central claim is that document-level relational structure, extracted as triples by an LLM, acts as a discriminative retrieval signal that separates the true citation from highly similar distractors. In the authors' experiments, relation-based retrieval at top-10 achieves recall 0.8506, precision 0.2156, and $F_1=0.344$, outperforming both TF-IDF-10 ($F_1=0.3297$) and dense retrieval-10 ($F_1=0.3354$). A second LLM inference stage raises recall to 0.4626 for the relation pipeline at lower precision 0.2125, giving $F_1=0.2912$; the authors interpret this as the LLM adding recall on top of the precision-oriented relation filter. The conclusion is that combining relation extraction with LLM-based retrieval is an effective approach for citation prediction.","pith_inferences":["A testable extension the authors do not run is fusing dense and relation-based rankings; the precision-recall spread in their table predicts a convex combination could beat both single retrievers.","The extracted relation triples could double as human-readable evidence for the chosen citation, giving citation-prediction systems an explanation channel the current LLM prompt does not expose.","Because the evaluation is on 1,000 sampled queries, the deployment question is whether the 0.344 top-10 $F_1$ survives on the full test set under a precisely defined scoring protocol.","Relation filtering could transfer to other long-text retrieval tasks where candidates are near-duplicates, such as prior-art search or related-work generation."],"forward_implications":["If the reported $F_1$ ordering holds, relation-based top-$k$ retrieval is a stronger first-pass filter than TF-IDF or dense retrieval alone for citation discovery over highly similar abstracts.","Adding an LLM inference stage to relation-based retrieval improves recall over raw retrieval, so the two-stage design behaves as a precision-recall controller.","The approach is zero-shot, meaning new domains or paragraph types could be handled without labeled training data as long as the relation-extraction prompt transfers.","The paper's own precision-recall discussion implies that combining dense retrieval for recall with relation-based filtering for precision could outperform either strategy alone.","The reported top-10 $F_1$ advantage is small, so reproducing the comparison on the full task evaluation would test whether the edge persists beyond the 1,000-query sample."],"supporting_citations":[{"why":"Supplies the TF-IDF term-weighting baseline that the relation-based retriever is compared against.","marker":"[12]"},{"why":"Supplies the dense passage retrieval baseline that is the strongest traditional retriever in the comparison.","marker":"[5]"},{"why":"Defines the document-level relation extraction setting that motivates using relation triples as retrieval features.","marker":"[15]"},{"why":"Supports the re-ranker design pattern that the second LLM citation-selection stage follows.","marker":"[9]"},{"why":"Gives the retrieval-augmented generation background that frames combining retrieval with an LLM.","marker":"[4]"}],"fun_headline_variants":["Relation triples boost citation F1 over TF-IDF","LLM-extracted relations beat dense search for citations","Citation discovery: relation-based retrieval tops F1","Triples, not vectors, refine citation ranking","Zero-shot relation filter sharpens citation prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes the reported $F_1$ scores are computed the same way for every method, yet the paper never defines the scoring formula or the precision-recall counting protocol.","fun_headline_variants_meta":{"raw":{"variants":["Relation triples boost citation F1 over TF-IDF","LLM-extracted relations beat dense search for citations","Citation discovery: relation-based retrieval tops F1","Triples, not vectors, refine citation ranking","Zero-shot relation filter sharpens citation prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1451,"prompt_tokens":845,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":533}},"tokens_in":461,"tokens_out":606,"duration_ms":5289,"temperature":1.0,"reasoning_tokens":533,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:51:39.386956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute precision and recall on the same 1,000 queries under a written-out protocol with one predicted citation per query judged by exact abstract ID, and check whether relation-based-10 still beats dense retrieval-10; if the ordering flips or the score differs materially, the central claim would not survive.","supporting_citations":[{"cited_title":"In: Empirical Methods in Natural Language Processing (EMNLP) (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the dense passage retrieval baseline that is the strongest traditional retriever in the comparison."},{"cited_title":"In: ACL (2019)","cited_arxiv_id":null,"evidence_quote":"Defines the document-level relation extraction setting that motivates using relation triples as retrieval features."}],"review_version":1}