Pith. sign in

REVIEW 5 major objections 6 minor 16 references

Team LA at SCIDOCA shared task 2025: Citation Discovery via relation-based zero-shot retrieval

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that extracting relation triples with an LLM and using them to retrieve top-k candidates, followed by a second LLM citation selection, is an effective zero-shot approach to citation discovery.

desk verdict A shared-task system report with a useful retrieval trick, undone by an undefined F1 metric and a central claim its own tables contradict. read the letter →

arxiv 2506.18316 v1 pith:TTTJDRPB submitted 2025-06-23 cs.IR cs.CL

Trieu An , Long Nguyen , Minh Le Nguyen This is my paper · ORCID
classification cs.IRcs.CL
keywords citationdiscoveryrelationextractionzero-shotretrievallargelanguagemodelsscientificdocumenttextdenseSCIDOCA2025
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that citation discovery in long, near-duplicate scientific abstracts can be handled by first extracting structured relation triples from the query paragraph with a large language model, retrieving the top-$k$ candidate abstracts against those triples, and then letting a second LLM pass select the final citation. The claim matters because candidate sets in the task are deliberately seeded with distracting abstracts, so surface-level keyword or dense matching alone tends to trade precision against recall. On a 1,000-query sample, the relation-based top-10 retrieval reports $F_1=0.344$, above TF-IDF ($0.3297$) and dense retrieval ($0.3354$). The paper concludes that combining relation extraction with LLM-based retrieval is an effective zero-shot recipe for citation prediction.

What carries the argument

The central object is relation-based retrieval: an LLM is prompted to convert the query paragraph into structured triples or key concepts, these relational features are used to score and filter the candidate pool to the top 20, and a second LLM compares the surviving abstracts against the paragraph to output the citation ID. The mechanism does the work of concentrating the candidate set before the expensive LLM decision, assuming the extracted triples capture the semantic connection the citation is meant to support.

What would settle it

Recompute precision and recall on the same 1,000 queries under a written-out protocol with one predicted citation per query judged by exact abstract ID, and check whether relation-based-10 still beats dense retrieval-10; if the ordering flips or the score differs materially, the central claim would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that document-level relational structure, extracted as triples by an LLM, acts as a discriminative retrieval signal that separates the true citation from highly similar distractors. In the authors' experiments, relation-based retrieval at top-10 achieves recall 0.8506, precision 0.2156, and $F_1=0.344$, outperforming both TF-IDF-10 ($F_1=0.3297$) and dense retrieval-10 ($F_1=0.3354$). A second LLM inference stage raises recall to 0.4626 for the relation pipeline at lower precision 0.2125, giving $F_1=0.2912$; the authors interpret this as the LLM adding recall on top of the precision-oriented relation filter. The conclusion is that combining relation extraction with LLM-based retrieval is an effective approach for citation prediction.

Load-bearing premise

The whole comparison assumes the reported $F_1$ scores are computed the same way for every method, yet the paper never defines the scoring formula or the precision-recall counting protocol.

Editorial extensions

If this is right

  • If the reported $F_1$ ordering holds, relation-based top-$k$ retrieval is a stronger first-pass filter than TF-IDF or dense retrieval alone for citation discovery over highly similar abstracts.
  • Adding an LLM inference stage to relation-based retrieval improves recall over raw retrieval, so the two-stage design behaves as a precision-recall controller.
  • The approach is zero-shot, meaning new domains or paragraph types could be handled without labeled training data as long as the relation-extraction prompt transfers.
  • The paper's own precision-recall discussion implies that combining dense retrieval for recall with relation-based filtering for precision could outperform either strategy alone.
  • The reported top-10 $F_1$ advantage is small, so reproducing the comparison on the full task evaluation would test whether the edge persists beyond the 1,000-query sample.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors do not run is fusing dense and relation-based rankings; the precision-recall spread in their table predicts a convex combination could beat both single retrievers.
  • The extracted relation triples could double as human-readable evidence for the chosen citation, giving citation-prediction systems an explanation channel the current LLM prompt does not expose.
  • Because the evaluation is on 1,000 sampled queries, the deployment question is whether the 0.344 top-10 $F_1$ survives on the full test set under a precisely defined scoring protocol.
  • Relation filtering could transfer to other long-text retrieval tasks where candidates are near-duplicates, such as prior-art search or related-work generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper describes Team LA's system for Task 1 of the SCIDOCA 2025 shared task on citation discovery. The proposed pipeline first uses Mistral-7B-Instruct to extract relation triples from a query paragraph, retrieves the top-k candidate abstracts using those relational features, and then prompts the same or a second LLM to choose the final citation. Experiments on 1,000 randomly selected queries compare TF-IDF, dense vector retrieval, and relation-based retrieval at top-k = 10, 15, 20, plus LLM-based final selection with each retriever, reporting precision, recall, and F1. The authors conclude that integrating relation extraction with LLM-based citation retrieval is effective.

Significance. If the reported numbers were valid and properly defined, the paper would provide a modest practical baseline for citation discovery in long-abstract settings: relation-based top-10 retrieval reaches F1 0.344, exceeding TF-IDF (0.3297) and dense retrieval (0.3354), and the relation-extraction idea is easy to transfer. The authors are transparent about evaluating on 1,000 queries and about using standard baselines. However, the evaluation protocol is not defined, the central claim about LLM integration is not clearly supported by the tables as written, and the experiments are run on a training sample rather than a held-out test set. Because the reported F1 values cannot be interpreted or compared as written, the positive retrieval-stage result cannot be credited without further details.

major comments (5)
  1. [Section 4, Tables 1 and 2] The paper never defines how precision, recall, and F1 are computed, and the reported pairs cannot be reconciled with a standard evaluation. For a task with one correct citation per query, a system that emits a single prediction has equal per-query precision and recall (both are 0 or 1), yet every row of Table 2 reports different values (e.g., LLM+TF-IDF: recall 0.4079, precision 0.2347). Even for ranked top-k retrieval, if exactly one abstract per query is relevant, precision should equal recall/k; for relation-based-10 this would give 0.8506/10 = 0.0851, not the reported 0.2156. If multiple relevant citations or multi-output predictions are allowed, that protocol must be stated so that the F1 scores across Tables 1 and 2 are comparable. As written, the table values cannot be used to rank the systems.
  2. [Section 4, top-k setting; Sections 1 and 5] The manuscript does not specify the candidate-set size used for the LLM stage: Section 4 says the first retrieval step collects the top 20 documents, but Table 2 is not labeled with a top-k and the text later discusses Table 1's top-10 results as the main retrieval outcome. This ambiguity is load-bearing. If the LLM operates on top-20 candidates, LLM with relation-based retrieval (F1 0.2912) improves over relation-based-20 retrieval alone (F1 0.2711); if the comparison is against the best retrieval row, relation-based-10 (F1 0.344), the LLM stage degrades. The paper must state precisely which retrieval results the LLM is applied to and which comparison supports the effectiveness claim.
  3. [Section 4, Tables 1 and 2] Several statements in the analysis are directly contradicted by the tables or misread the table entries. The claim that LLM inference 'significantly enhances recall compared to raw retrieval scores' is not supported under any reading: all LLM recalls in Table 2 (0.4079, 0.3253, 0.4626) are far below all retrieval recalls in Table 1, including at top-20 (e.g., relation-based-20 recall 0.967). The claim that relation-based retrieval shows a 'substantial improvement in precision (e.g., 0.8506 for top-10)' mistakes recall for precision; the precision of relation-based-10 is 0.2156, only slightly higher than dense retrieval-10 (0.2095). Also, because LLM with TF-IDF (0.2980) outperforms LLM with relation-based retrieval (0.2912), the conclusion that relation-based candidate refinement is 'important' before passing candidates to the LLM is not supported by the reported best configuration.
  4. [Abstract and Section 4, evaluation sample] The abstract states that the framework is evaluated on the 'training dataset provided by the SCIDOCA 2025 organizers,' and Section 4 says experiments use '1,000 randomly selected queries from the dataset' without stating whether these are held out or part of the training set. Reporting results on a training sample does not estimate performance on unseen paragraphs, and the selection of top-k on the same sample (with k=10 reported as best) adds a selection effect. The paper should either evaluate on the official held-out/test split or clearly limit the claims to the training-sample results.
  5. [Section 3, Methods] The proposed method is described at a level that prevents reproduction and attribution of the reported gain to relation extraction. The paper does not specify how the extracted relational triples are converted into a retrieval query, which similarity function is used for relation-based retrieval, how the TF-IDF and dense retrieval baselines are implemented (e.g., the embedding model for dense retrieval), or the exact prompts used for relation extraction and final citation selection. Without these details, the reader cannot determine whether the improvement comes from the relation-based query representation or from another component of the pipeline.
minor comments (6)
  1. [Keywords] The keyword 'Text retrival' is misspelled; it should be 'Text retrieval'.
  2. [Section 2.1] The reference for Gated Transformer Retrieval (GTR) appears as a placeholder '[?]' and must be completed.
  3. [Section 2.2] There is a stray space in 'T raditional' at the start of the subsection.
  4. [Section 4, Table 2] The row label 'LLM inference with Dense retrieval0.3253' is missing a space before the number, and 'relation-base' should be 'relation-based'.
  5. [Table 1 caption] The caption 'Each method is implemented with the most relevant documents from the different top-k' is unclear; it should say 'Each method is evaluated at different top-k values.'
  6. [Section 4] The paper introduces the acronym DVR (Dense Vector Retrieval) but then uses only 'dense retrieval'; please use the notation consistently or remove the acronym.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports a standard zero-shot retrieval pipeline with independent baselines; metric inconsistencies are correctness issues, not circular derivation.

full rationale

The paper's derivation chain is a standard retrieval pipeline: extract relation triples with an LLM prompt, retrieve top-k candidate abstracts, then use a second LLM prompt to select the final citation. There is no equation in which an output is defined in terms of the target, no fitted parameter that is later renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem. The relation-extraction prompt and LLM inference are zero-shot, so the reported scores are measured on hold-out queries rather than derived from the construction of the system. The main weakness is empirical: Tables 1 and 2 appear to use different, undefined F1 protocols, and the full LLM pipeline scores below the relation-only retrieval stage, which undermines the effectiveness claim. However, this is a correctness or evaluation-validity problem, not circularity. The system's predictions are not equivalent to its inputs by construction, and no quoted reduction from the paper exhibits a circular step. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The system rests on two load-bearing domain assumptions: that LLM-extracted relations capture citation intent, and that a 1,000-query training sample represents the official test setting. No new entities are introduced. The only tuned hyperparameter reported is top-k.

free parameters (1)
  • top-k retrieved documents = 10 (best performing in Table 1; also reports 15 and 20)
    The paper reports retrieval at k=10, 15, 20 and highlights the k=10 result as best, indicating a hyperparameter selection on the evaluation sample.
assumptions (3)
  • domain assumption LLM-extracted relation triples from the query paragraph are a faithful and sufficient representation for retrieving the target citation.
    Introduced in Section 3 as the core retrieval signal; no evaluation of extraction quality or ablation without relations is provided.
  • domain assumption The 1,000 randomly selected training queries are representative of the SCIDOCA 2025 test distribution.
    Section 4 states the evaluation uses 1,000 randomly selected queries from the dataset, with no test-set results.
  • domain assumption Mistral-7B-Instruct-v0.3 provides reliable zero-shot relation extraction and citation ranking for scientific abstracts.
    Sections 3 and 4 rely on this model for both stages without validation of its outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Team LA at SCIDOCA shared task 2025: Citation Discovery via relation-based zero-shot retrieval." pith.science (2026). https://pith.science/paper/TTTJDRPB

@misc{pith2026250618316,
  author       = {Pith},
  title        = {Pith review of: Team LA at SCIDOCA shared task 2025: Citation Discovery via relation-based zero-shot retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTTJDRPB}},
  note         = {Machine review of arXiv:2506.18316}
}
read the original abstract

The Citation Discovery Shared Task focuses on predicting the correct citation from a given candidate pool for a given paragraph. The main challenges stem from the length of the abstract paragraphs and the high similarity among candidate abstracts, making it difficult to determine the exact paper to cite. To address this, we develop a system that first retrieves the top-k most similar abstracts based on extracted relational features from the given paragraph. From this subset, we leverage a Large Language Model (LLM) to accurately identify the most relevant citation. We evaluate our framework on the training dataset provided by the SCIDOCA 2025 organizers, demonstrating its effectiveness in citation prediction.

Figures

Figures reproduced from arXiv: 2506.18316 by the authors.

Figure 1
Figure 1. Overview of our system. The process begins with a query paragraph, which serves as the context for citation retrieval. Since the paragraph may contain multiple concepts and research directions, it is first processed using a relation-extracting prompt. This prompt instructs a Large Language Model (LLM) to extract structured relational triples or key concepts that capture the essential connections between entities pre… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

  1. [1]

    Transactions of the Association for Computational Linguis- tics5, 135–146 (2017)

    Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguis- tics5, 135–146 (2017)

    work page 2017

  2. [2]

    In: EMNLP (2019)

    Christopoulou, F., Miwa, M., Ananiadou, S.: Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In: EMNLP (2019)

    work page 2019

  3. [3]

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tionaltransformersforlanguageunderstanding.In:AnnualConferenceoftheNorth American Chapter of the Association for Computational Linguistics (NAACL) (2019)

    work page 2019

  4. [4]

    arXiv preprint arXiv:2312.109972(2023)

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.109972(2023)

    arXiv 2023

  5. [5]

    In: Empirical Methods in Natural Language Processing (EMNLP) (2020)

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Fan, A., Yih, W.t.: Dense passage retrieval for open-domain question answering. In: Empirical Methods in Natural Language Processing (EMNLP) (2020)

    work page 2020

  6. [6]

    In: arXiv preprint arXiv:1907.11692 (2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. In: arXiv preprint arXiv:1907.11692 (2019)

    arXiv 2019

  7. [7]

    In: International Conference on Learning Representations (ICLR) (2013)

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word represen- tations in vector space. In: International Conference on Learning Representations (ICLR) (2013)

    work page 2013

  8. [8]

    In: arXiv preprint arXiv:2108.08877 (2021)

    Ni, J., Flores, G., Constant, N., Kale, M., Al-Rfou, R., Cer, D.: Sentence-t5: Scal- able sentence encoders from pre-trained text-to-text models. In: arXiv preprint arXiv:2108.08877 (2021)

    arXiv 2021

Show all 16 references
  1. [9]

    In: arXiv preprint arXiv:1901.04085 (2019)

    Nogueira, R., Cho, K.: Passage re-ranking with bert. In: arXiv preprint arXiv:1901.04085 (2019)

    2019 arXiv

  2. [10]

    In: Empirical Methods in Natural Language Processing (EMNLP)

    Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word repre- sentation. In: Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543 (2014)

    2014

  3. [11]

    Robertson, S., Walker, S., Hancock-Beaulieu, M., Gatford, M., Payne, A.: Okapi at trec-3 (1994)

    1994

  4. [12]

    Information Processing & Management24(5), 513–523 (1988)

    Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Information Processing & Management24(5), 513–523 (1988)

    1988

  5. [13]

    Communications of the ACM18(11), 613–620 (1975)

    Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Communications of the ACM18(11), 613–620 (1975)

    1975

  6. [14]

    Wang, J., Zhang, F., Li, Y., Liu, D.: Attention-based multi-fusion method for citation prediction. In: Advances in Intelligent Information Hiding and Multimedia Signal Processing: Proceedings of the 15th International Conference on IIH-MSP in conjunction with the 12th Internat...

    2019

  7. [15]

    In: ACL (2019)

    Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Sun, M.: Docred: A large-scale document-level relation extraction dataset. In: ACL (2019)

    2019

  8. [16]

    In: Proceedings of the 2012 SIAM international conference on data mining

    Yu, X., Gu, Q., Zhou, M., Han, J.: Citation prediction in heterogeneous biblio- graphic networks. In: Proceedings of the 2012 SIAM international conference on data mining. pp. 1119–1130. SIAM (2012)

    2012

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.