Pith. sign in

REVIEW 4 major objections 5 minor 17 references

No Stupid Questions: An Analysis of Question Query Generation for Citation Recommendation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that LLM-generated questions can, in specific cases, retrieve the target citation better than extractive keyword queries, and introduces MMR-RBO to flag the questions worth trusting.

desk verdict Modest but honest empirical paper: question queries sometimes beat keyword baselines, but the untested SPECTER-2 encoder confound and weak MMR-RBO correlations keep it incremental. read the letter →

arxiv 2506.08196 v1 pith:4XRTLZ5S submitted 2025-06-09 cs.IR cs.DL

Brian D. Zimmerman , Julien Aubert-Béduchaud , Florian Boudin , Akiko Aizawa , Olga Vechtomova This is my paper
classification cs.IRcs.DL
keywords questionqueriescitationrecommendationretrievalGPT-4o-miniMMR-RBORank-BiasedOverlapSPECTER-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether questions generated by a large language model can work as search queries for finding scientific papers to cite. On average, extractive keyword queries outperform the generated questions, but the strongest question from each batch sometimes retrieves and ranks the target paper better than the keyword baseline does. To make this predictable, the authors introduce MMR-RBO, a scoring function that combines how similar a question's retrieved results are to a known-good keyword query's results with how distinct the question is from other questions in the same batch. They find a positive correlation between MMR-RBO and the rank of the target document, and they report that every question yields a unique result set, so no question is redundant.

What carries the argument

The central object is the MMR-RBO scoring function, $\mathrm{MMR\text{-}RBO}(D_i) = \lambda \cdot \mathrm{sim}_Q - (1-\lambda)\cdot \mathrm{sim}_D$, where $\mathrm{sim}_Q$ is the Rank-Biased Overlap between a question's retrieved list and the keyword baseline's retrieved list and $\mathrm{sim}_D$ is the maximum RBO between that question and any other question in the batch. Rank-Biased Overlap is a rank-weighted similarity measure for two indefinite rankings, and using it inside a Maximal Marginal Relevance combination lets the authors score each question at inference time, when the target document is unknown, by its resemblance to a query that is verifiably good. The retrieval pipeline also depends on SPECTER-2 embeddings with an ad-hoc query adapter and a FAISS index, with a MiniLM cross-encoder performing reranking.

What would settle it

Fine-tune a query encoder on question-form text paired with scientific papers and re-run the retrieval and reranking experiments: if question queries then match or beat average keyword performance, the observed gap was largely an encoder artifact; if questions still lag, the paper's characterization of question queries stands.

Watch

Extended reading notes

Core claim

GPT-4o-mini, prompted with the context around a masked citation in a related-work paragraph, can generate twenty questions whose answers would strengthen the passage, and these questions can serve as retrieval queries against an ACL Anthology corpus encoded with SPECTER-2. In about 32% of paragraphs at least one generated question retrieves the target document in the top 50, and the best question in a batch can outperform an extractive five-keyword query generated by the same model, both at retrieval and after cross-encoder reranking. The paper's proposed MMR-RBO score—a rank-weighted overlap between a question's result set and the keyword baseline's result set, minus overlap with the other questions in the batch—correlates positively with the target document's final rank; the correlation becomes significant across more batches when more questions are generated per excerpt, though it remains weak on small batches.

Load-bearing premise

The evaluation assumes that SPECTER-2's ad-hoc query encoder, which was not trained on question-form text, embeds question queries as faithfully as it embeds keyword queries, so the reported underperformance of average questions could partly be an artifact of the encoder rather than a property of the questions themselves.

Editorial extensions

If this is right

  • Question-form queries are a viable complement to keyword queries for citation recommendation, with the best question per batch sometimes outperforming the extractive baseline underneath it.
  • MMR-RBO gives a rank-aware, inference-time proxy for question quality, and the proxy's signal strengthens as the number of generated questions per excerpt grows.
  • All question queries return unique result sets, so even a question that does not top the ranking adds a distinct path to the target document.
  • Generating more questions per excerpt makes it feasible to identify questions that beat the keyword baseline, which motivates batch sizes beyond the twenty used here.
  • Reranking by a cross-encoder amplifies the difference between the worst and best question in a batch, so choosing the right question matters most after reranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the MMR-RBO correlation holds beyond this dataset, the same two-part score could pick good queries in any retrieval setting where a trustworthy baseline query exists, not just for citations.
  • Because the query encoder was not trained on question-form text, some of the gap between average questions and keyword queries may be an encoder artifact; fine-tuning an encoder on natural-language questions could shift the comparison.
  • The observation that later-generated questions score higher in an LLM-based relevance check suggests ordering effects in generation, which could be tested by shuffling the prompt or sampling order.
  • A direct next experiment would treat the generated questions as document-question answering prompts against candidate papers, linking each question to the claims it surfaces rather than to a single masked citation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using GPT-4o-mini to generate natural-language questions from masked related-work excerpts and using those questions as queries for citation recommendation over the ACL Anthology. Against a keyword baseline produced by the same model, the authors report that the best question in a batch can outperform the keyword query in some cases, both for retrieval and after cross-encoder reranking. They also introduce MMR-RBO, a variant of Maximal Marginal Relevance that uses Rank-Biased Overlap to score questions by similarity to a verifiably good query and dissimilarity to other questions in the batch, and report a positive correlation between MMR-RBO and the rank of the target document. The paper concludes that question queries are a promising, underexplored query form for scientific retrieval.

Significance. If the central finding is secured, this is a useful contribution to citation recommendation and to the broader question of how LLM-generated natural-language queries behave in ad-hoc retrieval. The experimental design is methodologically sound in one important respect: the keyword baseline is generated by the same model as the question queries, so the comparison is controlled for generation model. The authors also deserve credit for clearly stating limitations, including GROBID extraction errors and the variance of MMR-RBO on small batches. However, the current evidence is limited to a single domain (EMNLP 2024 papers), a single query encoder, and a modest correlation (global Spearman 0.29), so the strength of the claims is considerably weaker than the rhetorical framing suggests. The paper is a reasonable analysis-style contribution, but several load-bearing points need additional support before the conclusions can be accepted.

major comments (4)
  1. [Section 2.2, retrieval pipeline] All retrieval claims are mediated by the SPECTER-2 ad-hoc query adapter, which was not trained on question-form queries. The paper never checks whether question embeddings lie in the same region of the embedding space as the keyword and context queries the adapter was designed for. If question queries are systematically off-distribution, then (a) the average underperformance of questions could be an encoding artifact rather than a property of questions, (b) the few questions that outperform the keyword baseline could be precisely those whose embeddings happen to fall in-distribution, and (c) the MMR-RBO correlations could reflect distributional similarity of result sets rather than query utility. This is load-bearing for the central interpretation. Please add a validation step, for example: measure the distance between question and keyword embeddings on the same paragraphs, compare retrieval with an encoder that has been fine-tuned on question-style queries, or evaluate the same questions after reformulating them as declarative statements to see whether retrieval performance is stable under paraphrase.
  2. [Section 3.1, Table 2 and Figure 4] The global Spearman correlation of 0.29 is reported without a confidence interval or a p-value, and the table shows several batch-size/lambda combinations with near-zero or negative correlations (e.g., batch 5, lambda 0.0 gives -0.1182; batch 20, lambda 0.0 gives -0.1059). The statement that there is a 'positive monotonic relationship between MRR and MMR-RBO' is therefore not yet established for the operational regime. Please report per-batch correlation distributions with confidence intervals, apply a proper significance test at the batch level (accounting for multiple comparisons), and state clearly for which lambda and batch sizes the relationship holds. The paper's own limitation paragraph notes that MMR-RBO is 'highly susceptible to variance in small datasets,' which further undercuts the strength of the current claim.
  3. [Figures 1 and 3] The central claim that 'in some cases' generated questions outperform the keyword baseline is supported by mean values and min/max whiskers, but no paired significance test is reported between the best question in each batch and the keyword query. Because the best question is selected as the maximum over 20 generated questions, the comparison is subject to a strong selection effect; without a paired test or an adjustment for multiple comparisons, it is unclear whether the observed advantage is beyond what chance would produce. Please add paired tests (e.g., Wilcoxon signed-rank) for hit ratio and MRR, and report the proportion of paragraphs in which the best question significantly beats the keyword baseline.
  4. [Equation (2)] Equation (2), which defines RBO, is not syntactically well-formed as printed. The expression 'RBO(S, T, p, k) = Xk k ·p k + 1−p p kX d=1 Xd d ·p d' appears to have missing summation limits and misplaced terms, making the method impossible to reproduce from the text. Please correct the formula to the standard Rank-Biased Overlap definition, with explicit summation bounds and the p^{d-1} weighting, and ensure it matches the software implementation.
minor comments (5)
  1. [Section 4 / Abstract] The concluding statement 'there are no stupid questions' is a rhetorical flourish rather than an empirical result; the data only show that each question yields a unique result set, which does not imply that every question is useful for retrieval. Consider softening this to a statement about the value of exploring diverse query formulations.
  2. [Figure 4] The color scheme (blue for increased rank, red for decreased rank) is not self-explanatory; please add a legend or caption text that explicitly states what the colors encode.
  3. [Section 2.3 / Eq. (3)-(5)] The notation sim_Q and sim_D is used before Eq. (5) without an explicit definition of the set S over which the maximum in sim_D is taken; please clarify whether the maximum is over all other questions in the same batch or over the top-ranked questions.
  4. [Appendix A] The LLM-based relevance scoring in Appendix A is evaluated on only 20 questions and no reliability statistics are reported; this is appropriate as a preliminary analysis, but the text should label it as such and avoid over-interpreting the increasing trend in Figure 5.
  5. [Section 1] The reference to 'Figure 1' in the introduction appears before Figure 1 is introduced; consider moving the reference to the first mention in Section 2 or 3, or reordering the figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's comparisons and correlations are measured, not fitted, and no load-bearing claim reduces to its own inputs by construction.

full rationale

The paper's derivation chain is self-contained. Both the generated questions and the extractive keyword baseline are produced by the same GPT-4o-mini model from the same masked excerpts, so the 'some questions beat keywords' claim is an empirical comparison rather than an identity. MMR-RBO is a defined scoring function built from RBO over retrieval result sets, and the paper reports a measured correlation between MMR-RBO and MRR/rank; no parameter is fitted to the target ranks, and the authors explicitly acknowledge the variance and sensitivity of the correlation in small batches. The retrieval and reranking pipelines use SPECTER-2 and a MiniLM cross-encoder, which are external tools cited as prior work, not results derived from the paper's own claims. The concern that SPECTER-2 may encode question-form queries off-distribution is a legitimate external-validity or correctness risk, but it is not a circular reduction: the paper does not define question quality in terms of SPECTER-2 scores, and the main experimental claims would not be true by construction even if the encoder were perfectly calibrated. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling step appears. The limitations section openly states the main threats to robustness, which further supports that the work is an empirical analysis rather than a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities; it relies on standard IR models (SPECTER-2, MiniLM cross-encoder, FAISS) and a new scoring function MMR-RBO.

free parameters (3)
  • MMR-RBO lambda = swept over 0, 0.5, 1
    Chosen by hand to weight similarity to a verifiably good query against diversity from other questions; results are sensitive to this value.
  • Number of generated questions per batch = 20
    Design choice; the paper tests 5, 10, and 20 and reports that correlation stability improves with more questions.
  • Retrieval top-k for result sets = 50
    Used to compute MRR, RBO, and MMR-RBO; no sensitivity analysis reported.
assumptions (4)
  • domain assumption ACL Anthology corpus is a valid retrieval corpus for citation recommendation evaluation.
    The search space is restricted to ACL Anthology, which may not represent the general scientific literature that citation recommendation targets.
  • domain assumption The masked-citation paragraph with exactly one target is a valid operationalization of the citation recommendation task.
    Real citation recommendation is exploratory and has no single ground truth; this simplification may not capture real-world utility.
  • domain assumption SPECTER-2's ad-hoc query encoder adequately represents question-form queries.
    The encoder was not trained for question queries, and no analysis shows that question embeddings are well-calibrated relative to keyword embeddings.
  • domain assumption MS MARCO-trained cross-encoder generalizes to scientific citation relevance.
    The reranker is used without domain adaptation; its relevance judgments may differ from citation intent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Stupid Questions: An Analysis of Question Query Generation for Citation Recommendation." pith.science (2026). https://pith.science/paper/4XRTLZ5S

@misc{pith2026250608196,
  author       = {Pith},
  title        = {Pith review of: No Stupid Questions: An Analysis of Question Query Generation for Citation Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XRTLZ5S}},
  note         = {Machine review of arXiv:2506.08196}
}
read the original abstract

Existing techniques for citation recommendation are constrained by their adherence to article contents and metadata. We leverage GPT-4o-mini's latent expertise as an inquisitive assistant by instructing it to ask questions which, when answered, could expose new insights about an excerpt from a scientific article. We evaluate the utility of these questions as retrieval queries, measuring their effectiveness in retrieving and ranking masked target documents. In some cases, generated questions ended up being better queries than extractive keyword queries generated by the same model. We additionally propose MMR-RBO, a variation of Maximal Marginal Relevance (MMR) using Rank-Biased Overlap (RBO) to identify which questions will perform competitively with the keyword baseline. As all question queries yield unique result sets, we contend that there are no stupid questions.

Figures

Figures reproduced from arXiv: 2506.08196 by the authors.

Figure 1
Figure 1. Context, Keyword (extractive), and the best [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We compare context, keywords, and questions on their utility as retrieval queries. Questions are sorted by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Difference in MRR by reranker query over the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The observed correlation between MMR-RBO [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Although questions are scored by Llama indi [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal

    Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal. 2025. https://arxiv.org/abs/2402.01788 Litllm: A toolkit for scientific literature review . Preprint, arXiv:2402.01788

    arXiv 2025

  2. [2]

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Litsearch: A retrieval benchmark for scientific literature search. arXiv preprint arXiv:2407.18940

    arXiv 2024

  3. [3]

    Juan Pablo Bascur, Suzan Verberne, Nees Jan van Eck, and Ludo Waltman. 2023. Academic information retrieval using citation clusters: in-depth evaluation based on systematic reviews. Scientometrics, 128(5):2895--2921

    work page 2023

  4. [4]

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. https://arxiv.org/abs/2308.13418 Nougat: Neural optical understanding for academic documents . Preprint, arXiv:2308.13418

    arXiv 2023

  5. [5]

    Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335--336

    1998

  6. [6]

    Jingshan Huang and Ming Tan. 2023. The role of chatgpt in scientific communication: writing better scientific review articles. American journal of cancer research, 13(4):1148

    work page 2023

  7. [7]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535--547

    2019

  8. [8]

    Patrice Lopez. 2009. Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International conference on theory and practice of digital libraries, pages 473--474. Springer

    work page 2009

Show all 17 references
  1. [9]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292

    2024 arXiv

  2. [10]

    Shutian Ma, Chengzhi Zhang, and Xiaozhong Liu. 2020. A review of citation recommendation: from textual content to enriched context. Scientometrics, 122(3):1445--1472

    2020

  3. [11]

    Roberto Mart \' nez-Cruz, Alvaro J L \'o pez-L \'o pez, and Jos \'e Portela. 2025. Chatgpt vs state-of-the-art models: a benchmarking study in keyphrase generation task. Applied Intelligence, 55(1):50

    2025

  4. [12]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  5. [13]

    Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022. https://api.semanticscholar.org/CorpusID:254018137 Scirepeval: A multi-format benchmark for scientific document representations . In Conference on Empirical Methods in Natural Language Processing

    2022

  6. [14]

    Alex D. Wade. 2022. https://doi.org/10.1145/3487553.3527147 The semantic scholar academic graph (s2ag) . In Companion Proceedings of the Web Conference 2022, WWW '22, page 739, New York, NY, USA. Association for Computing Machinery

    2022

  7. [15]

    William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1--38

    2010

  8. [16]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  9. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.