Pith. sign in

REVIEW 3 major objections 5 minor 11 references

Question-to-Question Retrieval for Hallucination-Free Knowledge Access: An Approach for Wikipedia and Wikidata Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that indexing each Wikipedia paragraph and Wikidata triple with LLM-generated questions, then matching user queries to those questions by cosine similarity, retrieves the correct source with scores above 0.9 and makes…

desk verdict Repackaged document expansion with no evaluation; a coherent proposal, not a demonstrated result. read the letter →

arxiv 2501.11301 v3 pith:KR652SLP submitted 2025-01-20 cs.CL cs.AI

Santhosh Thottingal This is my paper · ORCID
classification cs.CLcs.AI
keywords question-to-questionretrievalhallucination-freeQAdensevectorWikipediaquestionansweringWikidatagenerationcosinesimilaritymatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a retrieval-only design for open-domain question answering. Instead of embedding passages and comparing a user's question to them, an instruction-tuned LLM first generates a set of plausible questions for every Wikipedia paragraph and every Wikidata statement; those questions are embedded into a vector store, mapped back to the original content by hash. A user query is embedded in the same space, and the highest cosine similarity selects the generated question whose corresponding paragraph or triple is returned. Because both sides are questions, similarity is high—above 0.9 for relevant pairs—so the top match points reliably to the content that answers the query. Presenting the retrieved original text directly eliminates the answer-generation step and, with it, the risk of hallucinated or fabricated responses.

What carries the argument

The mechanism that carries the argument is the generated-question index. For each content unit—a paragraph with its article and section context, or a Wikidata triple rendered as text—an instruction-tuned LLM produces a set of natural-language questions the unit can answer; these are embedded with a text-embedding model, stored in a vector store, and linked to the source through a SHA-256 hash of the unit. At query time, the user query is embedded and the argmax over cosine similarity selects the closest generated question, whose hash retrieves the original unit. The design's force comes from making the indexed objects questions rather than declarative passages, so query and index entry share the same interrogative form.

What would settle it

Run a held-out evaluation where real user queries (for example from Wikipedia search logs) are matched against the question index, with annotators judging whether the retrieved paragraph answers the query; if a large share of answerable queries fail to retrieve the correct content, the coverage assumption—and with it the claim of consistently high precision—breaks.

Watch

Extended reading notes

Core claim

The central claim is that question-to-question retrieval—indexing the knowledge base with generated questions rather than the content itself—achieves both high retrieval precision and hallucination-free responses. The paper reports cosine similarity consistently above 0.9 for relevant question pairs, against the 0.4–0.7 range typical of question-to-passage matching, and argues that directly returning the retrieved source text makes answer generation unnecessary. It further claims that the same approach works for Wikidata by converting each triple to a text form and generating questions from it, and that through triple metadata it makes images, audio, and 3D models addressable as answers.

Load-bearing premise

The method assumes that the generated question set for each paragraph or triple is complete enough that every user question answerable by that content will match one of its generated questions more closely than any content generated from elsewhere.

Editorial extensions

If this is right

  • Because inference-time generation is replaced by a vector lookup and a hash dereference, query latency and cost fall well below generation-based RAG systems.
  • The answer shown to the user is always the original Wikipedia text or Wikidata statement, so the system cannot fabricate content that did not come from the knowledge base.
  • Selective re-indexing via content hashes means Wikipedia's constant editing can be accommodated by regenerating questions only for changed paragraphs.
  • The approach treats textual facts and metadata for images, audio, and 3D models uniformly, so multimodal questions like 'show me the flag' or 'how does a lion roar' resolve to the same retrieval pipeline.
  • The roughly tenfold expansion of the index from question generation still fits within modern vector database capacities, keeping the approach scalable to full Wikipedia.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure generation coverage; the decisive extension would be a held-out test using real search queries to quantify how many user phrasings fall outside the generated question sets.
  • A second inference: because the method retrieves the source unit itself, a minimum-similarity threshold could act as a built-in refusal mechanism, letting the system say 'I don't know' instead of guessing—a property current RAG systems typically lack.
  • Third, the same index design would likely transfer to non-encyclopedic corpora such as legal or technical documentation, where the mismatch between interrogative queries and declarative text is equally acute.
  • Fourth, multi-hop questions, which the paper explicitly leaves out, could be addressed by generating questions over chained content units (adjacent paragraphs or linked Wikidata triples), trading index size for reasoning scope.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a question-to-question (Q2Q) retrieval architecture for factoid question answering over Wikipedia and Wikidata. For each logical content unit (a paragraph or a textualized Wikidata triple), an instruction-tuned LLM generates a set of natural-language questions; these questions are embedded with a dense encoder and stored in a vector index. At inference time, a user query is embedded and matched against the question index via cosine similarity; the argmax match is used to retrieve the original content through a hash mapping. The paper claims that this approach achieves cosine similarity scores consistently above 0.9, high-precision retrieval, and hallucination-free responses, because no answer-generation step is used. The experiments reported are limited to hand-selected similarity examples in Tables 1-3 and a prototype indexing fewer than 1,000 Wikipedia articles; no retrieval precision, recall, accuracy, or comparison with existing systems is reported.

Significance. The core idea is interesting and potentially useful: converting declarative passages into a question space may reduce the well-known question-passage embedding mismatch in dense retrieval, and the method eliminates the cost and hallucination risk of answer generation at inference time. The paper also correctly identifies the relevant prior work, including document expansion by query prediction (Nogueira et al.) and unified structured/unstructured QA (UniK-QA). If the method were shown to work at scale on standard QA benchmarks, it could be a practical contribution to open-domain factoid QA. However, the paper as written provides only anecdotal evidence; the central claims are not backed by a systematic evaluation, and the load-bearing assumption that the generated question set covers all plausible user queries is merely asserted. The authors do clearly state the extractive scope and the limitation concerning question-generation coverage, which is helpful transparency but does not replace the missing measurements.

major comments (3)
  1. [Section 5 and Table 2] The central empirical claim, 'cosine similarity scores consistently above 0.9 for relevant question pairs,' is supported only by a handful of hand-picked examples. Several of these queries are near-verbatim paraphrases of the generated questions (e.g., 'France nuclear energy percentage?' vs. 'What percentage of France's electricity is nuclear?'), so the high scores are largely by construction. The paper itself defines the proper metric, top-k retrieval accuracy, in Section 2, but never reports it. A systematic evaluation is needed, e.g., using a standard QA benchmark such as Natural Questions or a random sample of natural user queries, with retrieval precision, recall, and end-to-end answer accuracy.
  2. [Section 4.3 and Section 3] The load-bearing assumption is that the LLM-generated question set is comprehensive enough that any user query answerable by a passage will be nearest to a generated question that maps to that passage. The Limitations section concedes 'it might be possible to improve question generation coverage,' but no coverage measurement is provided and no analysis of failure modes is given. If a user query has no generated question in its vicinity, the argmax selection silently chooses an unrelated question and returns the wrong passage with high confidence. The paper should estimate coverage, for example by measuring retrieval accuracy on held-out queries that were not used in generation or on natural user queries, and reporting recall at various k.
  3. [Section 4.2 and Abstract] The term 'hallucination-free' overstates the safety of the system. Removing the LLM generation step eliminates generation-based hallucination, but it does not eliminate retrieval error: a wrong argmax match will present incorrect content as the answer. Since retrieval accuracy is never measured, the claim that the system is 'hallucination-free' in any practical sense is not supported. The paper should revise the claim to 'no generated answer' or 'no generation hallucination,' and evaluate the system's accuracy on answer-bearing queries, rather than asserting that retrieval error cannot occur. This is not a matter of wording only; it changes the meaning of the central contribution.
minor comments (5)
  1. [Throughout] Unresolved cross-references to 'Appendix ??' appear in Section 3; the prompts are actually in Appendices A.1 and A.2. Also, Listing 1 is not referenced in the text as Appendix B. All cross-references should be fixed before resubmission.
  2. [Table 1 and Section 4.1] There is an inconsistency in the embedding models: Table 1 reports 'text-embedding-004. Dimensions: 798,' while Section 4.1 states the prototype used baai/bge-small-en-v1.5 with 384-dimensional vectors. The authors should clarify which model was used for which experiment and whether the dimensionality value 798 is a typo (e.g., 768).
  3. [Section 2] The notation is inconsistent: the text first defines D documents and then uses M for both the number of passages and the number of content units. The formal definitions would be easier to follow if the symbols were unified and defined once.
  4. [Section 4.1] The claim that the vector store size is 'approximately ten-fold' larger than passage-based indexing is given without empirical justification. Since the number of generated questions per passage is not reported, the reader cannot verify this ratio. Reporting the average number of questions per passage and the actual index size would make this quantitative claim checkable.
  5. [Figure 2] The screenshot in Figure 2 is described only in its caption; the text would benefit from a sentence explaining the user interaction (query input, article navigation, scrolling, and highlighting) so the reader knows how the prototype behaves.

Circularity Check

1 steps flagged · score 5.0 of 10

Demonstration of high precision is self-selected: reported >0.9 scores are paraphrase similarities to LLM-generated questions, not retrieval accuracy; no top-k metric is reported.

  1. renaming known result [Abstract; Section 3 (after Table 2), Table 2 caption]
    "Our method achieves high cosine similarity ( > 0.9 ) for relevant question pairs, enabling highly precise retrieval. ... As Generated questions are mapped to a passage that an answer it, the effective retrieval similarity for all user queries are same as the Similarity Score."

    The reported 'prediction' of retrieval precision is the cosine similarity between a user query and a question the LLM generated from the indexed passage. The queries in Table 2 and Appendix C are near-verbatim paraphrases of those generated questions (e.g., 'France nuclear energy percentage?' vs. 'What percentage of France's electricity is nuclear?'; 'India's capital city' vs. 'What is the capital of India?'). For embedding models, paraphrase pairs are known to be close, so the >0.9 scores are forced by the construction of the examples, not by any measurement of whether the retrieved passage answers the query.

full rationale

The method itself is a legitimate retrieval architecture and not circular: indexing LLM-generated questions and matching user queries to them is a coherent design, and no self-citation chain is involved. The circularity is confined to the demonstration: the only quantitative support for 'high precision' is hand-picked query-question pairs that are paraphrases of the indexed questions, so the reported scores reflect the known embedding property of paraphrase proximity, not retrieval accuracy. No external benchmark, no top-k recall, and no coverage test are reported; the Limitations concede that question generation coverage could be improved, which is exactly the assumption on which the method's recall depends. The 'hallucination-free' claim is definitional for generation (no LLM answer generation) but does not account for retrieval errors, which remain unmeasured. Overall, the central 'high precision' evidence reduces to a renamed paraphrase-similarity result, warranting a partial circularity score of 5 rather than a higher score, since the architecture itself is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the coverage and quality of LLM-generated questions and on the reliability of cosine similarity in question space. Neither is validated with data. No fitted parameters or new entities are introduced; the system is a straightforward pipeline.

assumptions (4)
  • domain assumption For each content unit, the LLM generates a comprehensive set of questions covering all plausible user queries for that unit.
    Stated in Section 3: 'we employ an instruction-tuned LLM to generate a comprehensive set of questions for each logical content unit.' System recall depends on this coverage; no coverage measurement is provided, and the Limitations section admits coverage may be insufficient for complex questions.
  • domain assumption Cosine similarity between user-query embedding and generated-question embedding above 0.9 reliably indicates the corresponding passage answers the query.
    The precision claim rests on this; the paper itself cites Steck et al. (2024) warning that cosine similarity can be arbitrary, yet no calibration or error analysis is offered.
  • domain assumption Factoid questions are extractive, with the answer contained within a single paragraph or triple.
    Stated in Background: 'We assume the question is extractive, in which the answer is available within such a paragraph.' This restricts scope but is explicit.
  • domain assumption LLM question generation quality is sufficient across Wikipedia topics and languages.
    The paper uses llama-3.1-8b-instruct-awq for generation and notes the limitation that LLMs are efficient only in a small set of languages. Generation errors would propagate to retrieval failures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Question-to-Question Retrieval for Hallucination-Free Knowledge Access: An Approach for Wikipedia and Wikidata Question Answering." pith.science (2026). https://pith.science/paper/KR652SLP

@misc{pith2026250111301,
  author       = {Pith},
  title        = {Pith review of: Question-to-Question Retrieval for Hallucination-Free Knowledge Access: An Approach for Wikipedia and Wikidata Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KR652SLP}},
  note         = {Machine review of arXiv:2501.11301}
}
read the original abstract

This paper introduces an approach to question answering over knowledge bases like Wikipedia and Wikidata by performing "question-to-question" matching and retrieval from a dense vector embedding store. Instead of embedding document content, we generate a comprehensive set of questions for each logical content unit using an instruction-tuned LLM. These questions are vector-embedded and stored, mapping to the corresponding content. Vector embedding of user queries are then matched against this question vector store. The highest similarity score leads to direct retrieval of the associated article content, eliminating the need for answer generation. Our method achieves high cosine similarity ( > 0.9 ) for relevant question pairs, enabling highly precise retrieval. This approach offers several advantages including computational efficiency, rapid response times, and increased scalability. We demonstrate its effectiveness on Wikipedia and Wikidata, including multimedia content through structured fact retrieval from Wikidata, opening up new pathways for multimodal question answering.

Figures

Figures reproduced from arXiv: 2501.11301 by the authors.

Figure 1
Figure 1. Overall Architecture of Question-Question retrieval and question answering [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Screenshot from a prototype showing the answer presentation using our approach for the question "length [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 4 linked inside Pith

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. https://doi.org/10.18653/v1/P17-1171 Reading W ikipedia to answer open-domain questions . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870--1879, Vancouver, Canada. Association for Computational Linguistics

    doi:10.18653/v1/p17-1171 2017

  4. [4]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    arXiv 2024

  5. [5]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769--6781, Online. Ass...

    doi:10.18653/v1/2020.emnlp-main.550 2020

  6. [6]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-intensive nlp tasks . In Proceedings of the 34th International Conference o...

    arXiv 2020

  7. [7]

    Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, and Monica S

    Shicheng Liu, Sina J. Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, and Monica S. Lam. 2024. https://arxiv.org/abs/2407.11417 Spinach: Sparql-based information navigation for challenging real-world questions . Preprint, arXiv:2407.11417

    arXiv 2024

  8. [8]

    Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. https://arxiv.org/abs/1904.08375 Document expansion by query prediction . Preprint, arXiv:1904.08375

    arXiv 2019

Show all 11 references
  1. [9]

    Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.115 U ni K - QA : Unified representations of structured and unstructured knowl...

    2022 doi

  2. [10]

    Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. 2024. https://doi.org/10.1145/3589335.3651526 Is cosine-similarity of embeddings really about similarity? In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 887–890. ACM

    2024

  3. [11]

    Denny Vrande c i\' c and Markus Kr\" o tzsch. 2014. https://doi.org/10.1145/2629489 Wikidata: a free collaborative knowledgebase . Commun. ACM, 57(10):78–85

    2014 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.