REVIEW 3 major objections 5 minor
Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a fully synthetic pipeline can generate realistic conversational QA dialogs and annotations from plain documents, and that fine-tuning light rewriters or retrievers on this synthetic data improves retrieval and…
desk verdict A useful pipeline with public artifacts, but the synthetic annotation loop is self-confirming and the real-world transfer gains, while positive, are smaller and less statistically grounded than the prose claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the proposition-first pipeline. A 'proposition' is defined as a standalone simple sentence that conveys information from a document which a user might actually ask about; Step 1 has an LLM split compound sentences, remove anaphora, and discard content no one would query. Step 2 then samples sublists of these propositions and prompts the same LLM to generate a dialog, first writing self-contained (decontextualized) user questions, then re-introducing context to create the contextualized versions, and finally asking the model to annotate which propositions ground each question-answer pair, marking pairs 'accepted' or 'not_accepted'. The accepted pairs become the training signal, and BM25 is used to replace the regenerated ground-truth propositions with their closest Step-1 originals, preventing vocabulary drift. This machinery matters because it removes the one-sentence-to-one-question assumption of earlier dialog-inpainting methods and lets retrieved propositions, rather than whole documents, carry the answer.
What would settle it
Run the full pipeline on a held-out collection of plain documents from a fresh domain for which a small human-annotated OR-CONVQA test set exists, then measure retrieval MAP using the synthetic-trained rewriter against the raw contextualized questions as baseline; if the rewriter's MAP is not clearly above the baseline in repeated seeded runs, the transfer assumption is falsified. A cheaper check is to repeat the DOC2DIAL and MultiDoc2Dial real-world experiments while swapping the Dialog-LLM for a different model and seeing whether the synthetic-trained models still beat using the raw contextualized question.
Extended reading notes
Core claim
The central claim is that the quality of synthetic dialog generation depends less on the LLM's ability to write questions than on the representation of the source material: documents first distilled into propositions—short, self-contained sentences carrying information a user would plausibly ask about—yield dialogs whose retrieval performance is substantially higher than dialogs generated from raw document sentences. On top of that, the self-annotated decontextualized questions and ground-truth propositions in the synthetic dialogs provide enough supervision to fine-tune lightweight models: a small question rewriter and a small retriever trained only on synthetic data outperform using the raw contextualized user question by a clear margin on DOC2DIAL and MultiDoc2Dial, and the retriever's performance is comparable to prompting a larger LLM for rewrite generation. The paper also shows that fine-tuning a small retriever on the synthetic data is at least as effective as question rewriting, and that a joint 'rewrite/no_rewrite' classifier can halve average rewriter latency without hurting accuracy.
Load-bearing premise
The whole training signal rests on the assumption that the LLM's self-generated dialogs and self-annotated grounding propositions approximate real user questions and document grounding closely enough that models trained on them transfer to genuine human dialogs.
Editorial extensions
If this is right
- Organizations with only plain text documents, such as product documentation or policy manuals, can build domain-specific conversational QA systems without manually annotated dialogs.
- Lightweight question rewriters trained on synthetic data can replace expensive LLM prompting for decontextualization, cutting inference cost while preserving retrieval quality.
- Existing dialog-unaware retrievers can be reused as-is, because the rewriter turns each user turn into a self-contained query.
- A small retriever fine-tuned on synthetic data can match or exceed the usefulness of gold decontextualized questions, contradicting the common assumption that retriever fine-tuning needs large amounts of human-annotated data.
- The 'no_rewrite' decision mechanism roughly halves rewriter latency with no measured performance drop, which matters for real-time assistants.
Reading between the lines
- Because the pipeline needs only plain text, the same prompts could be run through a multilingual LLM to generate OR-CONVQA dialogs for low-resource languages, an extension the authors name but do not test; the main risk is that proposition quality degrades in lower-resource language models.
- If the synthetic dialogs were scaled up, for example by resampling propositions or using multiple LLM seeds, the authors' own preliminary result that small response generators underperform prompted LLMs might reverse, since they attribute that gap to insufficient synthetic data.
- The proposition granularity likely interacts with the retriever: the largest gains appear when propositions are short enough for a sparse-dense fusion to match them exactly, so domains with longer or more technical passages may need adjusted proposition lengths.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-step LLM-prompting pipeline that converts plain documents into propositions and then generates OR-CONVQA dialogs containing contextualized and decontextualized user questions, system responses, and grounding propositions. The authors use the resulting synthetic dialogs to fine-tune lightweight question rewriters (Mamba, GPT-2, T5) and a MiniLM retriever, and evaluate them on synthetic test dialogs and on the human-annotated DOC2DIAL and MultiDoc2Dial test sets. The central claims are that proposition-based dialogs are higher quality than sentence-chunk-based dialogs, and that models trained only on synthetic data improve real-world retrieval and response generation relative to using the original conversational questions. The paper also introduces a conditional rewriting mechanism that reduces inference latency and releases source code and a synthetic dataset.
Significance. If the central claims hold, the pipeline would be a practical answer to the scarcity of domain-specific OR-CONVQA training data: it needs only plain documents, avoids the one-to-one sentence/question mapping assumption, and produces training signal for lightweight rewriters and retrievers. The public release of code and a synthetic dataset is a concrete strength, as is the use of several complementary automatic metrics (MAP, R@k, SBLEU, METEOR, BERTScore, perplexity) and the comparison against a strong prompted-LLM rewriter. However, the evidence is currently weakened by self-annotated grounding labels, the absence of statistical significance testing, and a transfer evaluation that uses the same document collection for both synthetic training and real-world testing. These issues affect the two main claims, so the contribution is promising but not yet fully demonstrated.
major comments (3)
- [§3.2, Prompt 2.3; Tables 2–4] The supervision and the synthetic-retrieval ground truth are produced by the same Dialog-LLM that generates the dialogs. Prompt 2.3 explicitly instructs the model to “act as a human annotator,” to select the propositions each QA pair is grounded in, and to always accept the first and last pairs. Because the answers, the propositions, and the grounding labels all come from the same model, the labels may reflect generator consistency (for example, lexical overlap between an answer and the propositions it was generated from) rather than the document information a real user would need. These labels are load-bearing: they are used both as the evaluation targets in Tables 2–4 and as the training signal for the T5 rewriter and the MiniLM retriever, so the measured gains on synthetic data, and to some extent the transfer results in Table 5, inherit this bias. Section 4.8 reports a manual check of the Step 1 propositions but no independent human check of the Prompt 2.3 grounding labels. The paper should add a human annotation study on a sample of the Prompt 2.3 outputs, or otherwise validate the grounding labels against independent annotations, before the synthetic-retrieval numbers can be read as evidence of dialog quality.
- [§4.1, §4.5, Tables 5–6] The paper reports three-seed averages for the synthetic test sets but no standard deviations, confidence intervals, or significance tests, and the real-world DOC2DIAL/MultiDoc2Dial experiments are run once. The headline retrieval gains are modest in absolute terms (for example, T5 MAP 0.21 vs. 0.17 for Queryco in Table 5), and the response-generation improvements are uneven (T5 SBLEU 6.54 vs. Claude 6.52 on MultiDoc2Dial in Table 6), with only MiniLM showing a larger gain (8.89 vs. 6.16). Without variance estimates or paired significance tests, the claim that fine-tuning on synthetic data “substantially improves” real-world performance is not established at the reported precision. Please report per-seed results, standard deviations, and appropriate significance tests for the synthetic experiments, and at least bootstrap confidence intervals or paired tests for the real-world comparisons.
- [§3.1, §4.5] The real-world transfer evaluation uses the same 488 DOC2DIAL/MultiDoc2Dial documents both to generate the synthetic training dialogs and as the proposition repository for the human test dialogs. This setup tests transfer from synthetic to human dialogs within a single document collection, but it does not test the stated goal of applying the pipeline to a genuinely new application domain with different documents. The proprietary-document experiments are evaluated only on synthetic test dialogs, so no real-world test exists for a held-out document set. To support the “new application domain” claim, the authors should either hold out a document collection for real-world evaluation or explicitly soften the claim to same-corpus transfer.
minor comments (5)
- [§4.2, footnote] The stated BM25 parameters k1 = 0.05 and b = 5 are unusual: b is normally restricted to [0, 1] and k1 is typically near 1.2. Please clarify whether this is a typo and report the actual values used, since retrieval results can be sensitive to these settings.
- [§4.7] The conditional rewriting section states that “we find no difference in performance” without reporting the corresponding retrieval numbers in a table or in the text; please provide the quantitative comparison alongside the latency reduction.
- [§4.8] The manual evaluation discussion reports qualitative findings (“little to no hallucinated information,” “cases of propositions that convey no information”) without counts, example frequencies, or agreement statistics; please report the size of the sample and the proportion of cases in each category.
- [References] The reference list contains Lin et al. 2020a and Lin et al. 2020b with the same title and venue; please consolidate the duplicate or clarify the distinction between the two entries.
- [Abstract and §4.6] The abstract says that “the retrieved information and the decontextualized question” are passed to the response LLM, but for the fine-tuned MiniLM retriever the paper instead passes the dialog history along with the last user question; please make the abstract consistent with the experimental setup.
Circularity Check
Synthetic retrieval evidence is self-confirming: the same Dialog-LLM generates the dialogs and then labels their ground-truth propositions (Prompt 2.3), so Tables 1-3 partly measure generator consistency; the real-world DOC2DIAL/MultiDoc2Dial transfer results remain non-circular external evidence.
-
self definitional
[Section 3.2, Step 2, Prompt 2.3; Appendix B.2; used in Section 4.2, Tables 1-3]
"Prompt 2.3 (ground truth propositions): The third prompt of Step 2 feeds the Dialog-LLM with each sublist of propositions and the corresponding generated dialog, and instructs it to identify (generate again) the propositions each question-answer pair is grounded in. ... I need you to act as a human annotator ... Note which propositions are used in each answer. ... The first and last pairs should always be accepted."
Prompt 2.1 already instructs the same LLM that 'the user asks a question based on a given proposition' drawn from the sampled sublist, so the propositions used in an answer are the model's own generation input. Prompt 2.3 then has that same model 'act as a human annotator' and re-select those propositions as ground truth. Consequently, the synthetic relevance and retrieval evaluations (QRELSCORE, Tables 1-3) score retrievers against labels produced by the generator itself, so high scores partly measure self-consistency and lexical overlap rather than independent grounding. The forced acceptance of the first and last pairs further shows the 'accepted/not_accepted' filter is not an independent validity check.
-
other
[Section 4.2, Table 2 and surrounding text]
"We use RRF to retrieve either propositions or sentences ... Table 2 shows that proposition-generated dialogs lead to substantially higher retrieval performance, compared to sentence-generated dialogs ... We consider the superior retrieval performance of proposition-generated dialogs as an indication of higher-quality synthetic data, since ground truth decontextualized questions should lead to high retrieval scores."
The Table 2 comparison is the paper's main evidence that proposition-based dialogs are higher quality than sentence-based ones. Both the 'ground truth decontextualized questions' (Queryde) and the ground-truth propositions come from the same Dialog-LLM (Prompts 2.1 and 2.3), so the retrieval gap can reflect how well the proposition pool matches the generator's own style rather than how well the dialogs serve real user information needs. This makes the synthetic-data superiority claim partially circular; however, the real-world DOC2DIAL and MultiDoc2Dial results in Tables 5-6 use external human-annotated data and provide non-circular support for the pipeline's usefulness.
full rationale
The paper has one genuine circular step: the same Dialog-LLM that generates each user question and system response is later asked, in Prompt 2.3, to 'act as a human annotator' and select the propositions that ground each pair. Because Prompt 2.1 generated the questions from a known proposition sublist, the resulting ground-truth propositions are the generator's own input re-identified by itself, not independent annotations. Retrieval and relevance metrics computed against those labels (Tables 1-3, QRELSCORE) therefore partly measure generator self-consistency, and the instruction that the first and last pairs 'should always be accepted' further undermines the filter. This circularity is partial rather than total: the central transfer claim is tested on the real-world DOC2DIAL and MultiDoc2Dial test sets, which are external human dialogs, and the fine-tuned T5 rewriter and MiniLM retriever improve over baseline there (Table 5), so the main pipeline result does not reduce to the self-annotated labels. No load-bearing self-citation chain or imported uniqueness theorem is present; citations to Chen et al. and Feng et al. are external prior work. Score 4 reflects one self-referential evaluation loop while the core transfer result retains independent content.
Assumptions & free parameters
free parameters (4)
- proposition sublist size n =
30
- BM25 hyperparameters k1, b =
k1=0.05, b=5
- RRF fusion constant k =
60
- top-k retrieved propositions for response generation =
20
assumptions (5)
- domain assumption Plain-text documents contain information that users ask about, and an LLM can identify it as propositions.
- domain assumption The Dialog-LLM can generate realistic user dialogs and self-annotate ground-truth propositions.
- domain assumption QRELSCORE and QUANTIDCE are valid automatic proxies for dialog relevance and coherence.
- ad hoc to paper Retrieval performance against self-selected ground-truth propositions measures dialog quality.
- ad hoc to paper First and last QA pairs should always be accepted.
Cite this review
Pith. "Pith review of Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions." pith.science (2026). https://pith.science/paper/XH7DDVDL
@misc{pith2026250704884,
author = {Pith},
title = {Pith review of: Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions},
year = {2026},
howpublished = {\url{https://pith.science/paper/XH7DDVDL}},
note = {Machine review of arXiv:2507.04884}
}
read the original abstract
We consider open-retrieval conversational question answering (OR-CONVQA), an extension of question answering where system responses need to be (i) aware of dialog history and (ii) grounded in documents (or document fragments) retrieved per question. Domain-specific OR-CONVQA training datasets are crucial for real-world applications, but hard to obtain. We propose a pipeline that capitalizes on the abundance of plain text documents in organizations (e.g., product documentation) to automatically produce realistic OR-CONVQA dialogs with annotations. Similarly to real-world humanannotated OR-CONVQA datasets, we generate in-dialog question-answer pairs, self-contained (decontextualized, e.g., no referring expressions) versions of user questions, and propositions (sentences expressing prominent information from the documents) the system responses are grounded in. We show how the synthetic dialogs can be used to train efficient question rewriters that decontextualize user questions, allowing existing dialog-unaware retrievers to be utilized. The retrieved information and the decontextualized question are then passed on to an LLM that generates the system's response.
Figures
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.