{"id":"076ab490-b3f2-4686-9824-7dc644cbc578","arxiv_id":"2411.12395","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding a rephrasing or context-enrichment prompt improves LLM answer similarity on ambiguous QA over naive prompting, though gains are small and not statistically verified.","lead":"The paper tests three prompting strategies for answering ambiguous questions with GPT-4o and GPT-4o-mini and finds that simple rephrasing or adding context improves answer similarity to human ground truth. This is useful for practitioners who want cheap, training-free ways to reduce LLM misinterpretation of ambiguous queries.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains from disambiguation are smaller than the paper's own baseline variability: naive GPT-4o-mini GT overlap is 0.692 in Table II but 0.643 in RQ2, a 0.049 spread that exceeds every reported disambiguation gain (0.018–0.030), and no significance testing is provided.","rationale":"I read the paper's central claim as an empirical claim that simple prompt-based disambiguation improves LLM answer quality on ambiguous QA. For that claim to hold, the reported GT Answer Overlap differences must be real effects rather than sampling noise. The most direct internal evidence against this is the inconsistency between Table II (naive GPT-4o-mini = 0.692) and Section V RQ2 (naive GPT-4o-mini = 0.643). The spread between these two estimates of the same baseline, 0.049, exceeds every disambiguation gain the paper reports, yet no significance testing or confidence intervals appear anywhere. The reader's concern about the unvalidated cosine-similarity metric is valid and compounds the problem, but the baseline inconsistency is sharper because it does not depend on external assumptions about the embedding model. The paper is honest about its limitations and does not overclaim in the conclusion, which supports a conditional rather than a reject verdict. However, before the headline claim can be accepted, the authors need to demonstrate that the effect is not noise, ideally with paired tests and a repeated baseline sample. If the effect survives that check, the result is plausible; if not, the conclusion should be weakened to 'disambiguation changes outputs' rather than 'improves performance.'","tokens_in":8401,"tokens_out":8055,"duration_ms":71831,"concrete_test":"Recompute the RQ1 comparison as paired per-question differences between naive and context-disambiguated GT Answer Overlap on the same 1,000 questions; report the mean difference with a bootstrap 95% CI and a paired permutation test. Then run the naive GPT-4o-mini baseline on a second independent 1,000-question sample from the same AmbigQA train split (or reuse the RQ2 sample) to quantify between-sample variability; if the CI for the disambiguation effect overlaps zero, or if the between-sample baseline spread (0.692 vs 0.643) exceeds the claimed gain, the conclusion should be weakened. As a secondary check, calibrate the cosine metric against exact-match or token-level F1 on a labeled subset to confirm that measured cosine gains correspond to genuine correctness improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V RQ2 reports a fresh 1,000-question evaluation of naive GPT-4o-mini with GT Answer Overlap 0.643, while Table II reports the same naive GPT-4o-mini condition on the main 1,000-question sample as 0.692. The 0.049 difference between these two estimates of the same baseline is larger than the paper's headline effects: context disambiguation improves GPT-4o-mini by only 0.710−0.692=0.018 and GPT-4o by 0.789−0.759=0.030, and rephrasing gains are even smaller. No confidence intervals, standard errors, or paired tests are reported. The central claim that training-free disambiguation improves QA performance therefore rests on mean cosine-similarity differences that lie within the sampling variability the paper itself exhibits. The evaluation metric, cosine similarity between text-embedding-3-large embeddings of the model answer and a human ground-truth answer, is also unvalidated for correctness, so we cannot tell whether such small cosine shifts correspond to real answer improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether GPT-4o and GPT-4o-mini can handle ambiguous open-domain questions, comparing naive prompting with two prompt-level disambiguation methods (rephrasing via 'what' and adding contextual enrichment) on 1,000 randomly sampled AmbigQA questions. It reports gains in 'GT Answer Overlap' (cosine similarity between the model answer embedding and the ground-truth answer embedding) for both methods, a small-scale fine-tuning experiment on 50 examples, and a temperature ablation. The central claim is that simple, training-free, prompt-based disambiguation improves LLM performance on ambiguous question answering.","tokens_in":8587,"tokens_out":4618,"duration_ms":41298,"significance":"If the central claim were statistically supported, the paper would offer a low-cost, practical intervention for ambiguous QA and a useful comparison of disambiguation strategies. The paper has clear strengths: it uses a standard public dataset (AmbigQA), reports explicit prompt templates, evaluates two commercially relevant models, and includes an honest limitations section. However, the empirical support is currently insufficient: the reported effect sizes are small, no uncertainty quantification is provided, the evaluation metric is not validated as a measure of answer correctness, and the prompt-selection procedure risks circularity. The central finding is plausible but not established by the reported numbers.","major_comments":[{"comment":"The paper's own data show that the naive GPT-4o-mini baseline is not stable at the scale of the claimed improvements. Table II reports a naive GT Answer Overlap of 0.692, while the RQ2 evaluation of the same naive condition on a fresh 1,000-question sample reports 0.643; the 0.049 gap is larger than the largest reported disambiguation gain (0.030 for GPT-4o and 0.018 for GPT-4o-mini). No confidence intervals, standard errors, or paired significance tests are reported anywhere. As presented, the headline claim that disambiguation improves accuracy cannot be distinguished from sampling variability. Please provide paired bootstrap confidence intervals or significance tests, report the number of independent API runs, and state how the random samples were drawn.","section":"Section V, RQ2 (with Table II)"},{"comment":"The evaluation metric is an uncalibrated cosine similarity between text-embedding-3-large embeddings of the model answer and the human ground-truth answer. The paper does not show that a 0.01-0.03 cosine shift corresponds to a meaningful correctness improvement; no threshold, calibration against exact match, or human judgment is provided. Because all conclusions rest on this metric, please validate it on a labeled sample (e.g., by comparing cosine similarity against human-rated correctness or exact-match scores) and, if possible, report results under an additional metric such as exact match or token-level F1.","section":"Section IV-D"},{"comment":"The disambiguation prompts were chosen after 'experimenting with a variety of prompts' with no description of a held-out validation split. If the same 1,000-question sample was used both to select the prompts and to report the results, the reported improvements are at least partially an artifact of selection on the evaluation set. Please describe the prompt-search procedure, use a separate validation set for prompt selection, and report results on a held-out test sample.","section":"Section IV-C"},{"comment":"The fine-tuning conclusion that 'fine-tuning ... does not provide any improvement' is based on a comparison of 0.643 (naive 4o-mini) versus 0.626 (fine-tuned 4o-mini) on a fresh sample of 1,000 questions. This 0.017 difference is comparable in magnitude to the disambiguation gains the paper highlights, and no error bars or significance tests are reported for it. In addition, the comparison may be confounded by sampling a different set of questions for the fine-tuned evaluation. Please treat this negative result with the same statistical rigor as the main comparisons.","section":"Section V, RQ2"}],"minor_comments":[{"comment":"The methods are described as 'token-level disambiguation,' but both prompts operate on whole questions or on appended context passages; please correct this terminology to avoid overstating the granularity of the interventions.","section":"Abstract and Section IV-C"},{"comment":"The text says 'why the average is not going up when an LLM is prompted to insert context into a question,' but Tables I and II report that context disambiguation gives the highest GT Answer Overlap (0.789 and 0.710). This subsection appears to refer to the distributions in Figures 2 and 3 rather than the means in the tables; please rewrite to avoid the apparent contradiction.","section":"Section V, 'Problem with naive contextual enrichment'"},{"comment":"The fine-tuning data format example is not valid JSON: the 'messages' value is not shown as an array and the assistant content string has mismatched quotes. If this is meant to be illustrative, please correct it or mark it as pseudocode.","section":"Section V, RQ2"},{"comment":"The 'Upper-bound (via GT disamb. questions)' column is not defined in the text; please explain how this upper bound is computed (for example, whether it averages over all human disambiguations for each question) and why it constitutes an upper bound.","section":"Tables I and II"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims are, as written, within the range of the paper's own baseline variability: the naive GPT-4o-mini baseline differs by 0.049 between Table II and RQ2, while the headline disambiguation gains are 0.018-0.030. I would ask the authors to provide confidence intervals, validate the embedding-based metric, and move prompt selection to a held-out set. If these analyses cannot be supplied, the paper should not be published on the current evidence. The paper may otherwise be a reasonable fit for a specialized NLP venue, but it needs a substantive reanalysis first."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central claim — that simple training-free disambiguation prompts improve LLM performance on ambiguous questions — is plausible but not actually supported by the reported numbers. The baseline itself moves more than the treatment effect. Table II gives naive GPT-4o-mini GT Answer Overlap as 0.692; Section V RQ2 reports a fresh 1,000-question evaluation of the same naive condition at 0.643. That 0.049 spread is larger than every disambiguation gain they report (0.018–0.030). Without confidence intervals, paired tests, or a fixed evaluation sample, the mean differences might just be sampling noise.\n\nWhat is genuinely useful: the paper tries two concrete, reproducible prompt strategies — rephrasing to 'what' and appending model-generated context — on AmbigQA with GPT-4o and GPT-4o-mini, and the direction of the effect is consistent across both models and both strategies in the main tables. That is a legitimate practical datapoint for prompt engineering, and the authors are transparent about the limitations, including the problematic context-enrichment behavior.\n\nThe soft spots are real and they are load-bearing. The evaluation metric is cosine similarity between text-embedding-3-large embeddings of the model answer and the ground-truth answer, with no calibration against exact match or human judgment, so a 0.02 cosine shift could correspond to a genuinely better answer, a worse one, or just a different phrasing. The prompts were selected after 'experimenting with a variety of prompts' with no held-out validation described, which raises a selection-on-the-evaluation-sample concern. And the RQ2 baseline inconsistency is not explained — if the two naive numbers come from different random samples, the comparison should be reported as a range, and if they come from the same sample, something is wrong.\n\nWho is this for? Someone doing prompt engineering or robustness evals might cite the specific numbers, but the empirical claim needs stronger support before I'd trust the conclusion. The fine-tuning result (no improvement) is also underpowered at 50 examples and should be labeled as preliminary.\n\nRecommendation: send to peer review with the expectation of major revision. The question matters, the paper is honest, and a serious referee can push for significance testing, a validated metric, and a fixed evaluation protocol. As it stands, it is a conference-adjacent empirical note, not a settled finding.","headline":"The disambiguation gains vanish once you look at the paper's own baseline variability; still a useful prompt-engineering datapoint, but the evaluation needs rigor.","tokens_in":9153,"tokens_out":1801,"would_cite":false,"duration_ms":16506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding model-generated context or rewriting an ambiguous question with \"what\" improves how off-the-shelf LLMs answer ambiguous open-domain questions; small-scale fine-tuning and lower temperature do not.","keywords":["ambiguity","large language models","question answering","disambiguation","prompt engineering","AmbigQA","cosine similarity","fine-tuning"],"falsifier":"Score the same 1,000-question outputs with exact string match (or human annotation) instead of embedding cosine similarity; if the context-enriched and \"what\" conditions do not beat the naive condition on that stricter measure, the paper's central claim is refuted. A faster check is to compute bootstrap confidence intervals for the reported GT Answer Overlap means—overlapping intervals would show the gains are indistinguishable from noise.","tokens_in":8152,"feed_emoji":"❓","tokens_out":7287,"duration_ms":66392,"temperature":0.7,"pith_summary":"Using open-domain question answering as a test bed, this paper tries to establish that off-the-shelf LLMs are sensitive to ambiguity in the question, and that simple, training-free prompt edits can partially fix that sensitivity. On 1,000 ambiguous questions from AmbigQA, both GPT-4o and GPT-4o-mini scored higher against ground-truth answers when the question was first rewritten to start with \"what\" or enriched with model-generated context than when answered naively. The paper also claims that small-scale fine-tuning with 50 examples and lowering the sampling temperature do not improve performance on ambiguous questions. The central insight is that what matters is disambiguating the question before asking, rather than more training or more deterministic decoding.","feed_headline":"Adding context improves LLM answers to ambiguous questions","feed_subtitle":"Context prompts raised GPT-4o answer scores on 1,000 ambiguous questions; fine-tuning and low temperature did not.","key_machinery":"The mechanism is a two-pass prompting pipeline. In pass one, the LLM rewrites the ambiguous question either to begin with \"what\" (a rephrasing prompt) or to include a generated block of extra context (a contextual-enrichment prompt); in pass two, the modified question is fed back to the LLM for a concise answer. The paper's evaluation machinery is cosine similarity between text-embedding-3-large embeddings of the model's answer and the dataset's ground-truth answer, supplemented by similarity measures between the rewritten question and the human-provided disambiguated question (question coherence) and between naive and disambiguated answers (naive answer overlap). The ground-truth disambiguated questions serve as an upper-bound reference for how much improvement is possible.","core_discovery":"The paper's central claim is that explicit, training-free disambiguation at the prompt level improves LLM accuracy on ambiguous open-domain questions. In their experiments, adding context to the ambiguous question raised the mean cosine similarity between the model's answer and the ground-truth answer from 0.759 to 0.789 for GPT-4o and from 0.692 to 0.710 for GPT-4o-mini; rephrasing with \"what\" produced smaller gains. The same experiments show that a 50-example fine-tune of GPT-4o-mini actually lowered the score (0.643 vs. 0.626 for the base model under naive prompting), and that temperature 0.2 versus 1.0 made no meaningful difference. The authors conclude that even though LLMs often add wrong context when left to their own devices, the contexts they add on questions humans could disambiguate align with ground truth, so simple prompting strategies are a promising direction for reducing ambiguity-driven errors.","pith_inferences":["My inference: the reported differences are small on a 0–1 cosine scale and the paper reports no confidence intervals or significance tests, so the practical size of the effect is uncertain even if the direction is real.","My inference: because the evaluation metric is embedding similarity, a stricter exact-match test might rank the methods differently; the paper's \"accuracy\" language goes beyond what cosine similarity alone establishes.","My inference: the fine-tuning negative result applies only to a 50-example, single-configuration tune; larger or more targeted fine-tuning could still learn disambiguation, so the paper's training-free conclusion is scale-limited.","My inference: the finding that context enrichment works better exactly when the human disambiguator also found the correct reading suggests a testable extension—filtering generated context for entity-bearing phrases before answering could improve the hit rate."],"forward_implications":["A simple preprocessing step that rewrites user questions or adds context could improve open-domain QA accuracy without retraining, which matters for deployed systems where users rarely disambiguate.","Context enrichment is a high-variance strategy: it helps when the model happens to add relevant context and is held back when it adds irrelevant context, so systems using it need a way to detect when the added context is off-target.","Small-scale fine-tuning on ambiguous QA pairs is not an obviously better use of resources than prompt-level disambiguation, at least at the 50-example scale tested.","Lowering the temperature is not an effective ambiguity remedy; the model's misinterpretation persists under more deterministic decoding."],"supporting_citations":[{"why":"Supplies the AmbigQA dataset of 14,042 ambiguous questions with multiple answers and human-provided disambiguated rewrites; this is the test bed for all experiments.","marker":"[21]"},{"why":"Supplies NQ-Open, the real-user query source from which AmbigQA was built, grounding the claim that the questions reflect genuine open-world ambiguity.","marker":"[20]"},{"why":"Defines scope ambiguity in LLMs and motivates the contextual-enrichment prompt as a method for reducing that ambiguity.","marker":"[22]"},{"why":"Provides prior evidence that LLMs respond to task ambiguity, framing the paper's question-level ambiguity study.","marker":"[16]"},{"why":"Cited in the limitations to explain the fine-tuning result as probable catastrophic forgetting, supporting the paper's interpretation of why training did not help.","marker":"[24]"}],"fun_headline_variants":["Context prompts improve ambiguous LLM answers, fine-tuning does not","Adding context outperforms fine-tuning for ambiguous questions","LLM ambiguity: context helps, fine-tuning hurts","Disambiguation prompts beat fine-tuning for ambiguous LLM QA","Simple context prompts reduce LLM ambiguity errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results are only as valid as the assumption that cosine similarity between the model's answer embedding and the ground-truth answer embedding measures answer correctness; the paper does not validate that metric against exact matches or human judgment, and the reported improvements are small mean differences without significance testing.","fun_headline_variants_meta":{"raw":{"variants":["Context prompts improve ambiguous LLM answers, fine-tuning does not","Adding context outperforms fine-tuning for ambiguous questions","LLM ambiguity: context helps, fine-tuning hurts","Disambiguation prompts beat fine-tuning for ambiguous LLM QA","Simple context prompts reduce LLM ambiguity errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1610,"prompt_tokens":880,"completion_tokens":730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":654}},"tokens_in":496,"tokens_out":730,"duration_ms":7076,"temperature":1.0,"reasoning_tokens":654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:34:14.302597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same 1,000-question outputs with exact string match (or human annotation) instead of embedding cosine similarity; if the context-enriched and \"what\" conditions do not beat the naive condition on that stricter measure, the paper's central claim is refuted. A faster check is to compute bootstrap confidence intervals for the reported GT Answer Overlap means—overlapping intervals would show the gains are indistinguishable from noise.","supporting_citations":[{"cited_title":"AmbigQA: Answering ambiguous open-domain questions,","cited_arxiv_id":null,"evidence_quote":"Supplies the AmbigQA dataset of 14,042 ambiguous questions with multiple answers and human-provided disambiguated rewrites; this is the test bed for all experiments."},{"cited_title":"Natural questions: a benchmark for question answering research,","cited_arxiv_id":null,"evidence_quote":"Supplies NQ-Open, the real-user query source from which AmbigQA was built, grounding the claim that the questions reflect genuine open-world ambiguity."},{"cited_title":"Scope ambiguities in large language models,","cited_arxiv_id":null,"evidence_quote":"Defines scope ambiguity in LLMs and motivates the contextual-enrichment prompt as a method for reducing that ambiguity."}],"review_version":1}