{"id":"daece22e-b53b-4b44-9e11-08131356b4ae","arxiv_id":"2507.03726","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An agent-based classify-resolve-answer transducer improves LLM answer accuracy on five of six QA benchmarks, but the gains come with extra latency and unstable classification.","lead":"This paper tests whether LLM agents that classify, rewrite, and then answer incomplete or ambiguous questions can improve question-answering systems. On five of six benchmark datasets, rewriting with an agent improved answer accuracy over answering directly, at the cost of extra LLM calls and some latency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's accuracy gains may be leakage artifacts: Procedure 2 inserts the resolver's ReAct explanation `e` (and, as printed, an answer message `!α(q')`) into the responder's context, and no experiment strips `e` or checks whether `e` contains the answer.","rationale":"I read the paper as an empirical claim: a zero-shot ReAct classify-resolve-answer loop around an LLM improves QA accuracy on deficient-question datasets. For that claim to hold, the accuracy measure must isolate the effect of the transducer's outputs. The strongest threat is not statistical noise or the missing one-step baseline (though both matter); it is that the protocol changes the responder's input in a way that can trivially reveal the answer. Procedure 2 explicitly inserts the resolver's explanation `e` into the context, and ReAct explanations are chains of thought that typically contain the answer. On MedDialog, MultiWOZ, and ShARC, the fraction of possibly incomplete/ambiguous questions is high, so `e` is present for most of the very instances where accuracy jumps. The paper's own Table 6 shows the agent producing the answer inside its reasoning. The authors report manual AGREES verification but do not record whether the human verifier saw the `e`-containing context or only the final answer; in either case, the responder already had the answer. This concern is internal to the paper, not a disagreement with field consensus. It is stronger than the reader's other points because it changes what is being measured. Credit where due: the formal messaging definitions are consistent, the cost tables are honest, and the paper explicitly admits Classify is unpredictable (Sec. 4.4, Limitations). None of that addresses leakage. I agree with the reader's weakest_assumption. I keep the CONDITIONAL verdict: the claim is plausible but unverified, and the single-context ablation described above would decide it. If the ablation shows the gains vanish, the paper should be rejected or heavily revised.","tokens_in":17470,"tokens_out":6706,"duration_ms":76626,"concrete_test":"Re-run the §4.3 protocol with a single change: replace Procedure 2 line 4 with `c' = (m1, ..., mk, ?α(q'))` -- pass only the resolved question, omitting `e` (and fixing the printed `!` to `?`). Keep the responder and all other settings identical. Recompute Table 1 on MedDialog, MultiWOZ, and ShARC. If the accuracy gains over the no-transducer column collapse to near zero or invert, the headline gains are leakage artifacts; if gains persist, answer leakage is not the explanation. As a secondary diagnostic, compute the rate at which a 3+ token n-gram from `e` appears in the ground-truth answer; a rate near zero would further exonerate leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the accuracy differences in Table 1 measure the transducer's detection/resolution rather than answer leakage. In Procedure 2 (line 4), when Classify labels a question incomplete or ambiguous, Resolve produces `c' = (m1, ..., mk, e, !α(q'))` -- as printed, the context contains the resolver's explanation `e` and an answer message, not just a rewritten question `?α(q')`. Even if the `!` is a typo, `e` alone is a leak channel: ReAct explanations routinely state the answer before the final action, and the resolver has to reason about the answer to rewrite the question. In the datasets where gains are largest, almost every instance is affected (MedDialog 0.92 possibly incomplete; MultiWOZ 0.75 and ShARC 0.61 possibly ambiguous, Table 2). Thus the 'with transducer' condition supplies the responder with extra text containing the answer, while the 'without' condition does not. Table 6 shows the same reasoning style: the agent names John Powell before the final answer. No experiment in §4.3 or §4.4 removes `e`, masks answer spans, or measures substring overlap between `e` and ground truth, so the reported 3-40% gains could be copying artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a context-transducer for interactive question-answering with LLMs. The transducer is implemented as zero-shot ReAct LLM agents that first classify a question as incomplete, ambiguous, or normal and then, if needed, rewrite the question before the responder-LLM answers. The authors evaluate on six QA datasets (SQuAD, NQ-open, AmbigNQ, MedDialog, MultiWOZ, ShARC) using GPT-3.5-Turbo as the responder and either GPT-3.5-Turbo or Llama-4-Scout as the transducer-LLM. They report accuracy gains in five of the six datasets by the third turn, claiming improvements from 3% to 40%, as well as shortened interactions, at the cost of extra LLM calls and latency. The central claim is that including the question-transducer improves interactive QA performance.","tokens_in":17668,"tokens_out":6463,"duration_ms":70418,"significance":"If the reported gains are reliable, the paper describes a lightweight, zero-shot wrapper that could make LLM-based QA systems more robust to incomplete and ambiguous questions. The formal oracle-based definitions in Section 2 are a useful conceptual contribution, and it is a genuine strength that the transducer's Classify and Resolve components are implemented zero-shot with no parameter fitted to the test labels; the dataset characterization in Appendix A is explicitly retroactive and is not available to the transducer, so circularity is avoided. The paper also honestly lists several limitations, including the unpredictability of the Classify function and the need for diverse human verifiers. However, the current evaluation does not yet establish the central claim because of a plausible answer-leakage channel and the absence of statistical controls.","major_comments":[{"comment":"The evaluation does not control for answer leakage from the resolver's explanation into the responder's context. Procedure 2 (line 4) constructs c' = (m1,...,mk, e, !α(q')), placing the resolver's explanation e (and, as printed, an answer message) directly in front of the responder-LLM. Since Λr is implemented as a zero-shot ReAct agent, e may state or imply the final answer, as the worked example in Table 6 indeed shows the agent discussing 'John Powell' before the final answer. The accuracy gains in Table 1 could therefore reflect the responder copying the answer from e rather than an improved question. I request an ablation that removes e from c' (or masks any answer spans), and a measurement of substring/n-gram overlap between e and the ground-truth answer, for all datasets. Without such a control, the central claim of Section 4.4 is not supported.","section":"Section 4.3, Procedure 2"},{"comment":"The reported accuracies lack measures of uncertainty. The method states that there are about 600 (c,r) entries overall (Section 4.3, last bullet), so each dataset has roughly 100 instances per turn. Differences such as NQ-open turn 1 (0.81 vs 0.83) and AmbigNQ turn 1 (0.63 vs 0.65) are well within sampling error. The authors should report confidence intervals (e.g., bootstrap) or significance tests for the with- versus without-transducer comparisons in Table 1, and should state the exact number of instances per dataset. In addition, the manual AGREES verification was performed by a single human; an inter-annotator agreement measure on a subset is needed.","section":"Section 4.3"},{"comment":"The design does not include a control for the extra inference step itself. The 'without transducer' condition is a single LLM call answering the original question; the 'with transducer' condition is a multi-step ReAct loop that produces a rewritten question and explanation. To attribute gains to detection and resolution of incompleteness/ambiguity, the authors should add a baseline in which the question is rewritten by a simple LLM prompt (e.g., 'rewrite this question to be complete and unambiguous before answering') without the classify-resolve machinery, and a condition in which the responder receives the rewritten question without the explanation e. Without these controls, the gains could be caused by the additional compute or by the general benefit of reformulation, not by the claimed agent-based detection.","section":"Section 4.1, Section 4.3"},{"comment":"The behavior of Classify is inconsistent with the dataset characterisation used to motivate the evaluation. For AmbigNQ, Table 2 estimates 0.36 of questions are possibly ambiguous, yet the GPT-based transducer labels only 0.04 as ambiguous at turn 1 (Table 3); for MedDialog, 0.92 possibly incomplete is roughly matched (0.99), but for SQuAD, 0.00 possibly incomplete is labeled 0.73 incomplete at turn 1. Since the manuscript's Limitations section itself calls the Classify function 'quite unpredictable', the reader cannot tell which component (Classify, Resolve, or the responder's reaction to the added text) drives the Table 1 gains. The authors should report accuracy stratified by the Classify label and by whether Resolve actually rewrote the question, for each dataset and turn.","section":"Section 4.4, Table 3"}],"minor_comments":[{"comment":"The message is printed as '!α(q′)' but the resolver's output is defined as a rewritten question '?α(q′)'. Please correct the typo and, in the revision, explicitly state whether the responder sees only the rewritten question or also the resolver's answer message; this is directly related to the leakage concern.","section":"Procedure 2, line 4"},{"comment":"The dataset name 'SQuaAD' appears with an extra 'a' in the text; also, the table caption 'Role of increasing context on the proportions of correct answers' is unclear because the table actually compares accuracy with and without the transducer.","section":"Section 4.4, Table 1"},{"comment":"The notation in the set definitions Cd,k and CT,d,k is confusing: the predicate uses 'a' both for the ground-truth answer and inside AGREES(a, aλ). Please clarify the notation, for example by denoting the ground-truth answer as a* and the LLM output as a_λ.","section":"Section 4.3"},{"comment":"The reference list contains several duplicated entries (e.g., [11] and [12] are the same paper in preprint and EMNLP versions, and [45] and [46] are similarly duplicate AmbigQA entries). Please consolidate these to avoid citation clutter.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The answer-leakage concern is the most serious issue and should be resolved with an ablation that removes the resolver's explanation from the responder's context, together with a measure of answer-span overlap. The absence of significance tests and the lack of a simple-rewrite baseline also need to be addressed. The paper's formal framework and zero-shot implementation are strengths, but the empirical evidence as it stands does not support the main claim. The manuscript is within scope for the journal, but the evaluation section requires substantial strengthening before it can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you're doing anything on LLM robustness wrappers, this is a useful paper to know about, but the headline result is not yet supported. The empirical claim—that a zero-shot ReAct classify-resolve-answer loop lifts QA accuracy on deficient-question datasets by 3-40%—is interesting and consistent across most of the six datasets. That consistency is the paper's strength, along with its honesty: the authors report extra LLM calls and latency, admit the Classify step is unpredictable, and provide resolved-and-accurate breakdowns.\n\nThe soft spot is real and load-bearing. Procedure 2 puts the resolver's explanation e (and, as printed, an answer message !α(q')) into the context that goes to the responder LLM. ReAct explanations routinely include the answer before the final action—Table 6 shows exactly that reasoning style—so the responder can copy the answer from e. The 'with transducer' condition gives the responder extra text containing the answer; the 'without' condition doesn't. No experiment strips e, masks answer spans, or checks overlap between e and ground truth. So the reported gains could be leakage artifacts, not evidence that detection and resolution improve QA.\n\nThe other problems are smaller but still real. The evaluation is thin: about 600 QA pairs total, no significance tests or confidence intervals, manual verification with no inter-annotator agreement. There's no baseline where a single LLM prompt does the rewrite without a multi-step agent, so you can't tell what the agent loop adds over a simple rewrite. And the 'shorter interaction' claim in the abstract isn't directly measured—they report accuracy at fixed turns, not turns-to-accuracy.\n\nThe formal transducer definitions are light notation; that's fine, they don't do much work. The authors themselves mark the oracle definitions as motivational.\n\nBottom line: the idea is worth taking seriously, and the paper deserves a serious referee. But a referee should send it back and ask for a no-explanation control (send only the rewritten question), removal of the stray answer message, significance testing, and a one-step rewrite baseline. Until then, read the headline numbers as possibly inflated.","headline":"The agent-based QA wrapper idea is plausible and worth discussing, but the reported accuracy gains are not yet reliable: the evaluation inserts the resolver's explanation into the responder's context, creating a likely answer-leakage channel, plus the statistics are thin.","tokens_in":18284,"tokens_out":3723,"would_cite":false,"duration_ms":40793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding an agent that rewrites unclear questions boosts LLM answer accuracy on 5 of 6 datasets.","keywords":["question answering","LLM agents","ReAct","incompleteness","ambiguity","question transducer","zero-shot classification","interactive dialogue"],"falsifier":"Re-run the evaluation but remove the explanation from the transduced context before the responder answers, keeping only the rewritten question; if accuracy falls substantially, the explanation is leaking the answer. A complementary check is to measure token overlap between the resolver's explanation and the ground-truth answer on the resolved questions.","tokens_in":17244,"feed_emoji":"🤖","tokens_out":5950,"duration_ms":58896,"temperature":0.7,"pith_summary":"The paper tests a simple idea: before an LLM answers a question, an agent-based 'question-transducer' checks whether the question is incomplete or ambiguous, rewrites it if it is, and only then lets the LLM answer. The authors argue that this classify-resolve-answer loop improves interactive question-answering performance, shortens human clarification turns, and makes the resolution of deficiencies explainable. Across six benchmark datasets with varying levels of incompleteness and ambiguity, the transducer improved answer accuracy on five of them, with gains ranging from 3 to 40 percentage points by the last turn tabulated. The approach does not help when questions already carry enough context, because the classifier then tends to alter normal questions. The costs are additional LLM invocations and, in some cases, higher latency, which the paper judges acceptable except in the already-well-specified case.","feed_headline":"Question-rewriting agents lift LLM accuracy by up to 40%","feed_subtitle":"A classify-resolve-answer loop fixes incomplete or ambiguous questions and cuts clarification turns.","key_machinery":"The question-transducer is a context-transducer for questions: a function that maps a context ending in a question to a new context ending in a rewritten question. It is implemented as two zero-shot ReAct LLM agents: Classify labels the question as incomplete, ambiguous, or normal, and Resolve rewrites deficient questions into clear and complete ones or returns a clarifying question. The rewritten question, together with the explanation produced by Resolve, is inserted into the context that the responder LLM receives, and the responder then answers the resolved form. The ReAct loop lets the agent alternate reasoning and action until its classification or resolution goal is achieved.","core_discovery":"The central claim is that the conjecture holds: the inclusion of a question-transducer to detect and resolve incomplete or ambiguous questions improves the interactive question-answering performance of an LLM. Accuracy is measured after each turn against human-labeled ground truth on 600 interactions from SQuAD, NQ-open, AmbigNQ, MedDialog, MultiWOZ, and ShARC, using GPT-3.5-Turbo as the responder and either GPT-3.5-Turbo or Llama-4-Scout as the transducer LLM. The paper reports that for 5 of the 6 datasets, introducing the transducer increases accuracy even from the first turn, and that the gain ranges from 3% to 40% by the last turn. It also reports that the GPT-based transducer outperforms the Llama-based one on 3 datasets, and vice versa. The exception is SQuAD, where accuracy drops because the Classify function mislabels many normal questions, so the paper concludes the transducer is not useful when questions already contain sufficient context.","pith_inferences":["The protocol inserts the resolver's explanation into the responder's context, and ReAct explanations often state the answer before the final response; the paper does not test whether the accuracy gain is partly answer leakage rather than genuine resolution.","A simple ablation—handing the responder only the rewritten question, without the explanation—would isolate whether the gain comes from the resolution or from copying the explanation.","The Classify component's unpredictable behavior suggests a small trained classifier on human labels, with the agent-based Resolve retained, might be a more reliable and cheaper architecture.","The same transducer pattern could transfer to other LLM-facing tasks where a deficient prompt blocks a deterministic downstream step, such as code generation or database queries."],"forward_implications":["On five of the six datasets tested, adding the transducer improves answer accuracy, with gains of 3 to 40 percentage points by the third turn.","The transducer shortens interactions by automatically resolving incompleteness and ambiguity, reducing the number of human clarification turns needed.","The approach costs extra LLM invocations and added latency, so it is best applied selectively rather than on every query.","When questions already contain sufficient context, the transducer can reduce accuracy, so a reliable Classify function is essential to avoid rewriting normal questions."],"supporting_citations":[{"why":"Supplies the low-incompleteness, low-ambiguity benchmark on which the transducer is expected not to help and indeed loses accuracy.","marker":"[54]"},{"why":"Supplies the open-domain Natural Questions benchmark used for accuracy comparison across turns.","marker":"[37]"},{"why":"Provides the AmbigNQ dataset of curated ambiguous questions, the key test for ambiguity resolution.","marker":"[46]"},{"why":"Provides the MedDialog medical dialogue dataset with high incompleteness, where the largest accuracy gains appear.","marker":"[70]"},{"why":"Supplies the MultiWOZ multi-domain task-oriented dialogues used as a high-ambiguity, high-incompleteness test.","marker":"[6]"},{"why":"Supplies the ShARC conversational machine-reading dataset with rules and reasoning, used as a mid-to-high deficiency test.","marker":"[58]"},{"why":"Provides the agent software environment used to implement the zero-shot ReAct Classify and Resolve agents.","marker":"[66]"}],"fun_headline_variants":["Agents detect and fix vague questions, boosting LLM answers","LLM agents resolve fuzzy questions, accuracy up 40%","Auto question-cleanup lifts LLM accuracy to 40%","Classify-resolve-answer loop sharpens LLM responses","Agents rewrite vague questions for better LLM answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the explanation inserted by the resolver into the responder's context does not already contain the answer; if it does, the measured accuracy gain could be a copying effect rather than evidence that detection and resolution improve question answering.","fun_headline_variants_meta":{"raw":{"variants":["Agents detect and fix vague questions, boosting LLM answers","LLM agents resolve fuzzy questions, accuracy up 40%","Auto question-cleanup lifts LLM accuracy to 40%","Classify-resolve-answer loop sharpens LLM responses","Agents rewrite vague questions for better LLM answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3580,"prompt_tokens":1105,"completion_tokens":2475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":2390}},"tokens_in":721,"tokens_out":2475,"duration_ms":20434,"temperature":1.0,"reasoning_tokens":2390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:04:06.052950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation but remove the explanation from the transduced context before the responder answers, keeping only the rewritten question; if accuracy falls substantially, the explanation is leaking the answer. A complementary check is to measure token overlap between the resolver's explanation and the ground-truth answer on the resolved questions.","supporting_citations":[{"cited_title":"Creating large language model applications utilizing langchain: a primer on developing llm apps fast","cited_arxiv_id":null,"evidence_quote":"Provides the agent software environment used to implement the zero-shot ReAct Classify and Resolve agents."}],"review_version":1}