{"id":"8f46fa66-3c96-4176-b45c-4d2bf0ad6a1c","arxiv_id":"2509.06631","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Guided decoding backends show different false positive rates across 0, 1, and 2-turn RAG, but the paper's reported numbers are internally inconsistent and lack a no-guidance baseline.","lead":"This paper compares three guided decoding backends (Outlines, XGrammar, and LM Format Enforcer) on multi-turn retrieval-augmented generation over Turkish legal documents, measuring reference success and hallucination rates. It reports that decoding choice and conversation depth affect structured output quality, but the supporting numbers contain internal contradictions and there is no baseline without guided decoding.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No unguided baseline means the 'guided decoding is critical' claim is untestable; Table III also contradicts the prose's backend ranking.","rationale":"The reader's verdict is REJECT with high confidence, and I agree that the paper's central claim is not supported. I identify a different load-bearing issue than the reader's narrowest assumption: although regex-extraction fidelity is a real risk, the more fundamental problem is that the experiments never include an unguided condition, so the headline 'guided decoding is critical' cannot be evaluated at all. The paper also contains a direct contradiction between the prose ranking of LMF and the numbers in Table III, which independently undermines the backend-comparison results. These issues are internal to the paper's argument, not disagreements with external consensus. No formal verification, code release, or parameter-free derivation is available to offset them; the Hugging Face dataset is only partially public and the exact regex in ExtractIDs is not specified. A targeted rerun with an unguided control and a reconciliation of Table III would resolve whether the central claim has any empirical basis.","tokens_in":5734,"tokens_out":4896,"duration_ms":43749,"concrete_test":"Use the 507 public samples to rerun Algorithm 1 with guided decoding disabled (same prompts, same models, no grammar or regex). Compare success and false-positive rates against the three backends across 0/1/2 turns with at least 5 repeated runs per condition to estimate variance. Also re-extract the 1-turn results from raw logs to determine whether Table III or the prose ranking is correct; if the table is correct, the prose's LMF 'lowest rates' claim is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is the absence of an unguided control. Section III compares Outlines, XGrammar, and LM Format Enforcer against each other, and Algorithm 1 always runs under a guided-decoding backend. The paper's central claim that 'guided decoding is critical for reliable RAG deployments' and that 'decoding strategy is critical for ensuring factual consistency and reference accuracy' requires showing that guided decoding changes outcomes relative to no guided decoding. Without that baseline, the low false-positive rates could be produced by the few-shot exemplars, the model's instruction following, or the RAG context alone. The only large-scale argument, the 10M-chunk/100k-query extrapolation in Section IV, is also not derived from Table III: a 0.12 percentage-point difference (0.61% vs 0.49%) is multiplied into 1,600 additional missed references without stating the base rate, and no confidence intervals or significance tests are reported anywhere. The internal evidence is additionally contradictory: the prose claims LMF achieved 'the lowest rates in 1-turn (0.73% and 0.33%)', but Table III lists LMF at 0.73% for Qwen while Outlines is 0.32% and XGrammar is 0.41%, and for Llama lists LMF at 0.33% while Outlines is 0.24%. The reported ranking of backends is therefore not reproducible from the paper's own data, and the central empirical claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates three guided-decoding backends (Outlines, XGrammar, LM Format Enforcer) for structured citation output in a retrieval-augmented generation (RAG) pipeline, using Qwen2.5-72B-Instruct and LLaMA-3.3-70B-Instruct across 0-, 1-, and 2-turn exemplar prompting. It reports false positive rates of document identifiers, defines success as at least one correct identifier with zero false positives, and extrapolates the results to a 10M-chunk, 100k-query scenario. The paper concludes that guided decoding and few-turn prompting are critical for factual consistency and reference accuracy in RAG systems.","tokens_in":6142,"tokens_out":5268,"duration_ms":40518,"significance":"The practical question addressed—which guided-decoding backend to use in RAG—is relevant, and the authors provide a public dataset and compare three widely used tools. However, the paper lacks an unguided control, contains internal contradictions between the prose and Table III, and presents an extrapolation that is not supported by the reported rates. If the findings were properly supported, they would be useful for practitioners, but in its current form the main claims are not established.","major_comments":[{"comment":"The evaluation never runs an unguided baseline. All experiments use one of the three guided-decoding backends, so the central claim in Section IV and the Conclusion that guided decoding is critical and that decoding strategy is critical for factual consistency cannot be separated from the effect of few-shot exemplars, instruction following, or the RAG context alone. An unconstrained decoding condition must be added before these claims can be assessed.","section":"Section III.A and Algorithm 1"},{"comment":"The prose contradicts the table. The text states that LM Format Enforcer (LMF) achieved the lowest false positive rates in 1-turn (0.73% and 0.33%) and maintained superior robustness, but Table III shows Outlines at 0.32% (Qwen 1-turn) and 0.24% (Llama 1-turn), both lower than LMF, and XGrammar at 0.12% (Qwen 2-turn), lower than LMF's 0.30%. The reported ranking of backends is therefore not reproducible from the paper's own data.","section":"Section IV, Table III"},{"comment":"The claim of 1,600 additional missed references for replacing LMF with XGrammar in the zero-turn setting is not derivable from Table III: the difference is 0.12 percentage points (0.61% vs 0.49%), which over 100,000 queries is 120, not 1,600. Moreover, in the 1-turn Qwen condition, Table III shows XGrammar with a lower false positive rate (0.41%) than LMF (0.73%), so replacing LMF with XGrammar would not increase misses as claimed. This extrapolation should be removed or re-derived with explicit base rates and a consistent interpretation of Table III.","section":"Section IV, extrapolation paragraph"},{"comment":"Sample sizes are unclear and no uncertainty is reported. Table I lists Total Samples 750, 375, and 250 without specifying whether these are per-turn subsets or cumulative totals, and no confidence intervals, standard errors, or significance tests are provided anywhere. Many headline differences (e.g., 0.12 percentage points) are too small to interpret without such information, especially given the statement that only 507 of 750 samples are publicly accessible.","section":"Section III.D and Table I"},{"comment":"The regex-based ExtractIDs is assumed to behave identically across all three decoding backends, but this is not validated. If a backend changes the spacing or formatting of the document identifiers, or if the generator emits the closing /doc_id tag differently, the false positive and success metrics would measure parsing artifacts rather than model citation behavior. The paper should report extraction validation, for example by showing example outputs per backend or measuring the regex failure rate.","section":"Algorithm 1"}],"minor_comments":[{"comment":"In the Related Work section, 'Y AML' should be 'YAML'.","section":"Section II"},{"comment":"Clarify what the numbers 750, 375, and 250 represent; the sentence 'we report a total of 750 samples' is ambiguous relative to the table.","section":"Table I"},{"comment":"The statement that 'false positive rates dropping dramatically from hundreds to single digits' is not supported by Table III, which reports percentages rather than counts; specify whether the sentence refers to counts and, if so, report counts.","section":"Section IV"},{"comment":"The figure plots 'Correct References (%)' while the text mostly discusses false positive rates; define what is plotted and add error bars or confidence bands.","section":"Figure 1"},{"comment":"The citation formatting is inconsistent, especially in references [3] and [4], where author names are partially reversed; a consistent style would improve readability.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper addresses a useful practical question and makes its dataset public, which I view positively. However, the absence of an unguided baseline, the internal contradictions between the prose and Table III, and the unsupported extrapolation are load-bearing flaws. The reported results cannot currently be trusted, and a proper revision would require substantial additional experiments rather than local corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to see someone finally benchmark Outlines vs XGrammar vs LM Format Enforcer on a real legal RAG task with two 70B models. The dataset is partly public and the multi-turn evaluation procedure is simple enough to reproduce. The concrete result that one or two exemplar turns cut false-positive rates from ~3% to ~0.3% is believable and practically useful.\n\nThe paper is not ready to support its own title. The central claim that guided decoding is critical is untestable because there is no unguided baseline. Every condition runs under a guided backend; the few-shot exemplars alone could be doing the work. That is a real missing control, not a nitpick.\n\nWorse, the prose and Table III disagree. The text says LMF achieved the lowest 1-turn rates (0.73% and 0.33%), but in the table LMF's Qwen 1-turn rate of 0.73% is the highest of the three, and its Llama rate of 0.33% is not lowest either. So the reader cannot trust which backend actually won. The 10M-chunk extrapolation is also not derivable from the tables: a 0.12 percentage-point gap (0.61% vs 0.49%) would give 120 misses per 100k queries, not 1,600, unless some hidden base rate is in play, and none is stated. There are no confidence intervals or significance tests anywhere, and the per-turn sample sizes are unclear (750 total across three turn conditions?).\n\nMinor but real: the regex extraction (ExtractIDs) may behave differently across backends if their output formatting differs; the evaluation metric could then measure parser artifacts rather than actual citation behavior. The paper also lists thoughtful limitations, which I appreciate. One paragraph in III.C.1 about PDA states and parser configurations reads like it wandered in from another paper; it does not connect to the experiment.\n\nWho this is for: engineers choosing a decoding backend for a structured-output RAG pipeline, provided they treat the numbers as a starting point rather than a result. The qualitative direction — multi-turn prompting dramatically reduces false positives, and backend choice matters — is plausible and likely correct. But the specific rankings and the extrapolated cost numbers are not reliable.\n\nRecommendation: send it to peer review rather than desk reject? Yes, a serious referee can get value out of re-analyzing the data and demanding a baseline. But as is, I would not accept; big revision needed. I would not cite it until the contradictions and the unguided baseline are fixed.","headline":"A useful but flawed comparison of guided decoding backends for RAG: the multi-turn result is believable, but the missing unguided baseline and contradictions between prose and tables gut the headline claim.","tokens_in":6494,"tokens_out":3025,"would_cite":false,"duration_ms":27066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in retrieval-augmented generation, the choice of guided decoding backend materially changes reference accuracy and hallucination rates, and that multi-turn prompting sharpens the differences between backends.","keywords":["retrieval-augmented generation","guided decoding","structured output","hallucination reduction","multi-turn prompting","Outlines","XGrammar","LM Format Enforcer"],"falsifier":"Re-run the same three backends through Algorithm 1 but replace the regex extraction with an independent parser or human annotation on the 507 publicly available responses; if the backend rankings of false-positive rates change or compress, the reported differences are extraction artifacts rather than genuine decoding effects.","tokens_in":5612,"feed_emoji":"🎯","tokens_out":8245,"duration_ms":66704,"temperature":0.7,"pith_summary":"The paper is trying to show that in retrieval-augmented generation, the decoding method that forces an LLM into a structured format is not an implementation detail but a driver of correctness. It compares three guided decoding backends — Outlines, XGrammar, and LM Format Enforcer — over zero, one, and two conversational turns, scoring success as at least one correct document reference and no invented references. Across the reported runs, added turns collapse false-positive citation rates from hundreds of errors to single digits, and the best backend changes with model and turn count. The authors extrapolate that at a scale of ten million chunks and one hundred thousand queries, the backend choice can account for thousands of missed references. That matters because production RAG systems often pick a default decoder without measuring its effect on grounding.","feed_headline":"Guided decoding backend changes RAG citation accuracy by thousands","feed_subtitle":"Fewer turns of context, or the wrong decoder, can cost thousands of correct document citations in a large RAG system.","key_machinery":"The central object is Algorithm 1, a multi-turn evaluation loop that builds a chat history with n exemplar turns, retrieves RAG context, obtains the model's response, extracts document IDs with a regex, and classifies the result as success (at least one correct ID, zero false IDs) or hallucination (more than zero false IDs). That evaluation definition transforms token-level decoding behavior into measurable success and false-positive rates. It is paired with three decoding mechanisms: Outlines, which uses finite-state machines to map regex and grammar states to valid token sets in constant time; XGrammar, which uses pushdown automata with precomputed token masks and a persistent execution stack; and LM Format Enforcer, which filters token probabilities to admit only format-compliant tokens. The algorithm's success/hallucination split is what produces the headline numbers.","core_discovery":"The paper's central claim is that guided decoding — constraining an LLM's output to a specified format — is critical for factual consistency and reference accuracy in retrieval-augmented generation, and that the specific backend chosen measurably changes outcomes. Using an evaluation where success means at least one correct document ID with zero false IDs, the reported false-positive rates fall with added conversational turns; for example, XGrammar on the Llama model drops from 3.08% at zero turns to 0.31% at two turns. The ranking between backends shifts with model and turn count: LM Format Enforcer leads in several zero-turn settings, while Outlines and XGrammar show the largest gains from added turns. The paper estimates that at a scale of ten million chunks and one hundred thousand queries, swapping LM Format Enforcer for XGrammar could add thousands of missed references in some settings, implying that decoding choice is a first-order lever in RAG deployments.","pith_inferences":["Because each turn condition uses a different sample subset (750, 375, and 250 samples), a matched-subject study on identical queries would be needed to separate the effect of added turns from dataset composition; the paper's turn comparisons conflate the two.","The regex-based ID extraction is the measurement bottleneck; testing alternative parsers or human annotation would tell whether the backend rankings are stable or partly artifacts of parsing failures.","A practical extension implied by the results is to tune the number of in-context exemplar turns jointly with the decoding backend, since the interaction between them appears to matter more than either alone.","The consistently high judge scores suggest content quality is roughly constant across backends, so the actionable surface for improving RAG reliability may be reference fidelity rather than answer quality."],"forward_implications":["Multi-turn prompting substantially reduces false-positive citation rates for all three backends, with errors dropping from hundreds to single digits in the reported runs.","The best backend depends on the model and the turn condition, so no single guided decoding method dominates in every RAG setting.","In a large-scale RAG deployment of ten million chunks and one hundred thousand queries, backend choice can change the count of missed references by thousands, meaning default decoding setups risk grounding errors.","Generation-time costs are comparable across backends for each model, so the decision between backends is primarily about accuracy rather than speed.","Semantic answer quality stays high across all methods, concentrating the measurable differences in reference fidelity rather than in general output quality."],"supporting_citations":[{"why":"Introduces the retrieval-augmented generation paradigm that the paper's evaluation scenario builds on.","marker":"[1]"},{"why":"Motivates the need for user-centered structured output constraints in LLM applications.","marker":"[2]"},{"why":"Proposes the Outlines backend, one of the three guided decoding methods compared in the study.","marker":"[3]"},{"why":"Proposes XGrammar, the second backend, which the paper notes is the production default in its setup.","marker":"[4]"},{"why":"Provides the LM Format Enforcer backend used as the third comparator in the experiments.","marker":"[5]"},{"why":"Supports the description of Outlines' finite-state machine token validity mapping in constant-time terms.","marker":"[8]"},{"why":"Cited for the limitation that Outlines does not accommodate beam search or batched generation.","marker":"[9]"},{"why":"Cited for the limitation that partial JSON handling can yield suboptimal outputs when slight deviations are acceptable.","marker":"[10]"}],"fun_headline_variants":["Backend choice in guided decoding can cost thousands of RAG citations","Turn count and decoder backend shift RAG reference accuracy","RAG citation errors sensitive to guided decoding backend and turns","Swap decoder backend to lose thousands of correct RAG references"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the regex that extracts document IDs from model responses works equally accurately for all three decoding backends and all turn conditions, so differences in false-positive rates reflect the model's citation behavior rather than differences in how the backends format IDs.","fun_headline_variants_meta":{"raw":{"variants":["Backend choice in guided decoding can cost thousands of RAG citations","Turn count and decoder backend shift RAG reference accuracy","RAG citation errors sensitive to guided decoding backend and turns","Swap decoder backend to lose thousands of correct RAG references"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1414,"prompt_tokens":877,"completion_tokens":537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":493,"tokens_out":537,"duration_ms":5373,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:14:05.601988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three backends through Algorithm 1 but replace the regex extraction with an independent parser or human annotation on the 507 publicly available responses; if the backend rankings of false-positive rates change or compress, the reported differences are extraction artifacts rather than genuine decoding effects.","supporting_citations":[{"cited_title":"rag ctx: \\ ctx\\ query: \\ q\\","cited_arxiv_id":null,"evidence_quote":"Introduces the retrieval-augmented generation paradigm that the paper's evaluation scenario builds on."}],"review_version":2}