{"id":"b7ffca5c-7d72-42f0-82a2-36b104085d65","arxiv_id":"2601.16946","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LogitMatch constrains an LLM's decoding so that every span it outputs is an exact substring of the input, modestly improving matching-based span labeling on some tasks.","lead":"The paper categorizes how large language models label spans of text (tagging, indexing, matching) and introduces LogitMatch, a decoding method that forces generated spans to be exact substrings of the input. It evaluates the strategies on four tasks and finds tagging is the most robust baseline, with LogitMatch giving small, task-dependent gains over matching.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run, no-error-bar results leave the claimed LogitMatch improvement over matching baselines statistically unsupported; Table 3 differences are mostly 0.1–2.7 F1 points.","rationale":"The reader's weakest_assumption focused on tokenization handling in §4.3, which is a legitimate concern given the paper's own Limitations note about severe tokenization issues for the tagging variant. However, the most load-bearing problem for the central claim is statistical: Table 3 reports tiny, often negligible differences from a single seed, so the claimed improvement is not robustly demonstrated. The reader's rationale does mention the lack of seeds and error bars, but they did not elevate this to the weakest_assumption field. I agree with the CONDITIONAL verdict: the taxonomy and LogitMatch method are valuable and clearly described, but the empirical support for the headline improvement needs replication and statistical validation before it can be taken as strong evidence. The tokenization concern, while real, is partially mitigated by the fact that LogitMatch is tested on three open models across diverse tokenizers and shows near-zero span-matching errors in Figure 2; a definitive failure would require identifying a specific tokenizer mismatch, which is not demonstrated in the paper. Thus the statistical reliability is the stronger basis for the conditional verdict.","tokens_in":16718,"tokens_out":4050,"duration_ms":40505,"concrete_test":"Re-run the key comparisons (LOGITMATCH vs MATCH, and LOGITMATCH-OCC vs MATCH-OCC) on NER, GEC, and ESA-MT with at least 10 random seeds for Qwen3-8B and Llama-3.3-70B. Compute mean hard F1 with 95% bootstrap confidence intervals and a paired test across examples (e.g., paired bootstrap or Wilcoxon signed-rank). If the confidence intervals overlap zero or p ≥ 0.05, the claim that LogitMatch improves over matching baselines is not supported by the current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LogitMatch improves upon matching-based methods rests on differences in Table 3 that are small (0.1–2.7 F1 points) and sometimes negative (e.g., NER-70B: Match 72.7 vs LogitMatch 72.4; CPL often tied). Appendix A states that all experiments were performed as a single run with seed 0, and decoding uses temperature >0 (e.g., 0.6 for Qwen3-8B and Llama-3.3-70B), so sampling noise is non-trivial. Without multiple seeds, confidence intervals, or significance tests, these differences are indistinguishable from noise. The paper's own Figure 2 shows LogitMatch reduces span-matching errors to zero, but this does not translate into a decisive F1 gain; the mechanism works, but the claimed empirical advantage is not established. The tokenization handling in §4.3 is a genuine fragility, but the statistical weakness is more directly load-bearing because it undermines the headline comparison even under perfect tokenization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper surveys prompting strategies for span labeling with LLMs, organizing them into tagging, indexing, and matching families, and proposes LogitMatch, a constrained-decoding method that restricts the generated 'text' field to tokens drawn from valid input spans. The method is evaluated on NER, GEC, machine-translation error detection, and a synthetic conditional pattern lookup task, using three open-weight LLMs plus GPT-5-mini for methods that do not require logit access. The central claim is that LogitMatch eliminates span-matching errors of matching-based approaches and improves over competitive matching baselines in some setups, while tagging remains a robust baseline.","tokens_in":16911,"tokens_out":4654,"duration_ms":48941,"significance":"The taxonomy of span-labeling strategies is a useful organizing contribution, and LogitMatch is a simple, training-free, and reproducible method that addresses a genuine failure mode of matching-based pipelines: models' inability to copy input spans exactly. The paper ships code, evaluates across four tasks and several multilingual datasets, and provides an error decomposition that goes beyond overall F1. The qualitative example in Table 4 clearly illustrates the tokenization mismatch problem. If the empirical advantage over plain matching were statistically supported, this would be a solid and practical contribution to LLM-based span labeling.","major_comments":[{"comment":"All experiments are single runs at seed 0, with sampling temperatures above zero (up to 1.0 for GPT-5-mini). The reported F1 differences between LogitMatch and MATCH are mostly 0.1–2.7 points and sometimes negative (e.g., NER-70B: 72.4 vs. 72.7; ESA-MT-70B: 9.2 vs. 9.1; several CPL entries tied). With no multiple seeds, confidence intervals, or significance tests, the abstract's claim that LogitMatch 'improves upon competitive matching-based methods' is not statistically supported. Please add repeated runs with different seeds, report variance, and apply a significance test (e.g., paired bootstrap). If the difference is not significant, the claim should be softened to state that LogitMatch eliminates span-matching errors while performing within noise of matching baselines.","section":"§5.3, Appendix A, Table 3"},{"comment":"LogitMatch's central guarantee depends on the tokenization-handling rules: whitelisting prefix tokens and quote tokens at field boundaries. Algorithm 1 is explicitly a simplification that 'does not account for tokenization,' and the Limitations state that a similar constrained approach for tagging 'ran into severe issues with tokenization handling.' The paper does not provide a systematic evaluation of tokenizer-edge-case failures on the multilingual NER/ESA-MT data, such as byte-level BPE where the opening quote is fused with the following span token, or whitespace normalization across scripts. Because the headline property is that every decoded span is a valid input span, please add a robustness analysis or formal argument that the whitelist logic is complete for the tested tokenizers; otherwise the method's generality is not fully established.","section":"§4.3, Algorithm 1, Limitations"}],"minor_comments":[{"comment":"The paper uses both 'occurence_index' and 'occurrence' for the same field; please standardize the spelling.","section":"§3.3 and Table 7"},{"comment":"'analogically' should be 'analogously'.","section":"§5.1"},{"comment":"'hypothetize' should be 'hypothesize'.","section":"§6"},{"comment":"The algorithm uses X = [x1, ..., xn] as 'tokens' in V_select, but Section 2 defines X as characters. Please clarify the relationship between character-level task definition and token-level constrained decoding.","section":"Algorithm 1"},{"comment":"The heading 'our experiments with Qwen-8B' should be 'Qwen3-8B' for consistency with the model name used elsewhere.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The statistical fragility of the headline result is the main concern; the method itself is plausible and the taxonomy is useful. A revision that adds multiple seeds and significance testing, plus a tokenizer-robustness analysis, would make the claims defensible. I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look for the method, not for the numbers. LogitMatch is a clean, well-specified constrained decoding trick that guarantees the model's output spans are exact substrings of the input. That fixes a real failure mode — LLMs normalize punctuation and capitalization when copying — and the qualitative example in Table 4 shows it working. The taxonomy of tagging/indexing/matching is a reasonable synthesis of what people have been doing ad hoc, and the authors are honest about what they didn't solve, including tokenization snags that killed the tagging variant. That's good engineering reporting.\n\nWhat's weak is the evidence for the central claim. Every number in Table 3 comes from a single run at seed 0, with temperature above zero for the two larger models. The differences between LogitMatch and plain Match are mostly fractions of an F1 point — a few are slightly negative. Without error bars or repeated seeds, those numbers are indistinguishable from sampling noise. The method does eliminate span-matching errors (Figure 2), but that doesn't translate into a decisive F1 win, which is fine, but the abstract's 'improves upon' is not supported by the data as presented.\n\nThe tokenization handling is the other soft spot. The authors warn in Limitations that the tagging variant ran into severe tokenization issues, so the matching variant's correctness across tokenizers is not guaranteed. They handle the cases they saw, but it's the kind of thing that could break on byte-level BPE. Minor: the code should be pinned to a commit.\n\nBottom line: the taxonomy and the method deserve serious peer review. The experiments need multiple seeds or at least variance estimates before the improvement claim is credible. I'd send it to a main conference or a workshop with a request for that revision.","headline":"LogitMatch is a real engineering contribution with a useful taxonomy, but the single-run experiments leave the headline improvement over matching within the noise.","tokens_in":17434,"tokens_out":2853,"would_cite":true,"duration_ms":30936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"LogitMatch, a decoding-time filter, forces LLM span labels to be exact substrings of the input, removing a whole class of alignment errors without retraining.","keywords":["span labeling","constrained decoding","large language models","named entity recognition","grammatical error correction","logit manipulation","structured output","JSON decoding"],"falsifier":"Run LogitMatch on an input that starts a span with a byte-level BPE token that merges the opening quote with the first span character (e.g. a fused token like “London”). If the whitelist logic does not recognize this fused token as the beginning of a span, LogitMatch will reject the valid span or emit malformed JSON; a high failure rate on such inputs would show that the alignment guarantee only holds for favorable tokenizers.","tokens_in":16582,"feed_emoji":"🎯","tokens_out":3529,"duration_ms":36552,"temperature":0.7,"pith_summary":"Generative LLMs cannot naturally point at parts of their input, so span-labeling systems rely on ad hoc prompts. The paper sorts those prompts into three families — tagging the text, indexing positions, and matching span content — and shows the main weakness of matching: models often 'fix' the text they copy, so predicted spans do not align with the input. It introduces LogitMatch, a constrained decoding method that restricts the model's output tokens so every decoded span is necessarily a contiguous span from the input. Evaluated on named entity recognition, grammatical error correction, machine-translation error detection, and a synthetic pattern-lookup task, LogitMatch removes span-copy errors and beats other matching methods in several settings, while tagging remains a reliable overall baseline.","feed_headline":"LogitMatch locks LLM spans to the exact input text","feed_subtitle":"A decoding-time filter removes span-copy errors, making matching-based labeling competitive with tagging.","key_machinery":"LogitMatch is a logit-level constrained decoding procedure with three modes: DEFAULT (normal generation), SELECT (vocabulary limited to input tokens, so the span starts with a real input token), and COPY (only the next input token or a closing quote is allowed, so the generated text can only be extended along the input). The tokenization handler whitelists prefix tokens and boundary quote tokens so that different input/output tokenizations do not accidentally forbid valid spans. This machinery converts the model's span output into a guarantee: every decoded span is a substring of the input, eliminating the need for fuzzy post-hoc matching.","core_discovery":"On the paper's own terms, the central claim is that span-content matching fails not because matching is a bad idea but because unconstrained LLMs do not faithfully copy input substrings. LogitMatch makes the copy step guaranteed by intervening only at decoding: while the model generates the text field of a JSON span, the vocabulary is pruned to tokens appearing in the input, and continuation is limited to the next input token or the closing quote. This preserves the model's token-efficiency and does not need fine-tuning. The paper reports that LogitMatch improves over vanilla matching on NER and GEC, where input text uses standard NLP tokenization, and occasionally matches or exceeds tagging","pith_inferences":["Editorial inference: LogitMatch could in principle be extended to tagging strategies by constraining the entire copied text, but the paper reports that tokenization at tag boundaries caused severe issues and was abandoned; a working version would let tagging gain the same alignment guarantee.","Editorial inference: the method does nothing about label hallucination or category errors, so its ceiling is set by the model's semantic judgment; combining it with reasoning-trace generation might push both alignment and accuracy, at a token-cost trade-off the paper's reasoning experiments already hint at.","Editorial inference: a natural stress test is to apply LogitMatch to byte-level BPE inputs where opening quotes fuse with the first span token; the paper's whitelisting logic is the fragile piece and would need to handle such fused tokens without rejecting valid spans."],"forward_implications":["If LogitMatch works as reported, matching-based span labeling becomes reliable on tasks with non-canonical input tokenization, where models previously tended to normalize punctuation and casing.","Because it only touches decoding, the method transfers to any locally deployable LLM without fine-tuning or architecture changes.","LogitMatch can be combined with structured-output constraints and an occurrence-index field, covering repeated spans that plain content matching cannot disambiguate.","The approach requires logit access, so it is not directly applicable to closed API models, where the paper notes it could not be evaluated.","On tasks like GEC, tagging still wins, suggesting the ideal method depends on average span length and how naturally tags wrap around errors."],"fun_headline_variants":["Decoding trick makes LLM span copying exact","LogitMatch: stop LLM span drift at decode","Prune vocab to input tokens for reliable spans","Span labeling: force matches, not guesses"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The tokenization whitelisting in LogitMatch correctly handles every input tokenizer and model, so that no valid span is ever forbidden and no malformed JSON is produced; the paper's own limitations section notes that a similar constrained approach for tagging ran into severe tokenization issues.","fun_headline_variants_meta":{"raw":{"variants":["Decoding trick makes LLM span copying exact","LogitMatch: stop LLM span drift at decode","Prune vocab to input tokens for reliable spans","Span labeling: force matches, not guesses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2314,"prompt_tokens":655,"completion_tokens":1659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":1599}},"tokens_in":399,"tokens_out":1659,"duration_ms":14506,"temperature":1.0,"reasoning_tokens":1599,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:24:08.792082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LogitMatch on an input that starts a span with a byte-level BPE token that merges the opening quote with the first span character (e.g. a fused token like “London”). If the whitelist logic does not recognize this fused token as the beginning of a span, LogitMatch will reject the valid span or emit malformed JSON; a high failure rate on such inputs would show that the alignment guarantee only holds for favorable tokenizers.","supporting_citations":[],"review_version":1}