{"id":"63595ef8-5e94-42da-9794-6754ec8b13fb","arxiv_id":"2501.13831","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A target-phrase-only edit representation offers the best accuracy-versus-output-length trade-off for LLM-based ASR post editing, closing 50-60% of the WER gap to full rewriting while losing only 10-20% of the length savings.","lead":"This paper tests two compact 'edit phrase' formats that let a language model rewrite ASR transcripts while decoding fewer output tokens. The best format, target-phrase-only, recovers most of the full-rewrite model's accuracy gains while keeping most of the speed benefit of the prior span-based method on LibriSpeech.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target-only expansion tie-break is unablated and test recovery is unreported, so the 50-60% WER-gap closure may not be robust to the chosen matching heuristic.","rationale":"The reader's weakest_assumption identifies the tie-break rule as the most fragile point, and I agree. The target-only representation is only useful if the deterministic expansion reliably locates the intended source span. The paper's recovery-rate metric is a step in the right direction, but it is computed on the dev reference compressions, not on the actual autoregressive model outputs evaluated on test. Since the final WER is the downstream quantity of interest, a wrong tie-break in a small fraction of test examples could shift the reported 50-60% gap closure. The absence of any tie-break ablation means we cannot distinguish the contribution of the representation from the contribution of the matching heuristic. I considered the alternative concern that efficiency is measured by output length rather than wall-clock latency; while this limits practical impact, the paper explicitly defines its efficiency metric as output length reduction rate, so the internal efficiency claim is coherent. The lack of code/data is a reproducibility limitation rather than a correctness flaw. Therefore the tie-break/recovery-rate issue is the single most load-bearing concern for the empirical accuracy claim. The reader already conditioned on this weakness, so the verdict remains CONDITIONAL; no adjustment is needed.","tokens_in":9051,"tokens_out":11987,"duration_ms":108333,"concrete_test":"Compute the target-only k=3 recovery rate on LibriSpeech test-clean and test-other using the paper's Levenshtein-based compression and expansion functions. Then, holding the frozen PaLM-2 model outputs fixed, re-run the expander with two alternative tie-break rules (e.g., rightmost-first and longest-anchor-first) and recompute WER. If test recovery stays above 99.5% and WER changes by less than 0.1 absolute under both alternatives, the tie-break is unlikely to drive the headline result. If recovery drops or WER varies by 0.2 or more, the reported gap-closure claim is not robust to the expansion heuristic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that target-only has the best efficiency-accuracy trade-off depends on the deterministic expansion described in Section II-C. This expansion uses a hand-chosen 'leftmost and closest' tie-break when the anchor context matches the source in multiple places. The paper reports a 99.8% recovery rate on dev for k=3 (Table III), but this is measured on the reference compression, not on model predictions, and is not verified on the LibriSpeech test sets. If the tie-break selects the wrong anchor span on test, the expander either applies the edit at the wrong location or silently drops it, which would inflate WER relative to an ideal expansion. No ablation of alternative tie-break rules (rightmost, longest-anchor, etc.) is provided, so the reported 50-60% closure of the WER gap between span and full may be an artifact of this one heuristic rather than a property of the target-only representation itself. The WER numbers in Table II are internally consistent, but without test-set recovery rates and tie-break ablations we cannot tell how much of the gap closure is due to the representation versus the matching rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two compact edit-representation schemes for LLM-based rewriting applied to ASR post editing: a phrase-pair representation (source phrase plus target phrase with dilation context) and a target-phrase-only representation (dilated target phrase with anchor context, located by deterministic string matching). Both are compared against the edit-span representation of Kaneko and Okazaki (2023) and against a full-rewrite baseline. The authors fine-tune PaLM 2 Gecko and Otter models on LibriSpeech, freeze the USM ASR model, and report WER and decoder output length on test-clean and test-other. They claim that the target-phrase-only representation offers the best efficiency-accuracy trade-off, closing 50-60% of the WER gap between span and full rewrite while losing only 10-20% of the span model's length reduction rate. The paper also reports dev-set recovery rates for the expansion step and discusses limitations, including the use of edit-distance-based phrases and uniform dilation spans.","tokens_in":9255,"tokens_out":5997,"duration_ms":53764,"significance":"If the trade-off claim holds, the target-only representation is a practical way to reduce autoregressive decoding cost for ASR post editing with modest WER degradation, and the paper would be a useful extension of the edit-span idea to phrasal, more natural edit formats. The arithmetic in Table II is internally consistent: the 57% and 54% WER-gap closures and the 12.5% and 22.2% length-reduction losses follow directly from the reported numbers. The paper also merits credit for selecting k on dev and reporting test WER and length, for comparing two model sizes, and for including an explicit limitations section. However, the central efficiency claim is supported only by output-length reduction, not by measured latency or throughput, and the deterministic expansion heuristic is not ablated, so the robustness of the reported gap closure is not fully established.","major_comments":[{"comment":"The efficiency axis is decoder output length reduction, but the abstract and conclusion frame the contribution in terms of computational cost ('decoding cost still increases with output length') and even 'slowing down decoding by 10-20%'. No wall-clock latency, throughput, or FLOP measurements are reported. Since the Limitations section states that the target-only expansion is 'more algorithmically involved' than the other two compact representations, the real end-to-end speedup may not be proportional to output length. Please report measured latency/throughput for the full pipeline, or explicitly restrict the efficiency claims to output-length reduction.","section":"Section III-A; Table II"},{"comment":"The abstract claims the method 'loses only 10-20% of the length reduction rate of the edit span model.' On test-other, the reported numbers are span = 5 tokens (-72%) and target-only = 8 tokens (-56%), a relative loss of (72-56)/72 = 22.2%, which the main text itself states. The abstract should be corrected or qualified (e.g., '10-20% on test-clean').","section":"Abstract; Section III; Table II"},{"comment":"The reported recovery rate is computed on the reference compression C(x,y) from the dev set, not on LLM-predicted edits, and no test-set recovery rate is given. The WER numbers on test are end-to-end outcomes and therefore already include any expansion failures, but they do not separate LLM prediction error from expansion-rule error. Please report recovery rates on predicted edits for test-clean and test-other, and ablate the leftmost-and-closest tie-break rule (e.g., rightmost match, longest-anchor match, or drop-on-ambiguity) to show that the 50-60% gap closure is a property of the target-only representation rather than of one hand-chosen matching heuristic.","section":"Section III-B; Table III; Section II-C"},{"comment":"The expansion function E for the target-only representation is under-specified. The paper says only that string matching and replacement 'deals with discontiguous dilation spans' and gives one example with a leftmost-and-closest tie-break, but it does not define the matching algorithm, the priority order of the two tie-break criteria, or the behavior when the predicted anchor context appears multiple times or not at all. This makes the method difficult to reproduce and leaves a free parameter unexamined. Please provide a precise algorithm or pseudocode for E.","section":"Section II-C; Eq. (4)"}],"minor_comments":[{"comment":"The percentages in parentheses mix bases: WER reductions are relative to USM, while output length reductions are relative to the full model. Please state the reference value for each percentage in the caption or use a consistent notation.","section":"Table II caption"},{"comment":"The legend labels the series as 'target only (k=1,2,3,4)' but the individual points are not annotated with their k values; since the claim that k=3 is best is load-bearing, please label the points or provide a table with the exact WER/length coordinates.","section":"Figure 3"},{"comment":"The phrase 'leftmost and closest pair' is not precisely defined: in the example, both candidate anchor matches start at the same leftmost position, so the reader cannot tell how the two criteria are ordered. Please define the tie-break rule explicitly.","section":"Section II-C"},{"comment":"The abstract says the paper 'systematically compare[s]' representations, but the experiments cover only ASR post editing (one task) with two model sizes; consider qualifying 'systematically' or adding a sentence in the introduction noting that the comparison is task-specific.","section":"Abstract; Section I"},{"comment":"Table III reports dev-set recovery only; please clarify whether the same recovery behavior was observed on the test sets, or rename the table/heading to indicate that it is dev-only, to avoid implying test-set recovery.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The paper uses proprietary PaLM 2 and USM models and does not release code or data, which limits reproducibility of the expansion algorithm beyond the textual description. The abstract's '10-20%' claim is directly contradicted by the paper's own test-other number (22.2%), which should be fixed before publication. The core idea is a modest extension of Kaneko and Okazaki's span representation, but the reported Pareto dominance of target-only over phrase pair is a useful empirical result if the efficiency claim is backed by latency measurements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you care about compact edit representations for LLM rewriting. The target-only representation — replacing numeric span indices with natural-language anchor phrases — is a simple and sensible idea, and the authors evaluate it carefully against the span baseline and a full-rewrite baseline on LibriSpeech. The arithmetic holds up: the abstract's 50-60% WER-gap closure and 10-20% length-reduction loss are direct from Table II. The paper is clearly written, the expansion functions are defined precisely, and the Limitations section is candid about edit-distance alignment and uniform dilation.\n\nThe soft spots are real but mostly minor. The efficiency claim is about output length, not measured latency. That's standard in this line of work, but it means the '70% cost cut' is an estimate, not a benchmark. There are no error bars or significance tests; with 5-point WER differences and no variance, I'd want a sanity check on whether the target-only vs phrase-pair ordering is stable. The bigger question is the unablated tie-break in the target-only expansion. The stress-test concern is fair in principle — leftmost-and-closest is a heuristic — but the paper reports a 99.8% recovery rate on dev for k=3, which directly measures how often the gold compression survives expansion. That makes the tie-break a minor worry, not a load-bearing flaw. It would still be worth a one-line ablation or a test-set recovery number.\n\nThe lack of code and data is a real limitation at this venue; the models are proprietary, so nobody can reproduce the WER numbers independently. That is a moderate blow to confidence, not a reason to reject.\n\nOverall, this is a modest, honest contribution. The target-only representation is new, the comparison is systematic, and the central claim is consistent with the reported numbers. I'd send it to reviewers — they'll want the latency measurement and a tie-break ablation, but the core idea deserves a proper hearing.","headline":"A clean, modest paper: the target-phrase-only edit representation is a genuine improvement over numeric span indices for LLM-based ASR post editing, and the reported numbers are consistent.","tokens_in":9778,"tokens_out":2523,"would_cite":true,"duration_ms":22567,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A target-phrase-only edit representation gives LLM-based ASR post-editing its best efficiency-accuracy trade-off, closing 50-60% of the WER gap between span edits and full rewrites while losing only 10-20% of the length reduction.","keywords":["ASR post editing","compact edit representations","target-phrase-only representation","edit span representation","LLM rewriting","decoding efficiency","output length reduction","word error rate"],"falsifier":"A direct check is to re-expand the same decoded target-only outputs on the LibriSpeech test sets using a rightmost-match tie-break instead of the leftmost-and-closest rule and compare WER; if the 50-60% gap closure mostly disappears, the reported trade-off is an artifact of the tie-break choice rather than of the representation.","tokens_in":8847,"feed_emoji":"✂️","tokens_out":8110,"duration_ms":66548,"temperature":0.7,"pith_summary":"The paper tries to show that an LLM can correct ASR transcripts accurately and cheaply if, instead of rewriting the whole transcript, it only writes the replacement phrases, each surrounded by a few anchor words copied from the input. The compact representation is built from a Levenshtein alignment between the ASR output and the reference, and the LLM is fine-tuned to predict the compressed edit string. On the LibriSpeech test sets, the target-phrase-only representation reduces the average decoded output length by about 56-70 percent while closing 54-57 percent of the word-error-rate gap between the very short edit-span representation and the full-rewrite model. This matters because LLM decoding cost grows with the number of generated tokens, so a shorter output directly means cheaper and faster post-editing.","feed_headline":"Target-only edits close half the ASR accuracy gap","feed_subtitle":"Phrase-only rewrites with anchor context cut LLM decoding length by 56-70% while keeping most of the WER gain.","key_machinery":"The load-bearing object is a compressed edit string built by the compression function $C$ and expanded by the expansion function $E$, which together satisfy $E(x, C(x,y)) = y$. For each edit, the string contains a target phrase extended left and right by $k$ anchor tokens copied from the input, prefixed by a prompt word like 'rewrite'. The anchor tokens are matched against the input during expansion, and the target phrase replaces the matched span; with $k=3$ the anchors are long enough to identify the correct span in almost all dev examples. The dilation parameter $k$ is the control knob that trades output length against ambiguity, and $k=3$ is the setting the paper identifies as the best trade-off across two model sizes.","core_discovery":"The central claim is that the target-phrase-only edit string, with dilation span $k=3$, offers the best point on the efficiency-accuracy curve for LLM-based ASR post-editing. Each edit is written as a target phrase with three words of context on each side; those context words act as anchors that are matched against the input, and the matched span is replaced. The paper reports that this representation recovers the intended rewrite for 99.8% of dev examples, and on the LibriSpeech test sets it lands between the span and full-rewrite extremes: WER of 3.0 on test-clean and 6.8 on test-other, compared with 2.7 and 6.2 for full rewrites and 3.4 and 7.5 for span edits, with average output lengths of 6 and 8 tokens versus 20 and 18 for full rewrites. That is what the 50-60% gap closure and 10-20% length-loss numbers describe.","pith_inferences":["The paper does not ablate the leftmost-and-closest tie-break rule used when anchor words match the input in more than one place, so the reported 50-60% gap closure is conditional on that rule; varying the tie-break is the first testable check.","The same compression-expansion framework could be applied to other high-overlap rewriting tasks, such as grammatical error correction or style transfer, and the recovery-rate table suggests the dilation size would need to be tuned per task.","A natural next experiment is asymmetric dilation, giving more anchor context on the left than on the right, since left context may be more informative for a left-to-right decoder; the paper notes this possibility but does not test it."],"forward_implications":["LLM-based ASR post-editing can run with about a third to under half of the decoded output length of a full rewrite while keeping most of the WER gain.","The same representation works for both smaller and larger decoder-only LLMs, since the trade-off curve is consistent for the two model sizes tested.","Because the expansion step is a deterministic string-matching and replacement algorithm, the post-editing system does not need a second model to decode or reconstruct the final transcript.","The phrase-pair representation recovers the intended rewrite in 99.98% of dev examples, the highest recovery rate among the phrasal representations, making it the safer choice when reconstruction fidelity is the priority."],"supporting_citations":[{"why":"Introduces the edit span representation that this paper uses as the most efficient baseline and whose WER gap it aims to close.","marker":"[1]"},{"why":"Provides the phrase-pair idea from statistical machine translation that inspires the new source-target and target-only representations.","marker":"[10]"},{"why":"Supplies the frozen USM ASR model whose transcriptions are the input to the post-editing LLM.","marker":"[12]"},{"why":"Provides the LibriSpeech training, dev, and test data used for fine-tuning and evaluation.","marker":"[13]"},{"why":"Describes the PaLM 2 decoder-only LLMs that are fine-tuned for the rewrite task.","marker":"[5]"}],"fun_headline_variants":["Phrase-target edits trim LLM output, preserve ASR accuracy","Target-only rewrite strings halve the WER gap for ASR","Compact edit spans: 56-70% shorter, half the accuracy loss","Anchor-context phrasal edits: efficient ASR post-editing","Shorter edits close 50-60% of ASR rewrite error gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the anchor words copied around a target phrase will identify the one place in the transcript where that phrase should be applied; the paper's expansion stage enforces this with a leftmost-and-closest tie-break when several places match, and the reported gains depend on that rule picking correctly.","fun_headline_variants_meta":{"raw":{"variants":["Phrase-target edits trim LLM output, preserve ASR accuracy","Target-only rewrite strings halve the WER gap for ASR","Compact edit spans: 56-70% shorter, half the accuracy loss","Anchor-context phrasal edits: efficient ASR post-editing","Shorter edits close 50-60% of ASR rewrite error gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1565,"prompt_tokens":961,"completion_tokens":604,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":508}},"tokens_in":577,"tokens_out":604,"duration_ms":5678,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:33:59.908072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to re-expand the same decoded target-only outputs on the LibriSpeech test sets using a rightmost-match tie-break instead of the leftmost-and-closest rule and compare WER; if the 50-60% gap closure mostly disappears, the reported trade-off is an artifact of the tie-break choice rather than of the representation.","supporting_citations":[{"cited_title":"Reducing sequence length by predicting edit spans with large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces the edit span representation that this paper uses as the most efficient baseline and whose WER gap it aims to close."},{"cited_title":"Statistical phrase-based translation,","cited_arxiv_id":null,"evidence_quote":"Provides the phrase-pair idea from statistical machine translation that inspires the new source-target and target-only representations."}],"review_version":1}