{"id":"6aba59a7-1024-48fe-88a8-4b0848282df2","arxiv_id":"2501.12979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FlanEC maps five-candidate ASR lists to corrected transcripts and gets its best average word error rate (8.5%) from a 3B Flan-T5 model trained on all HyPoradise domains with full fine-tuning.","lead":"This paper fine-tunes three sizes of Flan-T5 to repair speech-recognition mistakes by rewriting the top five candidate transcriptions into one clean sentence. The best results come from the largest model trained on all eight benchmark domains at once with full fine-tuning, though the reported gains are small and the paper does not include error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main ranking may be within test-set noise: no confidence intervals or significance tests support the 8.5% vs 9.0% CD-over-SD and FT-over-LoRA differences in Table 2.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing threat to the central claim is not the benchmark's external validity but the internal validity of the reported rankings. The headline result (FlanEC-CD FT 3B, 8.5% average WER) is separated from the closest alternative (FlanEC-SD FT 3B, 9.0%) by only 0.5 macro-WER points, and several per-subset differences are under 0.3 points. With a single run per condition and no confidence intervals or significance tests, these differences could easily be sampling noise. A bootstrap or randomization test on utterance-level WERs would resolve this directly and is a standard, cheap check. The missing N-best T5 baseline is also a real omission, but the paper's own comparative claims about cumulative training and full fine-tuning are more central; if those differences are not statistically reliable, the main contributions are weakened even without introducing new baselines. Therefore the conditional verdict stands, pending the significance analysis.","tokens_in":11376,"tokens_out":8716,"duration_ms":80724,"concrete_test":"Compute bootstrap 95% confidence intervals for the per-subset test WER of each condition in Table 2 by resampling test utterances (1000 replicates), and run a paired approximate-randomization test comparing FlanEC-CD FT 3B vs FlanEC-SD FT 3B and FT vs LoRA for each subset and for the macro-average. Also retrain at least the 3B CD and SD FT conditions with 3 different seeds. If the 95% CI of the macro-average difference includes 0, or fewer than 6 of 8 subsets show significance (p<0.05), the 'generally outperforms' and 'best overall' claims should be weakened to 'numerically better in this benchmark run'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claims rest on single-run WER numbers in Table 2 with no error bars, confidence intervals, or significance tests. The headline 3B FlanEC-CD FT average of 8.5% is only 0.5 macro-WER points below the 3B FlanEC-SD FT average of 9.0%; per-subset differences are often fractions of a point (WSJ 2.6 vs 2.3; CV-accent 11.7 vs 11.1; Tedlium actually favors SD, 4.0 vs 4.2). Similarly, FT vs LoRA at 3B CD differs by 0.6 average (8.5 vs 9.1) with ties or near-ties on several subsets. Since each test WER is computed on a finite utterance sample and no seed or variance information is reported, it is plausible that several of these differences are within sampling noise. If so, the claims that CD 'generally outperforms' SD and that FT 'consistently shows better performance' are not established, and the model's status as 'best-performing' is unverified. Section 5 uses these rankings as the basis for all conclusions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FlanEC, an encoder-decoder post-ASR error-correction model built on Flan-T5, and evaluates it on Task 1 of the GenSEC Grand Challenge using the HyPoradise benchmark. It compares single-dataset (SD) and cumulative-dataset (CD) training, full fine-tuning (FT) versus LoRA, and model scales from 250M to 3B parameters, reporting WER on eight ASR domains. The headline result is that the 3B FlanEC-CD model with full fine-tuning attains the best average WER (8.5%), that CD training generally helps in the fully fine-tuned setting, and that full fine-tuning generally beats LoRA except at the smallest scale in SD training. The paper also reports zero-shot Flan-T5 and ChatGPT in-context-learning baselines and honestly notes that no model improves over the baseline on the CORAAL subset.","tokens_in":11596,"tokens_out":8872,"duration_ms":84051,"significance":"If the comparative findings hold, the paper provides useful engineering evidence for the GenSEC community: a single instruction-tuned encoder-decoder model trained on combined domains can match or beat per-domain specialist models, and LoRA is not a full substitute for full fine-tuning at larger scales. The work has concrete strengths: it uses a public benchmark, releases code for reproducibility, evaluates multiple model sizes and adaptation schemes, and reports negative results on CORAAL rather than selecting around them. The main weakness is statistical: all comparative conclusions rest on single-run WER values with no error bars, confidence intervals, or significance tests, and several headline gaps are small relative to the resolution of the metric. The stress-test concern about test-set noise therefore lands: the central claim of consistent superiority of CD over SD and FT over LoRA is plausible but not established at the reported resolution.","major_comments":[{"comment":"The sentence 'full fine-tuning (FT) outperforms LoRA across all datasets' is contradicted by Table 2 in the CD setting. For example, at 250M, ATIS LoRA gives 2.7 versus FT 3.0 and LRS2 LoRA gives 10.0 versus FT 10.6; at 800M, LRS2 LoRA gives 9.3 versus FT 9.6 and Tedlium-3 is a tie at 4.3. Please revise this to 'outperforms LoRA on most datasets' or provide a corrected quantitative statement that matches the table.","section":"Section 5, 'LoRA vs. Full Fine-tuning'"},{"comment":"The reported 'average improvement of 0.64 ± 0.46 absolute WER points' is not defined. If the ± value is the standard deviation across the eight subsets, it does not by itself convey statistical significance; if it is a standard error or confidence interval, the computation should be stated. In either case, the paired comparison and its test statistic should be reported so that the reader can assess whether the FT advantage is robust.","section":"Section 5, 'LoRA vs. Full Fine-tuning'"}],"minor_comments":[{"comment":"Please report the LoRA rank and alpha; stating that adapters are added to all fully connected layers is not sufficient for reproducibility.","section":"Section 4.2"},{"comment":"Please state how the validation split is constructed and how 'best model based on validation WER' interacts with the different epoch budgets for SD (10) and CD (2); otherwise the SD/CD comparison is not fully specified.","section":"Section 4.2"},{"comment":"There are spacing typos in the dataset names ('A TIS', 'T edlium-3'), and the caption should state that the 'Overall' row is an unweighted macro-average over the eight subsets.","section":"Table 1"},{"comment":"The prompt template does not include the confidence scores that the HyPoradise n-best lists provide, although Section 1 mentions associated confidence scores; please clarify whether confidence scores were used as model input.","section":"Section 3.1"},{"comment":"The abstract claims improvements in grammaticality and linguistic correctness, but the evaluation reports only WER; either add a text-quality or grammaticality metric or soften the language to match the reported evaluation.","section":"Abstract and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper builds heavily on the HyPoradise benchmark, which is co-authored by one of the current authors, and several citations are to the same group. This is not a correctness issue, but the editor may wish to ensure that the paper explicitly positions itself relative to prior published GenSEC results on the same benchmark rather than only comparing to the ASR baseline. The manuscript is also a published IEEE SLT workshop paper; the journal submission should clarify what is added beyond the workshop version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: FlanEC is a clean, limited-scope empirical paper. The new content is a systematic grid over Flan-T5 at three scales (250M, 800M, 3B), single-dataset (SD) vs cumulative-dataset (CD) training, and LoRA vs full fine-tuning, evaluated on HyPoradise. That grid is not in the cited literature, and the paper ships code and uses an external benchmark with fixed splits. Credit where due: the setup is straightforward, the negative result on CORAAL is reported rather than buried, and the headline direction—larger models help, full fine-tuning beats LoRA at the 3B scale, cumulative training helps a bit—is consistent across most subsets, not driven by one outlier.\n\nSoft spots, in proportion. The stress-test note is right on the main weakness: Table 2 reports single-run WERs with no seeds, error bars, or significance tests. Several contrasts are close: 3B CD FT vs SD FT is 8.5 vs 9.0 average, with Tedlium favoring SD; FT vs LoRA at 3B CD is 8.5 vs 9.1, with ties on LRS2 and CORAAL. So the claims that CD 'generally outperforms' SD and FT 'consistently shows better performance' are plausible but not firmly established. That said, the authors mostly hedge with 'generally' and 'on average,' and they do report a mean FT-vs-LoRA difference of 0.64 ± 0.46 WER points in the CD setting, so they are not overstating the effect size. The bigger omission is the N-best T5 baseline from [23]; it is a direct constrained-decoding alternative and its absence weakens the argument against decoding constraints.\n\nMild self-citation is present, but HyPoradise is an external public benchmark and the outcomes are not forced, so I would not call that a substantive flaw.\n\nWho is this for? Practitioners building ASR post-processors and people benchmarking instruction-tuned encoder-decoder models. It is not a breakthrough, but it is a usable recipe and a solid datapoint. A serious referee should engage with it. The main revision requests would be a couple of training seeds or a variance note, and the missing baseline. I would not block acceptance on the lack of significance testing alone at this venue level.\n\nRecommendation: accept to peer review. If you are the editor, ask for the N-best T5 baseline and at least a sentence about run-to-run variance. The core ranking is credible.","headline":"A clean, reproducible sweep of Flan-T5 for post-ASR correction; the main rankings are plausible but single-run WERs leave the headline FT-over-LoRA and CD-over-SD claims thinner than they look.","tokens_in":12149,"tokens_out":3766,"would_cite":true,"duration_ms":37540,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single Flan-T5 model fine-tuned on all eight HyPoradise domains with full parameter updates achieves the best overall post-ASR correction, reaching an average word error rate of 8.5 percent.","keywords":["Post-ASR error correction","Generative speech error correction","Flan-T5","HyPoradise","Encoder-decoder model","LoRA","Full fine-tuning","Word error rate"],"falsifier":"Train the same FlanEC-CD 3B model on HyPoradise but evaluate on n-best lists produced by a different ASR engine (e.g., a modern Whisper or an RNN-T model) across the same domains; if the cumulative model no longer beats per-domain models, or full fine-tuning no longer beats LoRA, the paper's central ranking fails to generalize.","tokens_in":11175,"feed_emoji":"🎙️","tokens_out":4354,"duration_ms":38775,"temperature":0.7,"pith_summary":"This paper tries to establish that a single instruction-tuned encoder-decoder model, Flan-T5, can serve as a general post-ASR error corrector across diverse speech domains. The authors train FlanEC at 250M, 800M, and 3B parameter scales on the HyPoradise benchmark, either on each domain separately or on all eight domains combined. They report that the 3B model trained on the combined data with full fine-tuning reaches an average word error rate of 8.5%, the best overall, and that cumulative training plus full fine-tuning generally beats per-domain training and LoRA-style adapters. The paper also finds that zero-shot Flan-T5 fails while fine-tuned FlanEC beats ChatGPT in-context learning, so task-specific parameter updates carry the result.","feed_headline":"One Flan-T5 model fixes ASR errors to 8.5% average WER","feed_subtitle":"Mixing all eight training domains and updating all parameters beats per-domain models and cheap adapters.","key_machinery":"The mechanism is the instruction-tuned encoder-decoder Flan-T5 adapted with a fixed natural-language prompt that maps an n-best list of ASR hypotheses into a single corrected transcription. The training design contrasts per-domain (SD) versus cumulative-dataset (CD) fine-tuning, and LoRA adapters versus full fine-tuning, at three model sizes; the key quantity is the word error rate on the HyPoradise test subsets, especially the proportion of reference tokens absent from all n-best hypotheses (new tokens), which motivates unconstrained decoding.","core_discovery":"The central discovery is that cumulative training works: a single Flan-T5 model fine-tuned on all eight HyPoradise subsets jointly outperforms models trained on each subset individually, and full fine-tuning consistently beats LoRA once data is pooled. The 3B FlanEC-CD model with full fine-tuning tops 4 of 8 subsets and reaches the lowest average WER of 8.5%, versus 9.0% for its LoRA counterpart, while the 250M SD-LoRA models beat their FT peers on most subsets. The paper interprets this as evidence that large parameter scale and diverse training data let the model exploit shared linguistic knowledge, and that overfitting from full fine-tuning is a small-model, small-data phenomenon.","pith_inferences":["The success of cumulative training suggests that a single error-correction model could be shipped as a universal ASR post-processor, updated once on pooled data rather than maintained per domain.","The paper's comparison to ChatGPT ICL is indirect (no Flan-T5 ICL); a fairer test with an encoder-decoder ICL variant or a decoder-only model of matched scale would clarify whether the gap comes from architecture or fine-tuning.","A testable extension is to measure whether cumulative training helps most on domains with overlapping vocabulary or error patterns; the paper's new-token analysis (Table 1) implies CORAAL's failures stem from n-best coverage, which could be probed by injecting oracle hypotheses."],"forward_implications":["A single FlanEC-CD 3B model can serve as a general post-ASR corrector across WSJ, ATIS, CHiME-4, Tedlium-3, CV-accent, SwitchBoard, LRS2, and CORAAL, lowering average WER from the 11.8% baseline to 8.5%.","Cumulative training should be preferred over per-domain training for GenSEC when data from multiple domains is available, since CD matches or beats SD on most datasets.","Full fine-tuning, not parameter-efficient adapters, is the right choice for larger models and pooled data; LoRA only helps small models trained on single datasets where overfitting threatens.","Task-specific fine-tuning is necessary: zero-shot Flan-T5 does not beat the baseline, while fine-tuned FlanEC outperforms ChatGPT with in-context learning.","The CORAAL subset remains unsolved; no model reaches the ASR baseline there, and the paper attributes this to reference tokens missing from all n-best hypotheses in over 90% of sentences."],"supporting_citations":[{"why":"Supplies the HyPoradise dataset, the benchmark, the per-subset ASR baselines, and the train/test splits that all FlanEC models are evaluated on.","marker":"[7]"},{"why":"Provides the Flan-T5 base model and the instruction-following design that FlanEC builds on.","marker":"[5]"},{"why":"Introduces LoRA, the parameter-efficient adaptation method compared against full fine-tuning.","marker":"[8]"},{"why":"Motivates the choice of n=5 hypotheses and presents the constrained decoding-space baseline that the paper deliberately avoids.","marker":"[23]"},{"why":"Supports the decision not to evaluate Flan-T5 with in-context learning, citing that encoder-decoder models struggle with ICL.","marker":"[34]"},{"why":"Supports the general preference for parameter adaptation over in-context learning that motivates fine-tuning FlanEC.","marker":"[20]"}],"fun_headline_variants":["Flan-T5 pooled data beats per-domain models","Single Flan-T5 model hits 8.5% WER with all data","All-in-one Flan-T5: full fine-tune tops 8.5% WER","Pooled data and full fine-tuning lower WER to 8.5%","Flan-T5 unifies ASR correction, reaches 8.5% WER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes the HyPoradise benchmark faithfully represents real post-ASR correction: its fixed n-best lists, references, and splits come from one ASR setup, and if its error patterns or domains are not representative of other ASR systems, the reported advantages of cumulative training and full fine-tuning may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Flan-T5 pooled data beats per-domain models","Single Flan-T5 model hits 8.5% WER with all data","All-in-one Flan-T5: full fine-tune tops 8.5% WER","Pooled data and full fine-tuning lower WER to 8.5%","Flan-T5 unifies ASR correction, reaches 8.5% WER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3352,"prompt_tokens":857,"completion_tokens":2495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":2388}},"tokens_in":473,"tokens_out":2495,"duration_ms":17489,"temperature":1.0,"reasoning_tokens":2388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:33:31.960067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same FlanEC-CD 3B model on HyPoradise but evaluate on n-best lists produced by a different ASR engine (e.g., a modern Whisper or an RNN-T model) across the same domains; if the cumulative model no longer beats per-domain models, or full fine-tuning no longer beats LoRA, the paper's central ranking fails to generalize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HyPoradise dataset, the benchmark, the per-subset ASR baselines, and the train/test splits that all FlanEC models are evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Flan-T5 base model and the instruction-following design that FlanEC builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces LoRA, the parameter-efficient adaptation method compared against full fine-tuning."},{"cited_title":"The CoNLL-2014 shared task on grammatical error correction,","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of n=5 hypotheses and presents the constrained decoding-space baseline that the paper deliberately avoids."},{"cited_title":"Few-shot fine-tuning vs. in-context learning: A fair comparison and evalua- tion,","cited_arxiv_id":null,"evidence_quote":"Supports the decision not to evaluate Flan-T5 with in-context learning, citing that encoder-decoder models struggle with ICL."},{"cited_title":"CoEdIT: Text editing by task-specific instruc- tion tuning,","cited_arxiv_id":null,"evidence_quote":"Supports the general preference for parameter adaptation over in-context learning that motivates fine-tuning FlanEC."}],"review_version":1}