{"id":"06b77d5e-0212-4503-9530-d081fcaede06","arxiv_id":"2501.07818","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Freezing the decoder during fine-tuning maintains or improves multilingual and natural-language-generation performance, and a larger frozen decoder recovers structured and QA performance.","lead":"This paper tests what happens when the decoder of an encoder-decoder language model is kept frozen while only the encoder is fine-tuned. It reports that this recipe helps multilingual and text-generation tasks, while a larger frozen decoder is needed to recover structured and question-answering performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The structured/QA 'larger frozen decoder' claim is confounded: no trainable AlexaTM 2B baseline, so gains on MTOP/SQuAD/XNLI may come from larger encoder or pretraining, not frozen decoder capacity.","rationale":"The reader's weakest_assumption exactly identifies the absent trainable 2B baseline as the load-bearing gap, and I agree. The paper's headline 'However' contribution is the broader task-type viability of frozen decoders, supported only by a larger-model frozen run with no same-scale trainable control. This is a concrete experimental confound rather than a mere lack of consensus. I do not think it warrants REJECT because the natural-language and multilingual claims have within-scale comparisons that are substantial and plausible, and because a single missing baseline is fixable. CONDITIONAL remains the correct verdict until the baseline is run. I also note the paper's internal inconsistency between 'AlexaTM 2B' and 'AlexaTM 3B' in Section 3.3, but the missing baseline is the primary issue. My concrete test directly settles the attribution question: if trainable 2B performs the same as frozen 2B, the structured/QA conclusion collapses to model scale; if it performs worse, the frozen-decoder mechanism gains support. Either outcome is informative, so the check is well-defined and feasible with the same infrastructure already used in the paper.","tokens_in":11881,"tokens_out":3279,"duration_ms":34324,"concrete_test":"Fine-tune AlexaTM 2B with a trainable decoder on MTOP, SQuAD, and XNLI under identical hyperparameters to the frozen run (batch size 128, learning rate 1e-6, linear decay to 5e-6, 100k updates, num_beams=3), and compare against the reported AlexaTM-2B (frozen) numbers. If 2B trainable matches or beats 2B frozen, the structured/QA claim reduces to model scale rather than decoder freezing. If 2B trainable is worse, the claim survives. Additionally, run at least three seeds for each configuration to assess whether the 2-point EM differences are within run-to-run noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's differentiating claim is that pairing a frozen decoder with a larger model 'can effectively maintain or even enhance performance in structured and QA tasks.' The only support is Table 3 (MTOP, SQuAD) and Table 5 (XNLI), where AlexaTM-2B (frozen) beats AlexaTM (trainable decoder) by 2 EM points on MTOP, about 2 EM on SQuAD, and about 18 accuracy points on XNLI. But Section 3.1 introduces AlexaTM 2B only as 'We keep the decoder here frozen too'; there is no measurement of AlexaTM 2B with a trainable decoder. Because the larger model differs in both encoder and decoder capacity, and possibly in pretraining data or recipe, the frozen-decoder-specific contribution is unidentified. Without the trainable 2B baseline, the 'However' conclusion is not established. The natural-language and multilingual-forgetting results are less affected, since they compare frozen versus trainable at the same 511M scale, but they also lack variance information and repeated-seed reporting. The central claim therefore rests on an unmeasured comparison, and the paper itself does not flag this missing baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether freezing the decoder of an encoder-decoder language model during fine-tuning is an effective parameter-efficient strategy. Using AlexaTM (511M) and a larger AlexaTM 2B model, it compares trainable decoder, frozen decoder at the same scale, and frozen decoder with the larger model across single-task and mixed-task settings on NLG (XSUM, WebNLG, CommonGen), semantic parsing/QA (MTOP, SQuAD), multilingual intent classification (MASSIVE), and XNLI. The authors report that freezing the decoder preserves or improves performance on natural-language-output tasks, mitigates catastrophic forgetting in multilingual tasks, and that a larger frozen decoder can retain or improve structured and QA performance. The paper does not provide code, checkpoints, repeated-run statistics, or a trainable 2B baseline.","tokens_in":12077,"tokens_out":3675,"duration_ms":36737,"significance":"If fully supported, the central claim would be practically useful: frozen decoders could reduce fine-tuning and deployment cost while preserving generation quality and preventing multilingual forgetting. The task coverage is broad and the inclusion of both single-task and mixed-task conditions is a strength. The strongest evidence is the same-scale MASSIVE comparison, where the frozen 511M decoder clearly outperforms the trainable 511M decoder across nearly all languages. The weakest part is the larger-decoder claim: because no trainable AlexaTM 2B condition is run, the observed gains on MTOP, SQuAD, and XNLI cannot be attributed to decoder capacity. The paper is an empirical report without reproducibility artifacts or error analysis, so its conclusions rest entirely on the presented tables.","major_comments":[{"comment":"The abstract and conclusion claim that 'pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks,' but this is not established. The only larger-model condition is introduced in §3.1 as 'We keep the decoder here frozen too,' with no trainable AlexaTM 2B baseline. Differences between AlexaTM-2B (frozen) and AlexaTM (trainable) on MTOP, SQuAD, and XNLI confound decoder size with encoder size, parameter count, and possibly pretraining scale/recipe. The conclusion that decoder capacity drives the improvement requires a trainable 2B baseline, or a reformulation of the claim to refer to 'a larger frozen model' rather than 'a larger frozen decoder.'","section":"§3.1, Tables 3 and 5"},{"comment":"No variance information or repeated-seed experiments are reported, and checkpoint selection is inconsistent: §3.3 says checkpoints were 'saved either based on the perplexity or exact match scores over the validation set.' It is not stated which criterion was used for each task or run. Without error bars or multiple runs, differences such as the 2-point exact-match gains on MTOP and SQuAD in Table 3 and small ROUGE differences in Table 2 are not interpretable as reliable effects.","section":"§3.3 and all result tables"},{"comment":"The prose misreports the table for WebNLG and XSUM. It says 'When fine-tuned on WebNLG, freezing results in a 2% drop in performance,' but Table 2 shows AlexaTM (frozen) with BLEU 32.01 compared with AlexaTM 29.76, i.e., improvement, and later text says 'freezing improves the performance in the single task setting.' For XSUM, the claim that performance 'is improved with the larger frozen decoder' is also not supported: AlexaTM-2B (frozen) ties or slightly trails the trainable AlexaTM on ROUGE-1, ROUGE-L, and BLEU. These contradictions make the analysis unreliable.","section":"§4.1 and Table 2"},{"comment":"The XNLI discussion is unclear and partly contradicted by Table 5. The column headings 'Accuracy (3)' and 'Accuracy (20)' are undefined; if these refer to decoding beam sizes, that should be stated explicitly. The text says 'Freezing the parameters improves performance,' but in the mixed setting AlexaTM-mix (84.29) outperforms AlexaTM-mix (frozen) (83.91), so the claim holds only for the single-task comparison. The additional statement that increasing beams from 3 to 5, 10, and 20 decreases the gap is not supported by any table, since only 3 and 20 are reported.","section":"§4.1 and Table 5"}],"minor_comments":[{"comment":"The learning-rate schedule is stated as 'a learning rate of 10 6 and a linear decay to lr = 5x106 over 100k updates,' which appears to contain missing superscripts or exponents; please correct the notation so the schedule is unambiguous.","section":"§3.3"},{"comment":"The paper repeatedly writes 'ROGUE' where it means 'ROUGE' (e.g., '3 scores are computed – the ROGUE scores'), and the metric names should be normalized throughout.","section":"§4.1"},{"comment":"For MTOP, the text describes 'almost a 14% drop' and later 'a 2% increase' from the larger frozen decoder, but the exact-match numbers (0.66, 0.52, 0.68) correspond to relative drops/gains of about 21% and 31% respectively from the frozen baseline; please clarify whether percentages are absolute or relative.","section":"Table 3 and §4.1"},{"comment":"The sentence 'Performances on mix tasks can be highly variable being dictated by the dataset proportion and the choice of other datasets' is grammatically incomplete; a connector such as 'and can be' would improve clarity.","section":"§6 Future Work"}],"recommendation":"major_revision","confidential_remarks":"The paper's exploratory results could be made publishable by adding a trainable AlexaTM 2B baseline or by rewriting the structured/QA claim to avoid attributing gains to decoder capacity. The missing baseline is not a minor omission; it directly undermines a headline conclusion. I would also ask the authors to provide variance or repeated runs for at least the core comparisons, and to reconcile the textual claims with Table 2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading if you care about parameter-efficient fine-tuning. The useful new thing here is the breadth: one model family, seven task types, single-task and mixed fine-tuning, and a clean demonstration that freezing the decoder of AlexaTM gives large multilingual-forgetting gains on MASSIVE (8–10% across non-English languages). That result is consistent with the earlier machine-translation finding by Cooper Stickland et al. and is a solid practical data point. The paper also reports the same-scale MTOP drop honestly, and the cross-task interference observations are fine.\n\nThe soft spot is the one the abstract leans on. The claim that a larger frozen decoder 'maintains or even enhances' structured and QA performance is supported only by the 2B-frozen versus 511M-trainable comparison. The 2B model was never trained with a trainable decoder, so size and frozenness are completely confounded. The gains on MTOP, SQuAD, and XNLI could be due to encoder capacity or pretraining differences. That doesn't sink the natural-language or multilingual results, which compare frozen and trainable at the same scale, but it does mean the 'However' sentence should be scaled back until the control exists.\n\nAlso missing: repeated seeds, error bars, and code or data. Checkpoint selection switches between perplexity and exact match, which is a small but real consistency issue. There is a typo in Section 3.3 where the 2B model is called '3B.'\n\nIf I were the editor, I would send this to reviewers. The core forgetting result is likely to survive a rework, and the task-level mapping is useful. But I'd ask for a trainable-2B baseline, variance reporting, and released artifacts before believing the broader claim.","headline":"Useful breadth on frozen-decoder fine-tuning, but the 2B 'freezing recovers structured/QA' claim is missing the trainable-2B control that would support it.","tokens_in":12581,"tokens_out":2779,"would_cite":false,"duration_ms":26661,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Freezing a decoder during fine-tuning is not merely a cost saver: it preserves generation quality, prevents multilingual forgetting, and, with a larger decoder, improves structured and QA performance.","keywords":["frozen decoder","parameter-efficient fine-tuning","catastrophic forgetting","multilingual","natural language generation","semantic parsing","question answering","AlexaTM"],"falsifier":"Train the same 2B AlexaTM with a trainable decoder on SQuAD and MTOP and compare exact-match scores; if the trainable 2B model reaches or exceeds the frozen-decoder scores, the claim that freezing rather than model scale drives the structured-task gains is falsified.","tokens_in":11638,"feed_emoji":"🧊","tokens_out":11316,"duration_ms":100613,"temperature":0.7,"pith_summary":"This paper argues that in fine-tuning a pretrained encoder-decoder language model, you can freeze the decoder and train only the encoder without paying a performance penalty, and in several settings you actually gain. The experiments cover seven task types on AlexaTM, a multilingual 511M-parameter model, plus a larger 2B variant with a frozen decoder. The results show that frozen decoders match or improve natural-language generation tasks, and they sharply reduce catastrophic forgetting when a model fine-tuned on English is evaluated on other languages. For structured and QA outputs, the same-size frozen decoder loses ground, but the larger frozen decoder recovers or exceeds the fully trainable baseline. The paper therefore proposes freezing as a viable parameter-efficient fine-tuning strategy, especially when one shared decoder serves multiple task-specific encoders.","feed_headline":"Freezing decoders curbs catastrophic forgetting in multilingual models","feed_subtitle":"With a larger frozen decoder, the same trick also lifts structured and QA scores.","key_machinery":"The central object is the frozen decoder: a pretrained decoder whose weights stay fixed throughout fine-tuning, so gradients never touch it and only encoder parameters are updated. The paper pairs this with a size contrast, testing a 511M-parameter AlexaTM with a 12-layer encoder and 12-layer decoder against a 2B-parameter version with the decoder also frozen, to see whether extra decoder capacity can compensate for the loss of trainability. The argument is carried by three stacked configurations on each task, a trainable decoder, a same-size frozen decoder, and a larger frozen decoder, evaluated on single-task and mixed-task fine-tuning with beam search. The title's implied architecture, a set of task-specific encoders feeding a single shared frozen decoder, is the deployment consequence the paper motivates but does not directly build.","core_discovery":"The paper's central claim is that the decoder of a pretrained encoder-decoder model can be frozen during task fine-tuning without losing downstream performance, and that freezing is not just a way to save computation. On natural-language generation tasks, the frozen decoder matches or slightly improves the fully trainable baseline: WebNLG BLEU rises from 29.76 to 32.01, and the 2B frozen decoder reaches 35.14, while XSUM ROUGE-1 falls from 0.39 to 0.37 and recovers most of the gap with the larger decoder. On multilingual intent classification (MASSIVE), freezing lifts non-English accuracy by 8 to 10 percentage points over the trainable model, evidence that keeping the decoder fixed prevents catastrophic forgetting of pretrained multilingual knowledge. On the structured and QA tasks where the same-size frozen decoder loses ground, such as MTOP exact match falling from 66% to 52%, the larger 2B frozen decoder overcomes the loss, scoring 68% on MTOP and 74.15% exact match on SQuAD versus 72.33% with the fully trainable model. The paper concludes that frozen decoders are a viable parameter-efficient strategy across task types, particularly when tasks with similar output formats share one decoder.","pith_inferences":["Extending the paper, the shared-decoder deployment suggested by the title can be tested directly by training several small task-specific encoders against one frozen decoder and measuring per-task accuracy and throughput.","The paper's results imply a testable predictor: the benefit of decoder freezing should track how similar a task's target sequences are to the decoder's pretraining text, measurable by the frozen decoder's perplexity on target strings before fine-tuning.","The MASSIVE gains suggest that freezing acts as a regularizer on the shared multilingual representation, which implies that freezing other modules such as the embedding layer could yield similar catastrophic-forgetting relief, although the paper does not test this.","The mixed-task numbers, with MTOP exact match falling to 28%, indicate that task grouping rather than freezing alone will determine multi-task success; a natural follow-up is to cluster tasks by output format and freeze one decoder per cluster."],"forward_implications":["Fine-tuning only the encoder is a viable default for tasks with natural-language targets, since WebNLG BLEU rises and CommonGen scores stay flat when the decoder is frozen.","Freezing the decoder is a concrete remedy for catastrophic forgetting in multilingual assistants, with non-English MASSIVE intent accuracy climbing 8 to 10 points over the trainable baseline while English also improves.","Structured and QA tasks can keep pace with fully trainable models if the frozen decoder is larger, as MTOP and SQuAD both post their best exact-match scores with the 2B frozen decoder.","Mixed-task fine-tuning still suffers cross-task interference, so the practical win is to group tasks with similar output formats and give each group a frozen decoder."],"supporting_citations":[{"why":"Prior result that freezing a decoder can improve machine translation, which directly motivates freezing on generation tasks.","marker":"Cooper Stickland et al. (2021)"},{"why":"mBART multilingual denoising pretraining, the architecture and pretraining style that AlexaTM follows.","marker":"Liu et al. (2020)"},{"why":"The larger 2B model used to test whether extra frozen decoder capacity recovers structured and QA performance.","marker":"Soltan et al. (2022)"},{"why":"AlexaTM pretraining and distillation, the base 511M model for all fine-tuning experiments.","marker":"FitzGerald et al. (2022a)"},{"why":"MASSIVE, the 51-language intent-classification dataset whose non-English results support the catastrophic-forgetting claim.","marker":"FitzGerald et al. (2022b)"},{"why":"MTOP, the multilingual semantic-parsing benchmark that shows the frozen-decoder drop and the larger-model recovery.","marker":"Li et al. (2021)"},{"why":"SQuAD v2, the reading-comprehension benchmark where the larger frozen decoder reaches the best exact-match score.","marker":"Rajpurkar et al. (2018)"}],"fun_headline_variants":["Frozen decoders beat full training on multilingual tasks","Bigger frozen decoder lifts QA and structured scores","Freeze the decoder, keep performance, cut compute costs","Frozen decoder: better multilingual, equal or better elsewhere","Freezing decoders: a surprising win for QA and multilingual"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2B model's gains come from its larger frozen decoder, because the paper never trains a 2B model with a trainable decoder; if the larger encoder or different pretraining is actually responsible, the claim that a bigger frozen decoder rescues structured tasks collapses.","fun_headline_variants_meta":{"raw":{"variants":["Frozen decoders beat full training on multilingual tasks","Bigger frozen decoder lifts QA and structured scores","Freeze the decoder, keep performance, cut compute costs","Frozen decoder: better multilingual, equal or better elsewhere","Freezing decoders: a surprising win for QA and multilingual"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3110,"prompt_tokens":914,"completion_tokens":2196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2117}},"tokens_in":530,"tokens_out":2196,"duration_ms":14402,"temperature":1.0,"reasoning_tokens":2117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:34:31.581865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 2B AlexaTM with a trainable decoder on SQuAD and MTOP and compare exact-match scores; if the trainable 2B model reaches or exceeds the frozen-decoder scores, the claim that freezing rather than model scale drives the structured-task gains is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior result that freezing a decoder can improve machine translation, which directly motivates freezing on generation tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The larger 2B model used to test whether extra frozen decoder capacity recovers structured and QA performance."}],"review_version":1}