{"id":"eb9a225a-db6d-4ddb-9bf1-3b7e40ad4a71","arxiv_id":"2412.12674","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Feed-forward LoRA with more parameters adapts Llama-3.2-1B to Icelandic better than attention LoRA, bottleneck adapters, IA3, or prefix tuning, and final-layer placement mitigates context-length degradation.","lead":"This paper tests several parameter-efficient fine-tuning methods to adapt a 1-billion-parameter language model to Icelandic using plain text, then measures how well the adapted models summarize Icelandic news. It reports that more trainable parameters help, that adapters placed in the feed-forward layers work best, and that training only the last layers protects the model's ability to handle longer contexts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unexamined overlap between CC100 adaptation data and RÚV Radio News evaluation data could make the headline PEFT ranking reflect memorization rather than language adaptation.","rationale":"I agree with the Reader that single-run automatic-metric comparisons are a real limitation, but the most load-bearing unanswered question is decontamination. The central claim depends on the RRN scores measuring generalized Icelandic generation; if the adaptation corpus contains the evaluation articles, those scores measure memorization, and every headline conclusion (more parameters better, feed-forward LoRA best, prefix tuning unsuitable, CCNet better than IGC) could be an artifact of capacity differences in memorization. This is a concrete, falsifiable concern: an overlap analysis plus a deduplicated rerun would resolve it. The paper's shared code and adapters, and the monotonic within-method trends, are genuine supporting evidence, but they do not address corpus-level leakage. The Reader's condition on variance/human evaluation remains appropriate; adding the overlap check strengthens the condition without changing the verdict.","tokens_in":10629,"tokens_out":9358,"duration_ms":91473,"concrete_test":"Compute exact and fuzzy overlap (e.g., MinHash or 13-gram containment) between (a) the 250k CC100 adaptation chunks and the IGC subsets used for training and (b) the RRN main/intro evaluation instances. If any overlap is found, retrain the central comparisons (LoRA-ff-256 vs baseline, and CCNet vs IGC) after removing overlapping articles from training/evaluation, or evaluate on a held-out split with no overlap, and check whether the reported margins persist; if no overlap is found, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 adapts on 250k chunks of CCNet-filtered CC100 Icelandic text, and Section 2.4 evaluates on RÚV Radio News (RRN) summarization. CC100 is built from Common Crawl and RRN is a news corpus from Iceland's main broadcaster, so the two domains are plausibly overlapping; the paper reports no decontamination or n-gram overlap analysis. With a causal-LM objective, a chunk that contains an RRN main/intro pair can memorize the reference summary. If such chunks are in the adaptation sample, high-capacity methods (LoRA-FF-256, bottleneck reduction 4) would be favored not because they adapt the language better, but because they memorize more of the evaluation set; prefix tuning's 30-token prefix cannot memorize at that scale, which would explain its 'not suitable' label. Table 4's CCNet-over-IGC result is also vulnerable if CCNet contains the news domain and IGC does not. Because the abstract's guidance is meant to generalize beyond this corpus, this missing check is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of parameter-efficient fine-tuning (PEFT) methods for adapting Llama-3.2-1B-Instruct to Icelandic using unstructured text. The author compares LoRA in attention and feed-forward modules across several ranks, bottleneck adapters, (IA)3, and prefix tuning, and evaluates 0-shot, 1-shot, and 5-shot abstractive summarization on the RÚV Radio News dataset with BERTScore and ROUGE-L. The main findings are that more trainable parameters help within a method family, feed-forward LoRA is the best configuration, bottleneck adapters are second, attention-placed LoRA is weaker for its parameter count, prefix tuning and (IA)3 are unsuitable, and restricting adapters to the final layers mitigates degradation at longer contexts. The code, prompt generator, and trained adapters are released.","tokens_in":10715,"tokens_out":7832,"duration_ms":66855,"significance":"If the results hold, the paper provides practical, actionable guidance for language adaptation of small instruction-tuned LLMs: place LoRA in feed-forward layers, provide sufficient learning capacity, avoid prefix tuning for unstructured-text adaptation, and consider final-layer-only adaptation to preserve long-context abilities. The study is carefully structured, with parameter-count comparisons, three targeted ablations (module placement, layer selection, and corpus choice), and a consistent evaluation protocol. The release of code and adapters is a concrete strength, as is the use of both a web-crawled and a curated adaptation corpus. The main caveats are that all conclusions rest on single-run point estimates and on automatic summarization metrics for one language, one model, and one task, and that potential overlap between the adaptation corpus and the evaluation data is not examined.","major_comments":[{"comment":"The adaptation sample in §2.2 consists of 250,000 random chunks from the Icelandic CC100 corpus after CCNet filtering, and the evaluation in §2.4 is abstractive summarization on RÚV Radio News (RRN). CC100 is built from Common Crawl and RRN is Icelandic broadcaster news content, so RRN articles may be present in the adaptation sample. If a training chunk contains an RRN main body followed by the reference introduction, the causal-language-modeling objective can memorize the reference summary, and high-capacity configurations such as LoRA-ff-256 and Bottlen.-4 would be rewarded for memorization rather than for language adaptation; the 30-token prefix-tuned model could not memorize at the same scale, which would artificially strengthen the 'prefix tuning is not suitable' conclusion. The same concern applies to the CCNet-versus-IGC comparison in Table 4 if CCNet contains the news domain and IGC does not. The manuscript reports no decontamination or n-gram overlap analysis, so this missing check is load-bearing for the headline ranking and for the abstract's general guidance.","section":"§2.2, §2.4"},{"comment":"Every result in the four tables is a single point estimate from one training run, with no standard deviations, seeds, or significance tests. Several comparisons that support the 'more trainable parameters is better' claim differ by less than one BERTScore point or less than one ROUGE-L point, e.g., LoRA-ff-128 versus LoRA-ff-256 in 1-shot (69.10/13.86 versus 69.06/13.89) and Bottlen.-16 versus Bottlen.-4 in 0-shot (63.33/8.38 versus 63.78/8.15). Without variance estimates or paired significance testing, the ranking of nearby configurations is not established, and the unconditional wording of the abstract ('are not suitable') is stronger than the evidence supports. Multiple seeds for at least the main configurations, or another variance-aware analysis, would make the ranking defensible.","section":"Tables 1–4"},{"comment":"The Limitations section explicitly states that automatic summarization metrics are 'questioned' and that human evaluation is needed, yet the abstract and Section 3.1 present the ranking as conclusive for language adaptation on the basis of BERTScore and ROUGE-L on a single task. The central claim is prescriptive for practitioners, so the paper should either add convergent evidence (for example, a human evaluation or a second task on a subset of the main configurations) or qualify the conclusions as preliminary. This is not a demand for full evaluation of every ablation, but the strength of the headline claims should match the strength of the evidence.","section":"Limitations"}],"minor_comments":[{"comment":"There is a typo: 'we use use α = 2r' should be 'we use α = 2r'.","section":"§2.3"},{"comment":"The sentence 'Moreover, it is slightly better than LoRA in both the attention and the feed-forward modules' is ambiguous; it should say that feed-forward LoRA slightly outperforms LoRA applied to both modules.","section":"§3.2"},{"comment":"The x-axis labels are dense and the excluded methods (prefix tuning and (IA)3) are not shown; add a note or a supplementary table with their parameter counts so the reader can see why they are excluded.","section":"Figure 1"},{"comment":"The note 'Self-attention (qv) LoRA rank 32' should appear in the table caption rather than below the table body, since it is needed to interpret the numbers.","section":"Table 3"},{"comment":"The sentence 'we do not test on any task where high-quality generation is important but on text summarisation' is potentially confusing: summarization is itself a generation task; the intended contrast is that the evaluation may reward copying rather than language quality. Rephrase for clarity.","section":"§3.4"},{"comment":"'the no adapters model' should be 'the no-adapter model'.","section":"Footnote 3"},{"comment":"Prefix tuning is tested with a single prefix length of 30; a short prefix-length sweep would strengthen the 'not suitable' conclusion, since prefix length directly controls the method's capacity.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a well-executed empirical case study with reproducible artifacts, but the headline claims are somewhat stronger than the single-run, single-task, single-corpus evidence supports. The most serious concern is the possible overlap between the CC100 adaptation data and the RÚV Radio News evaluation data; if overlap exists, the PEFT ranking could partially reflect memorization. I would require a decontamination analysis and some form of variance evidence before publication, but I do not see this as a rejection: the concerns are addressable within the manuscript's scope, and the released code and adapters make the checks feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on low-resource language adaptation or PEFT placement. The paper is a single-author case study adapting Llama-3.2-1B-Instruct to Icelandic with unstructured text, and it does a genuinely systematic job: LoRA in attention vs. feed-forward vs. both, bottleneck adapters, IA3, prefix tuning, rank sweeps, layer-exclusion ablations, and a corpus-quality comparison (CCNet vs. Icelandic Gigaword). The headline finding—feed-forward LoRA beats attention LoRA, and higher rank helps—is clearly presented and consistent across the reported tables. The final-layer-only result for mitigating context-length degradation is interesting and not something I had seen tested for this setup. The author also shares code and adapters, and is upfront in the Limitations section about the reliance on automatic summarization metrics and the need for human evaluation. That transparency earns real credit.\n\nSoft spots are real but proportionate. Every result is a single run. No seeds, no variance, no significance testing, so the ranking of methods could shift with noise; the margin between LoRA-ff-128 and LoRA-ff-256 in 1-shot is about 0.1 BERTScore, which is basically nothing without repeats. The evaluation is one task (RÚV Radio News summarization) with BERTScore/ROUGE-L, and the paper's own Limitations section concedes these metrics are questioned for news summaries. The stress-test note about possible overlap between CC100 adaptation data and RÚV news evaluation data is a fair concern, though the paper does not report decontamination; I checked the text and there is no n-gram overlap analysis or contamination check. That matters because a causal-LM objective could memorize reference summaries from the same domain, favoring higher-capacity adapters. I would not call it load-bearing flaw, because the pattern of results (feed-forward > attention, more params > fewer) is consistent across setups and the 0-shot gains also appear with the curated IGC corpus, which is less likely to contain RÚV text. But the concern is legitimate and the author should at least report overlap statistics or cite that CC100 and RRN are disjoint.\n\nAnother soft spot: the comparison between CCNet and IGC is confounded by domain and size (IGC is undersampled to 250k chunks from different subsets), so Table 4's conclusion that \"no positive effects from higher-quality data\" is weaker than stated. Still, the author acknowledges this.\n\nOverall: this is a solid empirical contribution for practitioners, not a breakthrough. It deserves a serious referee. I would accept it for peer review, with the main request being repeated runs with confidence intervals, a contamination check, and preferably at least one additional language or task to test generality. I would cite it if I were doing PEFT language adaptation work, and I might bring it to reading group as a good example of transparent ablations with a clear practical takeaway.","headline":"A careful, honest ablation study of PEFT placement for Icelandic adaptation of a 1B model; the single-run evaluation is the main soft spot, but the findings are plausible and worth refereeing.","tokens_in":11307,"tokens_out":732,"would_cite":true,"duration_ms":8128,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Increasing trainable parameters improves language adaptation, but only when placed in feed-forward layers; prefix tuning and (IA)3 hurt.","keywords":["language adaptation","parameter-efficient fine-tuning","LoRA","Icelandic","low-resource languages","text summarization","prefix tuning","bottleneck adapters"],"falsifier":"Rerun the full ablation with five random seeds and a human preference study on the generated summaries; a reversal in the ranking, say attention LoRA matching or beating feed-forward LoRA, would falsify the paper's central claim.","tokens_in":1622,"feed_emoji":"🇮🇸","tokens_out":3624,"duration_ms":137950,"temperature":0.7,"pith_summary":"This paper tries to establish how to adapt a small instruction-tuned language model to a new language using only unstructured text, without wrecking the abilities it already has. On Llama-3.2-1B-Instruct adapted to Icelandic and evaluated on news summarisation, it claims that more trainable parameters consistently improve language adaptation, and that where those parameters are placed matters as much as how many there are: LoRA in the feed-forward layers is best, bottleneck adapters come second, LoRA in attention is weaker for its parameter budget, and prefix tuning and (IA)3 are unsuitable. The paper also claims that some setups degrade performance on longer contexts, and that restricting adaptation to the final layers mitigates this. A sympathetic reader would care because these are practical choices for adapting small models to medium- and low-resource languages without expensive native instruction data.","feed_headline":"Feed-forward LoRA wins for adapting LLMs to new languages","feed_subtitle":"Case study on a 1B Llama-3.2 tuned on Icelandic: where PEFT parameters sit decides success, and prefix tuning fails.","key_machinery":"The argument is carried by the contrast between four parameter-efficient fine-tuning mechanisms placed inside a fixed Transformer: LoRA, which adds low-rank decomposition matrices to chosen weights; bottleneck adapters, which insert down- and up-projections between layers; (IA)3, which multiplies activations by learned vectors; and prefix tuning, which prepends learnable prefix vectors to the input. The paper varies the rank or reduction factor of each method and, for LoRA, the module (query/value attention versus feed-forward) and layer range, then reads off BERTScore and ROUGE-L on the RUV Radio News summarisation task. The key contrast is that parameter count alone does not explain performance: feed-forward LoRA and bottleneck adapters convert parameters into scores efficiently, attention LoRA does not, and prefix tuning interferes with generation despite its parameter count.","core_discovery":"The central discovery is a placement-and-capacity ranking for PEFT-based language adaptation. With 250,000 Icelandic text chunks of up to 1,024 tokens, the paper finds that adaptation quality rises with the number of trainable parameters, but only when the parameters are put in the right modules. Feed-forward LoRA with rank 256 reaches BERTScore 65.60 / ROUGE-L 09.72 in 0-shot summarisation, beating attention-placed LoRA of the same rank, and combining both modules is not better than feed-forward alone. Bottleneck adapters with reduction factor 4 are competitive, while attention LoRA needs far more parameters (rank 1024) to approach feed-forward results, and prefix tuning and (IA)3 actively hurt the model, with prefix tuning collapsing in 1- and 5-shot settings. The paper argues this shows feed-forward modules are the most promising target and that sufficient learning capacity is necessary.","pith_inferences":["Beyond the paper's explicit comparisons, its placement ranking suggests that for a fixed parameter budget, PEFT design should target where language-specific knowledge is stored, not just how many parameters are trained.","A testable extension the paper leaves implicit: train the same adapters with a 4,096-token adaptation context; if the 5-shot degradation from attention LoRA disappears, the short-context hypothesis is confirmed.","The result that curated data gave no benefit is about extractive summarisation; on a task requiring language-specific world knowledge, the ordering might shift.","Human evaluation of the top setups could reveal quality differences BERTScore and ROUGE-L miss, since those metrics reward lexical overlap."],"forward_implications":["Practitioners adapting instruction-tuned models with unstructured text should prefer feed-forward LoRA at high rank, or bottleneck adapters with a small reduction factor, over attention LoRA.","More trainable parameters help, but placement can dominate: attention LoRA at rank 1024 still trails feed-forward LoRA at rank 256, so parameter budgets should go to feed-forward modules first.","Prefix tuning and (IA)3 should be avoided for text-only language adaptation of instruction-tuned models.","Context degradation from limited adaptation context can be reduced by adapting only the final layers, at a slight cost in 0-shot performance.","In-context learning with target-language demonstrations is a viable alternative: the 1-shot baseline already matches many adapted setups."],"supporting_citations":[{"why":"Defines LoRA, the central method whose placement and rank are varied.","marker":"(Hu et al., 2022)"},{"why":"Introduces bottleneck adapters, one of the compared methods.","marker":"(Houlsby et al., 2019)"},{"why":"Introduces (IA)3, shown to be unsuitable for language adaptation.","marker":"(Liu et al., 2022)"},{"why":"Introduces prefix tuning, shown to hurt adaptation.","marker":"(Li and Liang, 2021)"},{"why":"Provides the RUV Radio News dataset used for all evaluations.","marker":"(Sverrisson and Einarsson, 2023)"},{"why":"Defines BERTScore, the primary automatic metric.","marker":"(Zhang et al., 2020)"},{"why":"Defines ROUGE-L, the secondary surface-overlap metric.","marker":"(Lin, 2004)"},{"why":"Supplies the CC100 Icelandic corpus used for adaptation.","marker":"(Conneau et al., 2020)"}],"fun_headline_variants":["PEFT placement beats parameter count for language adaptation","Feed-forward LoRA tops attention for Icelandic LLM adaptation","More trainable params help if placed in feed-forward layers","Prefix tuning fails, feed-forward LoRA thrives for LLM adaptation","LoRA rank 256 in feed-forward bests attention for Icelandic"],"cache_read_input_tokens":13440,"weakest_assumption_plain":"The ranking is built on single runs of BERTScore and ROUGE-L on one news-summarisation dataset, so if those automatic scores are noisy or do not track generation quality, the recommended setups could change.","fun_headline_variants_meta":{"raw":{"variants":["PEFT placement beats parameter count for language adaptation","Feed-forward LoRA tops attention for Icelandic LLM adaptation","More trainable params help if placed in feed-forward layers","Prefix tuning fails, feed-forward LoRA thrives for LLM adaptation","LoRA rank 256 in feed-forward bests attention for Icelandic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":2099,"prompt_tokens":901,"completion_tokens":1198,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1115}},"tokens_in":517,"tokens_out":1198,"duration_ms":8274,"temperature":1.0,"reasoning_tokens":1115,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:50:24.727946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the full ablation with five random seeds and a human preference study on the generated summaries; a reversal in the ranking, say attention LoRA matching or beating feed-forward LoRA, would falsify the paper's central claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RUV Radio News dataset used for all evaluations."}],"review_version":1}