{"id":"f89e8d58-74d4-48d6-a11d-85910db73559","arxiv_id":"2501.08008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TriAdaptLoRA, a triangular low-rank adaptation with importance-driven rank growth, achieves small average gains over LoRA, AdaLoRA, and IncreLoRA on GLUE and SQuAD 2.0, with mixed per-task results.","lead":"This paper introduces TriAdaptLoRA, a method for fine-tuning large language models that writes the update matrix as the product of triangular matrices and grows its rank adaptively during training. It reports small average gains over LoRA, AdaLoRA, and IncreLoRA on GLUE and SQuAD 2.0, but per-task results are mixed and no code is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's headline 'TriAdaptLoRA' row is a per-task blend of the Linear and Non-Linear rows; no single prescribed configuration attains the 89.34 average, so 'consistently outperforms' is unsupported.","rationale":"I focused on the strongest_claim. The reader's weakest_assumption (Equation 11 importance proxy) is a real theoretical gap, but it is not the most load-bearing: the paper's empirical comparison is what supports the central claim, and that comparison is internally inconsistent before any question of mechanism. Table I contains a row labeled 'TriAdaptLoRA' that is not one of the two configurations specified in Section III-D; cell-by-cell it is the per-task maximum of the Linear and Non-Linear rows, yielding an average no single configuration attains. This is an arithmetic fact, not a matter of external reproducibility or consensus. The Linear mode itself loses to AdaLoRA on five of eight tasks, so 'consistently outperforms' is contradicted by the paper's own numbers. The SQuAD 2.0 result lacks per-seed statistics and code, and no wall-clock or FLOP measurement backs the 'reduced computational overhead' claim, but those are secondary. Because the headline evidence is an undefined hybrid, I recommend moving from CONDITIONAL to REJECT: the central claim as stated is not supported by the reported data. If the authors clarify the row or release code showing a single configuration attains 89.34, the claim could be revisited.","tokens_in":17311,"tokens_out":11045,"duration_ms":103446,"concrete_test":"Analytically reconstruct the 'TriAdaptLoRA' row from Table I: for each task, take the better of the 'TriAdaptLoRA (Linear)' and 'TriAdaptLoRA (Non-Linear)' entries. If this per-task maximum equals the reported 'TriAdaptLoRA' row (it does for all tasks, with CoLA tied), then the 89.34 average is a per-task best-of-two selection rather than the result of one configuration. The decisive check is to ask the authors to state which single threshold mode, hyperparameter setting, and random seeds produced the 'TriAdaptLoRA' row; if no such single configuration exists, the headline comparison is invalid. Re-running even one GLUE task with a fixed mode and 5 seeds would also reveal whether the 0.35-point average gap over AdaLoRA is within seed noise.","verdict_should_be":"REJECT","load_bearing_attack":"Table I's row labeled 'TriAdaptLoRA' (average 89.34) is the primary numerical basis for the abstract's claim of consistent superiority. But Section III-D and Section IV-B3 define only two adaptive threshold modes, Linear and Non-Linear, whose own averages are 89.28 and 89.10. The 'TriAdaptLoRA' row is not a third configuration: its entries match the Non-Linear row exactly on MNLI (90.64), SST-2 (95.68), and STS-B (91.79), and the Linear row exactly on QQP (92.09), QNLI (94.37), RTE (87.84), and MRPC (90.77), with CoLA tied at 71.6. No single run can match two different rows on tasks where they differ, so the reported 89.34 is a per-task best-of-two selection. Furthermore, the defined Linear mode is worse than AdaLoRA on 5 of 8 GLUE tasks (MNLI, SST-2, QQP, QNLI, STS-B); the paper's own table therefore contradicts 'consistently outperforms existing PEFT methods.' SQuAD 2.0 (Table IV) is reported without per-seed variance or code, so the only numerical support for the headline is this undefined row.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TriAdaptLoRA, a parameter-efficient fine-tuning method that represents the incremental update as ∆W = B(L+U)A, where L and U are lower/upper triangular matrices and D = L+U is a dense transformation in the low-rank subspace. The rank of each incremental matrix is grown during training according to an importance score (Equation 11) and a dynamic budget governed by either a linear or nonlinear threshold (Equations 12–14). The authors evaluate TriAdaptLoRA on eight GLUE tasks with DeBERTaV3-base and on SQuAD 2.0, comparing against full fine-tuning, BitFit, adapters, LoRA, AdaLoRA, and IncreLoRA, and they include ablations on the orthogonality constraint, Frobenius-norm normalization, warm-up steps, rank-update interval, and reference rank. The headline claim is that TriAdaptLoRA consistently outperforms existing PEFT methods while reducing computational overhead.","tokens_in":17660,"tokens_out":10755,"duration_ms":102802,"significance":"The triangular split with adaptive rank growth is a plausible and potentially useful design, and the paper gives a fairly complete algorithmic description of the proposed procedure. The experimental protocol is partly thorough: three-seed averages are reported for GLUE, and the paper includes ablations and sensitivity analyses on RTE. However, the reported evidence does not currently support the central claim of consistent superiority. The best GLUE average comes from a row that matches neither of the two defined configurations, both defined modes lose to AdaLoRA on five of eight tasks, the SQuAD 2.0 results have no variance, the computational savings are theoretical rather than measured, and no code is provided. If the authors report the two defined configurations honestly and reword the claims, the method may still be of interest as a simple heuristic alternative with fewer AdaLoRA-style hyperparameters, but the paper as written overstates its results.","major_comments":[{"comment":"The row labeled 'TriAdaptLoRA' with average 89.34 is not a configuration defined in Sections III-D or IV-B3, and no single run can reproduce it. Its entries coincide with the Non-Linear row on MNLI (90.64), SST-2 (95.68), CoLA (71.6), and STS-B (91.79), and with the Linear row on QQP (92.09), QNLI (94.37), RTE (87.84), and MRPC (90.77). This is a per-task best-of-two selection, not a method result. Moreover, both defined configurations underperform AdaLoRA on five of eight tasks (Linear on MNLI, SST-2, QQP, QNLI, and STS-B; Non-Linear on MNLI, SST-2, QQP, QNLI, and RTE). The abstract's claim that TriAdaptLoRA 'consistently outperforms existing PEFT methods' and 'achieves superior performance' is therefore not supported by Table I. Please report each configuration separately, remove the blended row, and reword the claims accordingly.","section":"Table I, §IV-B3"},{"comment":"Several entries report exact zero standard deviations from three-seed averages, for example AdaLoRA MNLI 90.66±0, TriAdaptLoRA QQP/QNLI 92.09±0/94.37±0, and TriAdaptLoRA (Non-Linear) CoLA 71.6±0. Exact zero standard deviations across tasks are implausible and likely reflect rounding or reporting artifacts, and they cannot support the abstract's 'enhanced stability' claim. Table IV reports SQuAD 2.0 results with no standard deviations, number of seeds, or error bars, even though Section IV-B2 promises three-seed averages for GLUE. Please provide full variance and seed information for every reported result, or state the number of runs per table.","section":"Tables I, II, IV"},{"comment":"The computational-overhead claim is theoretical only. Section III-C derives arithmetic complexity O(r^2 MT) versus O((rn+dr)MT), and Section V concludes that TriAdaptLoRA 'reduces energy consumption,' but no wall-clock time, FLOPs, peak memory, or energy measurements are reported anywhere in the paper. The abstract's 'reduced computational overhead' is therefore not empirically validated. Please add measured runtime/memory/energy comparisons or soften the claim.","section":"§III-C, §V"},{"comment":"The written rank-growth algorithm does not implement the intended warm-up delay. In Algorithm 1, rank growth is performed at every step t with R(t−1)>0, and Equation 13 uses α(t)=(t−t0)/(T−t0). For t<t0, α(t) is negative, so k(t) becomes negative for the linear mode and the lower bound k(t)=max(k(t),1) forces one rank increment per step before warm-up. The nonlinear mode similarly yields k(t)=1 for t<t0. Thus the 'warm-up steps' t0 do not actually delay rank growth as described, which contradicts the sensitivity analysis in Section IV-D1. Please add an explicit t≥t0 condition (or the intended equivalent) and clarify how warm-up was implemented in the reported experiments.","section":"§III-D, Algorithm 1"},{"comment":"The theoretical justification of the importance metric is a loose motivation rather than a proof. Section III-E shows that D=L+U appears in the gradient expressions, but it does not establish that the normalized Frobenius-norm change in Equation 11 tracks task-relevant parameter importance; any differentiable statistic of D would appear in a chain-rule expansion. Since the adaptive rank-allocation advantage of TriAdaptLoRA rests entirely on this heuristic, either the derivation should be made rigorous or the paper should provide direct empirical validation, for example by comparing the importance ranks with held-out performance changes when individual matrices are expanded.","section":"§III-E, Eq. (11)"}],"minor_comments":[{"comment":"The 'Fixed-k' variant is used in Table I and Section IV-B4 but is never defined in the method section; please provide its definition or remove it.","section":"§III-D, Table I"},{"comment":"SQuAD 2.0 is described as a natural language generation task, but it is an extractive question-answering benchmark; please adjust the task taxonomy.","section":"§IV-C"},{"comment":"Table V contains formatting/typographical issues, including 'def ault' in the LoRA Dropout row and an unclear grouping despite the text saying 'the first fourteen are shared hyperparameters'; please reformat the table for clarity.","section":"Appendix A, Table V"},{"comment":"The MRPC dev-set size is listed as 408k, which is inconsistent with the actual MRPC dev set of 408 sentence pairs; this appears to be a typo.","section":"Appendix B, Table VII"},{"comment":"No code or data release is mentioned; providing a link to an implementation would materially improve reproducibility, especially for the exact rank-growth schedules and the initialization of new rows/columns during expansion.","section":"Reproducibility"},{"comment":"The symbol α is used both for the scaling factor in Equation 5 and for the time-varying threshold function α(t) in Equations 13–14; please use distinct notations to avoid confusion.","section":"§III-B, §III-D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of a PEFT/LLM fine-tuning venue, but the current presentation is not acceptable as is. The main issue is not the algorithmic idea but the empirical reporting: the blended 'TriAdaptLoRA' row in Table I, the zero/absent variances, and the unmeasured computational claims. I would encourage the editor to treat the abstract's 'consistently outperforms' and 'reduced computational overhead' claims as provisional until the authors either supply the missing measurements and seed-level data or explicitly narrow their claims to the defined Linear/Non-Linear configurations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the triangular-split idea is a genuine variant, but the headline result is a best-of-two blend, and the paper needs code and per-seed numbers before the central claim holds.\n\nWhat's new: B(L+U)A with L and U triangular is just a reparameterization of standard LoRA for fixed r—same expressiveness, same parameter count. The real contribution is the adaptive rank-growing scheme: a cheap importance score (change in rank-normalized Frobenius norm of D) plus a budget-based threshold deciding which matrices grow, avoiding per-step gradient-based importance like AdaLoRA. That combination, with the triangular split for row/column expansion, is not in the cited IncreLoRA or AdaLoRA. The paper gives clear ablations (orthogonality, normalization, fixed-k vs adaptive) and sensitivity runs on RTE. Hyperparameter tables are thorough.\n\nThe soft spots are real and mostly in the empirical claims. The stress-test note is correct: the 'TriAdaptLoRA' row in Table I is not a single configuration. Its entries match Non-Linear on MNLI, SST-2, CoLA, and STS-B, and Linear on QQP, QNLI, RTE, and MRPC. No single run can be both. So the 89.34 average is a per-task best-of-two selection, and the abstract's 'consistently outperforms' is not supported by the table. Worse, the defined Linear mode is worse than AdaLoRA on five of eight GLUE tasks. The zero standard deviations on several three-seed averages (e.g., QQP 92.09±0, QNLI 94.37±0) are implausible for these metrics; that needs per-seed disclosure. SQuAD 2.0 has no variance at all. The computational savings are argued from complexity, not measured—fine as a claim, but not as 'reduced computational overhead' in a results context. The theoretical justification for the importance metric (Sec. III-E) is a loose gradient-propagation sketch; it does not establish that Frobenius-norm change tracks true importance. And no code is released.\n\nNone of this kills the method. For a fixed total rank, TriAdaptLoRA is a plausible alternative to AdaLoRA with a cheaper importance evaluation, and the ablations suggest the normalization and orthogonality choices matter. But the paper currently overclaims. A serious referee should see it: the core idea is coherent, the experiments are reproducible in principle, and the issues are fixable with code, per-seed numbers, and a headline row that respects the actual configurations. I'd recommend engaging—but the authors need a revision, not a camera-ready.\n\nFor a reading group: maybe, if you work on PEFT. I wouldn't cite it until the numbers are cleaned up. My verdict: accept to peer review, expect heavy revision.","headline":"The triangular-split rank-growth idea is a real variant, but the headline GLUE number is a per-task best-of-two blend, and the paper needs code and per-seed data before the central claim holds.","tokens_in":18133,"tokens_out":2918,"would_cite":false,"duration_ms":25868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TriAdaptLoRA claims consistent gains over prior adaptive-rank PEFT methods by growing ranks where a cheap Frobenius-norm change score indicates importance, reporting 89.34 average GLUE and 85.82 EM on SQuAD 2.0.","keywords":["parameter-efficient fine-tuning","low-rank adaptation","adaptive rank growth","dynamic threshold","triangular decomposition","Frobenius norm importance","GLUE benchmark","SQuAD 2.0"],"falsifier":"Replace the importance score in Equation 11 with random noise drawn on the same scale while keeping all other components fixed, and evaluate on MNLI: the central claim predicts a clear drop below the reported 90.64 accuracy, whereas a null result would show the importance ranking is not load-bearing.","tokens_in":17119,"feed_emoji":"🧠","tokens_out":4973,"duration_ms":43474,"temperature":0.7,"pith_summary":"The paper sets out to show that parameter-efficient fine-tuning can be improved by deciding, during training, which low-rank incremental matrices deserve a larger rank. It introduces TriAdaptLoRA, which writes each incremental matrix as $\\Delta W = B(L+U)A$ with $L$ lower and $U$ upper triangular, so rank can grow by adding rows to $L$ and columns to $U$. An importance score based on the change in the rank-normalized Frobenius norm of $D=L+U$ picks which matrices receive rank increments, and a dynamic threshold decides how many matrices are updated at each step. On GLUE with DeBERTaV3-base the method reports an average of 89.34, above AdaLoRA (88.99) and IncreLoRA (88.90), and it also reports gains on SQuAD 2.0. The paper concludes that adaptive rank growth with this cheap importance proxy is both more accurate and cheaper than prior adaptive-rank methods.","feed_headline":"TriAdaptLoRA beats adaptive-rank rivals with 89.34 GLUE average","feed_subtitle":"Rank growth guided by a cheap Frobenius-norm score beats AdaLoRA and IncreLoRA.","key_machinery":"The load-bearing object is the dense triangular transformation matrix $D=L+U$ inside each low-rank increment, with $\\Delta W = B D A$. $L$ and $U$ are lower and upper triangular matrices that are augmented in opposite directions (rows to $L$, columns to $U$), letting the rank grow while reusing existing parameters. The importance metric in Equation 11 turns the change in the rank-normalized Frobenius norm of $D$ into a scalar score; the adaptive threshold in Equations 13 and 14 converts a remaining rank budget into the number $k(t)$ of matrices that get expanded. An orthogonality penalty on $A$ and $B$ (Equation 10) and a scaling factor $\\alpha/(r+\\epsilon)$ stabilize training as the rank grows.","core_discovery":"The central claim is that the triangular transformation matrix $D=L+U$ is a sufficient and efficient carrier of rank adaptation. The paper argues that tracking the normalized Frobenius norm of $D$ over training, $S_m^{(t)} = \\|L_m^{(t)}+U_m^{(t)}\\|_F / r_m^{(t)} - \\|L_m^{(t-1)}+U_m^{(t-1)}\\|_F / r_m^{(t-1)}$, identifies which incremental matrices benefit most from extra rank, and that a budget-driven threshold (linear or nonlinear) should control how many matrices grow at each step. The reported experiments support the claim: TriAdaptLoRA averages 89.34 on GLUE versus 88.99 for AdaLoRA and 88.90 for IncreLoRA, and reaches 85.82 EM / 88.90 F1 on SQuAD 2.0 versus 85.56 / 88.66 for IncreLoRA. The authors further claim reduced computational overhead, because importance evaluation costs $O(r^2 M T)$ instead of $O((rn+dr)MT)$, and no need for task-specific tuning of the growth threshold.","pith_inferences":["The triangular split roughly doubles the number of trainable entries per rank unit compared with a plain $BA$ update, so a comparison at equal nominal rank is not parameter-matched; re-evaluating at equal parameter counts would clarify whether the gain comes from the allocation policy or from more parameters.","Because the importance score uses only the Frobenius norm of $D$, it is architecture-agnostic and could be transferred to low-rank adapters in other domains, such as vision or multimodal models, without modification.","The dynamic threshold can be read as a form of budget scheduling; comparing it against simple cosine or exponential schedules of $k(t)$ would separate the effect of the importance ranking from the effect of the schedule shape."],"forward_implications":["Adaptive rank growth under a dynamic threshold removes the need to hand-tune the fixed $k$ used by IncreLoRA, while keeping the total rank budget bounded by $R_0$.","The $O(r^2 M T)$ importance evaluation makes rank adjustment cheap enough to run periodically on large models, cutting the per-step gradient-based scoring used by AdaLoRA and IncreLoRA.","The triangular split enables bidirectional rank expansion, so parameters are added both as new rows of $L$ and new columns of $U$, which the authors argue yields more uniform and continuous final rank distributions across layers.","On the reported benchmarks the method is claimed to consistently outperform the compared PEFT baselines, including full fine-tuning on SQuAD 2.0 under the nonlinear threshold mode."],"supporting_citations":[{"why":"Establishes the baseline low-rank adaptation method that TriAdaptLoRA extends and compares against.","marker":"[13]"},{"why":"The closest adaptive-rank baseline, supplying the orthogonality constraint form and the comparison numbers the paper must beat.","marker":"[21]"},{"why":"The other adaptive-rank baseline, also providing hyperparameter settings reused in the experiments.","marker":"[22]"},{"why":"Defines the GLUE benchmark suite used for the main natural language understanding evaluation.","marker":"[25]"},{"why":"Supplies the SQuAD 2.0 dataset with unanswerable questions used for the natural language generation evaluation.","marker":"[26]"},{"why":"The DeBERTaV3-base pre-trained backbone on which all fine-tuning methods are evaluated.","marker":"[32]"}],"fun_headline_variants":["Triangular split powers TriAdaptLoRA past rivals on GLUE","Adaptive rank by Frobenius norm wins 89.34 GLUE for TriAdaptLoRA","TriAdaptLoRA's triangle trick tops GLUE at 89.34 average","Cheap importance metric in TriAdaptLoRA yields 89.34 GLUE","Brain-inspired triangular LoRA outranks adaptive-rank rivals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the change in the rank-normalized Frobenius norm of the triangular matrix $D$ correctly identifies which incremental matrices should receive more rank; if this proxy does not track true task-relevant importance, the adaptive allocation advantage disappears.","fun_headline_variants_meta":{"raw":{"variants":["Triangular split powers TriAdaptLoRA past rivals on GLUE","Adaptive rank by Frobenius norm wins 89.34 GLUE for TriAdaptLoRA","TriAdaptLoRA's triangle trick tops GLUE at 89.34 average","Cheap importance metric in TriAdaptLoRA yields 89.34 GLUE","Brain-inspired triangular LoRA outranks adaptive-rank rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3161,"prompt_tokens":1026,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2033}},"tokens_in":642,"tokens_out":2135,"duration_ms":16460,"temperature":1.0,"reasoning_tokens":2033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:46.375878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the importance score in Equation 11 with random noise drawn on the same scale while keeping all other components fixed, and evaluate on MNLI: the central claim predicts a clear drop below the reported 90.64 accuracy, whereas a null result would show the importance ranking is not load-bearing.","supporting_citations":[{"cited_title":"Know what you don’t know: Unanswerable questions for SQuAD,","cited_arxiv_id":null,"evidence_quote":"Supplies the SQuAD 2.0 dataset with unanswerable questions used for the natural language generation evaluation."}],"review_version":1}