REVIEW 4 major objections 4 minor 28 references
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read CoTo claims that progressively turning on LoRA adapters during fine-tuning — most adapters randomly off at first, all on at the end — yields better accuracy, mergeable solutions, and pruning resilience than standard LoRA.
desk verdict Wide empirical survey of a simple progressive LoRA dropout schedule, but the main gains are entangled with higher learning rates and the theory overclaims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the time-dependent activation schedule $p(t) = \min(1, 4t/3T)$ with per-layer indicators $\delta_i \in \{0,1\}$ drawn independently with $\Pr(\delta_i = 1) = p(t)$, gating each adapter's contribution as $\delta_i \mathbf{1} \odot \Delta W_i$; here 'adapter' means all LoRA parameters inside a single Transformer layer. One mechanism does four jobs: it instantiates the subnetwork curriculum whose expected loss Theorem 3.1 shows to upper-bound a binomially weighted sum of subnetwork losses, it connects training to dropout stability and hence to linear mode connectivity, it makes each adapter's marginal contribution measurable through Shapley values computed by the multilinear extension, and it saves compute because inactive adapters are skipped entirely in forward and backward passes.
What would settle it
Re-run the headline 11-task image classification comparison with LoRA and LoRA-CoTo at the same learning rate (2e-4) and same rank; if the accuracy gain disappears or reverses, the schedule is not the cause. A second decisive check is the $\lambda = 0.5$ interpolation test on commonsense reasoning at matched learning rates, since the linear-mode-connectivity claim is the paper's most distinctive prediction.
Extended reading notes
Core claim
The central claim is that standard LoRA trains adapters as one monolithic update and pays for it: 'lazy-training' dynamics keep the whole set of low-rank updates near its initialization while a few high-layer adapters monopolize the gradient signal, so the converged solution is layer-imbalanced and survives neither weight averaging nor pruning. CoTo replaces this with a curriculum over subnetworks: at step $t$ each adapter is active with probability $p(t) = \min(1, 4t/3T)$, so early training is done by sparse random subsets of adapters, the model must learn to behave well under arbitrary adapter dropout, and by the final 25% training reverts to ordinary LoRA with all adapters on. The paper supports this with Theorem 3.1, showing for convex loss that the expected CoTo objective upper-bounds a binomially weighted sum of subnetwork losses, and with Shapley-value measurements in which vanilla LoRA concentrates 69% of adapter contribution in the top 4 of 12 layers while CoTo flattens deviations to within 8%. The payoff if the claim holds is that a schedule is worth more than a merger: interpolating two independently trained CoTo solutions retains near-full accuracy (79% versus 39% for LoRA at equal mixing), and merging task-specific CoTo adapters improves linear weight fusion from 47.17% to 58.53% average accuracy on LLaMA-2-7B.
Load-bearing premise
The reported gains come from the progressive on-off schedule itself and not from the higher learning rates CoTo is given in the main experiments, where rates run 2–5 times above the vanilla baselines.
Editorial extensions
If this is right
- On 11 image-classification tasks with ViT-B/16, CoTo raises average accuracy over vanilla LoRA from 82.95% to 83.48%, and similar or larger gains appear for DoRA (83.45% to 83.93%) and HiRA (83.98% to 84.34%).
- On 8 commonsense-reasoning tasks, CoTo lifts LoRA from 77.61% to 80.63% average accuracy on LLaMA-2-7B and from 80.79% to 85.02% on LLaMA-3-8B.
- Independently trained CoTo solutions are linearly mode connected: averaging their adapter weights at interpolation ratio $\lambda = 0.5$ keeps about 79% accuracy on commonsense reasoning, where vanilla LoRA drops to about 39%.
- Multi-task merging improves across strategies: on LLaMA-2-7B, linear weight fusion of CoTo adapters averages 58.53% versus 47.17% without CoTo, and LoRA-LEGO merging rises from 62.21% to 67.19%.
- CoTo-trained adapters survive pruning better — about 10 points higher accuracy at 50% unstructured sparsity — and training is faster by 7–24% in wall-clock time because deactivated adapters are skipped.
Reading between the lines
- Editorial extension: CoTo is essentially a curriculum that forces each adapter to be locally useful before it is allowed to rely on its neighbors; the same "on later, together later" principle could be applied to other parameter-efficient modules such as prompts, prefixes, or adapter layers, and it invites a scheduling theory that predicts which curriculum shapes help merging most.
- Editorial extension: the main comparisons run CoTo at 2–5 times the learning rate of the vanilla baselines (e.g., 5e-4 versus 2e-4 on image classification), so the headline gains are partly attributable to learning rate; the matched-rate evidence is confined to the ablation table, and the cleanest causal test is a matched-rate rerun on the headline benchmarks.
- Editorial extension: because CoTo trains adapters against random subset dropout, the approximated per-adapter Shapley values could serve as a cheap predictor of merging quality — layers with near-zero marginal contribution could be pruned first without waiting for a full merging experiment.
- Editorial extension: the activation schedule's shape is a free parameter; the paper's own ablations show exponential schedules trade early convergence for even better merging and pruning, so per-task tuning of the $p(t)$ curve is a plausible next lever.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CoTo, a progressive training schedule for LoRA fine-tuning in which each adapter's activation probability increases linearly from 0 to 1 over the first 75% of training steps and then stays at 1 for the remaining 25%. The authors claim that this schedule improves single-task accuracy, makes independently trained adapters more linearly mode connectable, improves multi-task merging and pruning robustness, and reduces training compute. The paper includes a Jensen-inequality bound (Theorem 3.1) and a Shapley-value analysis of layer-wise adapter contributions, along with experiments on vision classification, commonsense reasoning, mathematical reasoning, language understanding, and image generation.
Significance. If the causal effect of the schedule were established, CoTo would be a simple, broadly applicable, and practically useful addition to LoRA-based fine-tuning: it requires no architectural changes, is compatible with several LoRA variants, and the compute savings are plausible. The empirical program is extensive, with three seeds, multiple backbones, and both structured and unstructured pruning, and the authors ship code. However, the theoretical contribution is modest, and the main experimental comparisons currently conflate the schedule with a higher learning rate, so the primary empirical claim is not yet fully supported.
major comments (4)
- [§4.1, Table 8, Table 10] The headline comparisons in Tables 1–4 use different learning rates for the baseline and CoTo runs. Table 8 shows LoRA at 2e-4 and LoRA-CoTo at 5e-4 on all 11 vision tasks; Table 10 reports default versus CoTo learning rates of 1e-5 versus 5e-5 for commonsense reasoning, 2e-5 versus 1e-4 for mathematical reasoning, and 5e-5 versus 2e-4 for image classification. The only matched-learning-rate experiment, Table 6, shows a 0.13 pp gain at 5e-5, a 1.31 pp gain at 1e-4, and at 2e-4 the LoRA-Pro baseline collapses to 40.91 while CoTo maintains 56.25. This pattern suggests that the main gains could arise from CoTo's ability to stabilize or exploit a higher learning rate rather than from the progressive activation schedule itself. The paper should either report headline results with matched learning rates or explicitly frame CoTo as a method that enables higher-LR training and provide evidence that the progressive schedule contributes beyond the LR change.
- [§3.4, Theorem 3.1] Theorem 3.1 is a direct application of Jensen's inequality to the Bernoulli mixture; it holds for any dropout distribution over adapters and does not establish that CoTo trains solutions that are robust to adapter deletion or linearly mode connected. The sentence 'Prior work... shows that dropout stability often implies that independently trained solutions can be connected by a low-loss linear path' is a heuristic appeal rather than a proof, and the theorem itself does not mention the progressive schedule p(t). The abstract and Section 3.4 therefore overstate the theoretical support. I suggest restating the theorem as a generic bound on the expected subnetwork loss (which is correct) and presenting the LMC and pruning results as empirical findings only.
- [§3.5, §4.4] The Shapley-value analysis uses the loss ℓ as the value function v(R) in Eq. (7). Because a lower loss is better, the marginal contribution v(R ∪ {i}) − v(R) is expected to be negative when adding an adapter improves the model, and the reported 'contributions' in Figure 8 (e.g., 69% concentrated in the top layers) are not defined in terms of this sign. Please clarify whether the figure plots φ_i(v), −φ_i(v), or a normalized quantity, and how the concentration percentages are computed.
- [Table 1, Table 9] Several per-task gains in Table 1 are smaller than one standard deviation reported in Table 9 (e.g., Caltech 96.25±0.12 vs 96.26±0.10; ImageNet 73.49±0.06 vs 73.53±0.04). The paper does not test whether the average improvements are statistically significant. A paired test across the 11 tasks or reporting the number of tasks where the improvement exceeds the standard deviation would substantiate the claim of 'consistent' improvement.
minor comments (4)
- [Eq. (2)] The linear schedule starts at p(1) = 4/(3T) rather than exactly 0 as stated in the text; if the authors intend p(0) = 0, the formula should be adjusted accordingly, for example by using (t−1)/(3T/4) in the first branch.
- [Footnote 3] The citation (Zhao et al., 2024a) for the linear model ensemble appears to point to LoraRetriever, which does not define this operation; please cite the original source of the linear model ensemble formula.
- [Figure 2] The weight functions w_j(p) are plotted for L = 6, but the text does not specify the value of L used in the figure; adding this detail would aid readability.
- [Table 7] The reported training speedups come from skipping inactive adapters, but the paper does not discuss whether the wall-clock measurements include the overhead of sampling activation indicators or whether the speedup would persist on hardware with specialized kernels; a brief clarification would be useful.
Circularity Check
No circular derivation: CoTo's schedule is not fitted to the metrics it predicts, and its gains are benchmark comparisons; the learning-rate mismatch is a correctness risk, not circularity.
full rationale
The derivation chain is self-contained rather than circular. CoTo defines an activation schedule p(t) (Eq. 2) and an objective (Eq. 4); Theorem 3.1 is a direct application of Jensen's inequality to that objective and does not assume the paper's empirical conclusions about accuracy, merging, or pruning. The LMC and Shapley-value discussions are post-hoc analyses or empirical measurements, not fitted parameters renamed as predictions. The headline comparisons use baselines from prior work, including the authors' own HiRA paper (Table 2 note: "Results without CoTo are copied from (Huang et al., 2025)"), but this self-citation supplies external baseline numbers rather than a load-bearing axiom; the central claim does not reduce to it. The main substantive concern is that CoTo is often run at a higher learning rate than the baseline (Tables 8 and 10: e.g., 2e-4 vs 5e-4 on vision, 1e-5 vs 5e-5 on commonsense reasoning), which may confound the measured gains, and the matched-LR ablation in Table 6 shows small gains at the lowest learning rate. That is a threat to causal attribution, not a circularity of the derivation chain. No equation or fitted constant is shown to be equivalent by construction to the reported predictions, so the circularity score is low.
Assumptions & free parameters
free parameters (3)
- first-phase proportion =
75%
- activation schedule shape =
linear
- CoTo learning rate =
2e-4 to 5e-2 depending on task and variant
assumptions (3)
- standard math Convexity of the loss ℓ(·, y)
- domain assumption Layer-wise imbalance causes suboptimal minima
- domain assumption Dropout stability implies linear mode connectivity
Cite this review
Pith. "Pith review of Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/XNMM5OL5
@misc{pith2026250605713,
author = {Pith},
title = {Pith review of: Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/XNMM5OL5}},
note = {Machine review of arXiv:2506.05713}
}
read the original abstract
Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning technique for adapting large foundation models, yet it often locks adapters into suboptimal minima near their initialization. This hampers model generalization and limits downstream operators such as adapter merging and pruning. Here, we propose CoTo, a progressive training strategy that gradually increases adapters' activation probability over the course of fine-tuning. By stochastically deactivating adapters, CoTo encourages more balanced optimization and broader exploration of the loss landscape. We provide a theoretical analysis showing that CoTo promotes layer-wise dropout stability and linear mode connectivity, and we adopt a cooperative-game approach to quantify each adapter's marginal contribution. Extensive experiments demonstrate that CoTo consistently boosts single-task performance, enhances multi-task merging accuracy, improves pruning robustness, and reduces training overhead, all while remaining compatible with diverse LoRA variants. Code is available at https://github.com/zwebzone/coto.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[5]
image generation. For image classification, we follow Zanella & Ben Ayed (2024) and use 11 datasets: • Aircraft (Maji et al.,
work page 2024
-
[10]
Fine-grained visual classification of aircraft
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,
-
[11]
PiSSA: Principal sin- gular values and singular vectors adaptation of large lan- guage models
Meng, F., Wang, Z., and Zhang, M. PiSSA: Principal sin- gular values and singular vectors adaptation of large lan- guage models. arXiv preprint arXiv:2404.02948,
- [13]
-
[14]
SocialIQA: Commonsense reasoning about social interactions
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . SocialIQA: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728,
arXiv 1904
-
[15]
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,
-
[17]
LoRA meets dropout under a unified framework
Wang, S., Chen, L., Jiang, J., Xue, B., Kong, L., and Wu, C. LoRA meets dropout under a unified framework. arXiv preprint arXiv:2403.00812, 2024a. Wang, S., Yu, L., and Li, J. LoRA-GA: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems, 2024b. Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. LoRA- Pro: ...
-
[18]
HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,
Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,
arXiv 1905
Show all 28 references
-
[19]
LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning
Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303 , 2023a. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y ., Chen, W., and Zhao, T. Adaptive budget alloca...
-
[21]
(human action classification) For commonsense reasoning, we use 8 tasks from Commonsense170K (Hu et al., 2023): • ARC-c and ARC-e (Clark et al.,
2023
-
[22]
For language understanding, we follow Zhao et al
and test on GSM8K (Cobbe et al., 2021). For language understanding, we follow Zhao et al. (2024b) and use 9 tasks from GLUE (Wang et al.,
2024
-
[23]
and Flan Collection (Longpre et al., 2023): • CoLA (Dolan & Brockett,
2023
-
[25]
backbone. Learning Rate Method CoLA MRPC QNLI QQP RTE SST2 Avg w/o CoTo 5e-4 LoRA 87.44 89.46 94.31 91.05 85.56 95.18 90.50 Fusion 69.89 68.38 49.97 65.50 47.29 55.05 59.35 Ensemble 69.13 31.62 50.54 63.31 52.71 50.92 53.04 LoRA-LEGO 73.28 33.15 74.19 80.95 61.46 69.18 65.37 1...
2024
-
[26]
Linear interpolation accuracy on 8 individual commonsense reasoning tasks (Hu et al., 2023). All EveryOther Low Middle High Query Key Value 30 40 50 Aircraft All EveryOther Low Middle High Query Key Value 94 95 96 97 Caltech All EveryOther Low Middle High Query Key Value 70 75...
2023
-
[27]
lazy training
confirm LoRA adapters converge near initialization (i.e., “lazy training”) across learning rates. CoTo yields tighter clusters under initialization noise. 17 CoTo: A Progressive Strategy to Boost Low-Rank Adaptation 0 2000 4000 6000 8000 0.0 0.2 0.4 0.6 0.8 1.0 Activation Prob...
2000
-
[28]
Merging accuracy is measured at λ = 0.5
Performance evolution during training under different activation schedules on the visual texture classification task (Cimpoi et al., 2014). Merging accuracy is measured at λ = 0.5. Pruning accuracy is measured for EveryOther (i.e., removing alternating layers). 40 20 0 20 40 x...
2014
-
[2005]
S., Zhai, X., Poczos, B., and Singh, A
Du, S. S., Zhai, X., Poczos, B., and Singh, A. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054,
-
[2012]
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958,
1929
-
[2013]
The power of scale for parameter-efficient prompt tuning
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691,
-
[2014]
R., Angeli, G., Potts, C., and Manning, C
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326,
-
[2016]
(question-answering) • QQP 5 (Quora questions) • RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al.,
2005
-
[2018]
Training verifiers to solve math word problems
Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,
-
[2019]
Think you have solved question answering? Try ARC, the AI2 reasoning chal- lenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning chal- lenge. arXiv preprint arXiv:1803.05457,
-
[2020]
Higher layers need more LoRA experts
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y ., Guo, X., Yang, J., and Subrahmanian, V . Higher layers need more LoRA experts. arXiv preprint arXiv:2402.08562, 2024a. Gao, Z., Wang, Q., Chen, A., Liu, Z., Wu, B., Chen, L., and Li, J. Parameter-efficient fin...
-
[2021]
LoRA dropout as a sparsity regularizer for over- fitting control
Lin, Y ., Ma, X., Chu, X., Jin, Y ., Yang, Z., Wang, Y ., and Mei, H. LoRA dropout as a sparsity regularizer for over- fitting control. arXiv preprint arXiv:2404.09610,
-
[2023]
Y ., Pang, T., Du, C., and Lin, M
Huang, C., Liu, Q., Lin, B. Y ., Pang, T., Du, C., and Lin, M. LoraHub: Efficient cross-task generalization via dynamic LoRA composition. arXiv preprint arXiv:2307.13269,
-
[2024]
Llama 2: Open foundation and fine-tuned chat models
Meta. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,
-
[2025]
A rank stabilization scaling factor for fine- tuning with LoRA
Kalajdzievski, D. A rank stabilization scaling factor for fine- tuning with LoRA. arXiv preprint arXiv:2312.03732 ,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.