{"id":"1c23b213-0fd0-43f4-82d6-89e81dd2b389","arxiv_id":"2507.17706","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HydraOpt merges K LoRA adapters into one shared A' and M task-specific B' matrices, yielding a storage-performance curve that outperforms TA, TIES, DARE, and DARE-TIES once storage rises modestly above 20%.","lead":"HydraOpt compresses several trained LoRA adapters into fewer stored parameters by learning one shared low-rank matrix and a set of task-specific matrices. It lets developers choose how much storage to save versus how much accuracy to keep, and it beats standard merging methods when a little extra storage is allowed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HydraOpt's storage-accuracy claim relies on an unverified parameter-space-to-task-performance transfer, and the paper's abstract overstates the retention relative to the 40-task result.","rationale":"The reader's weakest assumption (parameter-space error as proxy for downstream performance) is the same load-bearing concern I identify. The paper demonstrates a plausible method with consistent gains in the 5-adapter per-language setting, but the abstract overstates the retention by quoting only the per-language numbers, the softmax temperature inconsistency between Eq. (1) and the code is a concrete correctness issue, and the absence of error bars weakens the comparisons. These are addressable with reproducibility and analysis, so the appropriate verdict remains CONDITIONAL rather than reject or accept.","tokens_in":29640,"tokens_out":1786,"duration_ms":17573,"concrete_test":"Re-run the English 5-application experiment (Table 1) with three random seeds; report mean and standard deviation for LoRA, HydraOpt(M=2), and HydraOpt(M=5). Then re-implement the loss using exactly Eq. (1) with σ(C'_i/T) and compare the resulting accuracy to the code's softmax(C*T) variant. Finally, compute per-task Spearman correlation between the achieved L1 reconstruction error and the per-task accuracy drop across the 40-task setup; if rho < 0.3, the parameter-space proxy assumption is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (0.2–1.8% accuracy drop at 48% storage reduction) rests on the assumption that L1 distance in parameter space between ΔW_i = B_i A_i and the reconstructed shared-A product is a faithful proxy for preserving each task's downstream behavior. The paper only establishes this empirically, never analytically, and the empirical support is incomplete. The 0.2–1.8% figure comes from the 5-application per-language setting (Table 1 and similar); Table 4 shows the 40-task merge incurs a 7.2% drop, which the abstract omits. Moreover, the implementation disagrees with Eq. (1): the paper defines σ(C'_i/T), but the code in Figures 7–9 uses softmax(C * T) in hydra_loss and softmax(C / T) in the mapping step, so the temperature if used is applied inconsistently. Without error bars or a code release, the reported margins over TA/TIES at modest storage increases cannot be distinguished from noise. The weakest link is thus the transfer from parameter-space reconstruction error to task performance, and the abstract overstates the headline retention.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HydraOpt, a data-free adapter-merging method that takes K low-rank adapter pairs (e.g., LoRA) and approximates them with one shared A' matrix and M task-specific B' matrices, minimizing an L1 parameter-space distance to the original updates ΔW_i = B_i A_i. The number M controls a storage-performance trade-off, and a softmax over learned coefficients C' selects which B' is used per task. The empirical study covers 5 applications x 8 languages (40 tasks), four LLMs, LoRA and VeRA, and rank ablations. The authors report that at the highest compression (M=1, 20% storage) HydraOpt is comparable to Task Arithmetic, while at higher M it approaches the unmerged LoRA upper bound; the abstract highlights a 48% storage reduction with only a 0.2-1.8% average performance drop, and claims consistent gains over TIES, DARE, and DARE-TIES when storage is allowed to increase.","tokens_in":29835,"tokens_out":8249,"duration_ms":84897,"significance":"If the findings hold, HydraOpt is a useful contribution to on-device LLM deployment: it is data-free, requires no task-specific samples, and offers a tunable storage-performance knob that existing merging methods lack. The evaluation is unusually broad for a merging paper (40 tasks, 8 languages, 4 model families, 2 adapter types, rank ablations) and the appendix includes full PyTorch-style code, which is a strength. The core optimization is straightforward and not circular: the original adapters are inputs, and evaluation is on held-out tasks. However, the significance is conditional on resolving several load-bearing issues: the headline accuracy claim omits the most difficult 40-task merging scenario, the reported storage percentages appear inconsistent with the paper's own parameter-count formula, the implementation's temperature handling disagrees with Eq. (1), and no error bars or multiple seeds are provided. Because the claimed advantages are on the order of 1-2% in average accuracy, the lack of variance estimates is a substantive concern, not a cosmetic one.","major_comments":[{"comment":"The abstract and Section 5 state a '0.2-1.8% drop' with a '48% storage reduction,' but Table 4 shows that merging all 40 tasks with HydraOpt(M=40) at S=41.5% incurs an average drop of 7.2% relative to LoRA (L1B: 21.9 vs 28.1; L3B: 25.2 vs 33.2; average 23.5 vs 30.7). The 0.2-1.8% range comes from the per-language 5-application setting, not from the full 40-task setting. This is a material overstatement of the method's general behavior and should be corrected or explicitly scoped in the abstract and conclusion.","section":"Abstract and Table 4 (task block)"},{"comment":"The implementation of the objective is inconsistent with Eq. (1). Equation (1) defines the softmax argument as C'_i/T, and the text says small values approximate one-hot vectors. However, the code in Fig. 8 (line 31) computes softmax(C_primes * T) in hydra_loss, while Fig. 9 (line 54) uses softmax(C_primes / T) for the final B' mapping, and Fig. 7 (line 51) initializes C with softmax(randn / T). Because the temperature scaling changes the sharpness of the coefficient distribution (and therefore the gradient signal during training), the implemented objective is not the stated one. Please align Eq. (1), the code, and the mapping step, and report the temperature value used.","section":"Section 3.2 and Figures 7-9"},{"comment":"The storage percentages in Table 1 are inconsistent with the paper's own parameter-count formula. For K=5 and square A/B matrices (d=k), HydraOpt(M=5) requires (5d+k)/(5(d+k)) = 60% of the original LoRA parameters, not the reported S=52%; for M=2 the formula gives 30%, not 28%. The reported S values correspond to d/k = 2/3, which does not match the Llama-3.2-1B q/k/v/o projection shapes described in Section 4.1. Because the '48% storage reduction' headline depends on S=52, this discrepancy must be resolved: either the S values in Tables 1-4 and Figure 4 are wrong, or the parameter-count formula in Section 3.2 is missing a term. Please clarify and report exact parameter counts.","section":"Section 3.2, Table 1, and Fig. 12"},{"comment":"All experiments appear to report a single run with no error bars, multiple seeds, or statistical significance tests. The paper's key claims are differences of 0.2-1.8% in average performance and a 1.5% gain over Task Arithmetic at 8% additional storage; without variance estimates these differences cannot be distinguished from evaluation noise. Please report at least three seeds with standard deviations (or confidence intervals) for the main tables and figures, and state whether the reported numbers are means or single runs.","section":"Tables 1-5 and Figure 4"},{"comment":"The method's core assumption is that minimizing L1 distance between ΔW_i and B'_j A' in parameter space preserves downstream task accuracy. This is verified only indirectly through task performance; the paper does not report the achieved reconstruction error or analyze its correlation with downstream accuracy. Given that the 40-task merging result shows a 7.2% drop despite presumably low L1 error (the setting is the same reconstruction problem), the transfer from parameter error to task behavior is not automatic. Please report the final training loss / L1 reconstruction error for each setting and, if possible, a plot of reconstruction error versus task performance across M values.","section":"Equations (1)-(2) and Section 4.2"}],"minor_comments":[{"comment":"Task Arithmetic on VeRA reports an average of 0.3, far below the zero-shot baseline of 17.2 and all other methods; this value looks like an evaluation or implementation artifact and should be checked or explained.","section":"Table 3"},{"comment":"HydraOpt(M=5) has a runtime of 8.6 minutes, which is lower than HydraOpt(M=3) at 17.2 and HydraOpt(M=4) at 20.6; this is likely a typo and should be corrected.","section":"Table 17"},{"comment":"The legend in Figure 4 uses 'HyperOpt' instead of 'HydraOpt' in several rows.","section":"Figure 4"},{"comment":"The list of applications introduces both the fourth and fifth items as '(iv)'; the numbering should be fixed.","section":"Section 4.1"},{"comment":"The sentence 'the total number of parameters reduces to 60% when merging 5 pairs of LoRA parameters' is ambiguous because it applies only to the M=K case; please state this explicitly and reconcile it with the S values in Table 1.","section":"Section 3.2"},{"comment":"The temperature T is a free hyperparameter, but no sensitivity analysis or chosen value is reported in the main text. Please state the value of T used and, ideally, show its effect on performance.","section":"Section 3.2"},{"comment":"The phrase 'The softmax function approximates categorical one-hot encoded vectors for small values' is imprecise; the relevant limit is T approaching zero for σ(C/T), or large T for σ(C·T). Please clarify the intended scaling.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong, broad empirical setup and a genuinely useful idea (a tunable storage-performance knob for data-free adapter merging), but the current version has several load-bearing inconsistencies: the abstract's 0.2-1.8% claim does not cover the 40-task result, the storage percentages in Table 1 disagree with the paper's own parameter-count formula, and the supplied code disagrees with Eq. (1) in the temperature scaling. All three are fixable with analysis reruns or clarifications, but they must be resolved before the claims can be accepted. The lack of multiple seeds is also a serious issue for a paper whose margins are around 1-2%."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HydraOpt is worth your time. The core idea is genuinely new: instead of merging adapters with a fixed recipe, it learns one shared A' plus M task-specific B' matrices, with a trainable softmax assigning each original task to one of the B' matrices. The M knob gives you a real storage-performance dial. That is not a routine extension of TA, TIES, or DARE, and the paper shows it clearly across 40 tasks, 8 languages, 4 LLMs, two adapter types, and rank ablations. At matched 20% storage it is roughly on par with TA; once you allow 28% storage it beats TA by 1.5–2 points in most settings. The limitations section is honest about the data-free upper bound.\n\nThe main soft spot is the abstract. The \"0.2–1.8% drop\" claim comes from the 5-application-per-language setting. The 40-task merge, which is the paper's most challenging block, shows a 7.2% drop at 41.5% storage. That is still better than TA at 2.5% storage (19.8 vs. 19.9 on L1B, actually a hair better), but it is not the same story. The abstract needs to say which setting it is describing.\n\nSecond, there are no error bars or multiple seeds anywhere. Given that the storage increments are small (20% to 28%), the reported margins over TA could easily sit inside run-to-run noise. The paper should report variance or at least a few seeds for the headline tables. Third, the softmax temperature is inconsistent between the math and the code: Eq. (1) writes sigma(C'/T), but the code uses softmax(C * T) in the loss and softmax(C / T) in the mapping step. Since T is set to 5 and never ablated, this is probably not load-bearing, but it will confuse any reader who tries to reproduce the method.\n\nThe deeper assumption, that L1 distance in parameter space between the original and reconstructed delta-W is a good proxy for preserving downstream behavior, is unproven. But the paper does not pretend otherwise; it is an empirical claim supported by a reasonably broad evaluation. For a data-free merging method, that is standard practice, and the limitation is acknowledged implicitly by the task-block results.\n\nBottom line: this deserves a serious referee. I would accept it conditionally, with three requests: report error bars, fix the abstract to scope the retention claim, and reconcile the softmax definition between text and code. A code release would also go a long way.","headline":"HydraOpt is a genuinely new tunable adapter-merging scheme with a broad, honest evaluation, but the abstract overstates the headline retention and a softmax inconsistency needs fixing before publication.","tokens_in":30449,"tokens_out":1320,"would_cite":true,"duration_ms":15324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HydraOpt merges many LoRA adapters into one shared A-matrix plus a few task-specific B-matrices, cutting storage by 48% while keeping average accuracy within 1.8% of the unmerged adapters.","keywords":["model merging","low-rank adapters","LoRA","parameter-efficient fine-tuning","data-free merging","multilingual tasks","storage reduction","on-device LLM deployment"],"falsifier":"Take one task's original LoRA adapter and add random noise to its $B$ matrix with the same L1 norm as HydraOpt's reconstruction error for that task; if the noisy adapter loses far more accuracy than the merged adapter, then parameter-space L1 error is not what preserves task behavior, and the reported accuracy retention would be specific to these tasks rather than a general property. A direct check is whether per-task reconstruction error predicts per-task accuracy loss across the 40-task grid.","tokens_in":29412,"feed_emoji":"💾","tokens_out":8951,"duration_ms":85825,"temperature":0.7,"pith_summary":"Large language models that serve many specialized tasks usually keep one low-rank adapter per task, and that storage grows quickly on memory-limited devices. HydraOpt claims this overhead can be cut by exploiting an asymmetry observed in LoRA training: the $A$ matrices across tasks remain fairly similar while the $B$ matrices carry task-specific information. The paper's proposal is to fit a single shared $A'$ matrix and $M$ task-specific $B'$ matrices, with softmax coefficients assigning each task to a $B'$, so that the reconstructed products approximate the original adapter updates. On 40 tasks built from 5 applications and 8 languages, this yields a 48% storage reduction with only a 0.2–1.8% average performance drop, and it outperforms Task Arithmetic, TIES, DARE, and DARE-TIES once slightly more storage is allowed. The key contribution is a dial: choosing $M$ navigates the storage–accuracy frontier instead of accepting the fixed trade-off that one-shot merging methods impose.","feed_headline":"One shared A-matrix halves adapter storage with minimal accuracy loss","feed_subtitle":"A tunable merge method keeps scores within 1.8% of unmerged LoRAs while using half the storage.","key_machinery":"The machinery is a parameter-space reconstruction objective: HydraOpt minimizes $f(B_i A_i, \\sum_j \\sigma(C_i'/T)_j B_j' A')$ over a shared $A'$, $M$ task-specific $B'$ matrices, and trainable softmax coefficients $C_i'$, with $f$ chosen as mean absolute error to match the sparsity of adapter parameters. The softmax term approximates a one-hot assignment of tasks to $B'$ parameters, and the coefficients are discarded after training once each task is assigned a $B'$. This objective carries the argument because it converts model merging into an optimization problem whose storage cost is set explicitly by the single integer $M$.","core_discovery":"For a set of LoRA adapters with updates $\\Delta W_i = B_i A_i$, HydraOpt claims that a reconstruction $\\Delta W_i' = \\sum_j \\sigma(C_i'/T)_j B_j' A'$ with one shared $A'$ and $M$ task-specific $B'$ matrices can preserve task performance when the L1 distance between $\\Delta W_i$ and $\\Delta W_i'$ is minimized, with softmax coefficients selecting the $B'$ used by each task. When $M$ equals the number of tasks, the coefficients drop out and each task keeps its own $B'$. The method rests on the observed asymmetry that $A$ parameters initialized from the same distribution stay similar across tasks while $B$ parameters diverge, so the shared $A'$ captures common structure and the $B'$ matrices absorb task-specific detail. Empirically, merging five LoRA adapters with $M = 5$ stores about 52% of the original parameters (a 48% reduction) and averages 0.2–1.8% below the unmerged adapters across four LLMs, and the same recipe extends to VeRA adapters by sharing the $\\Lambda_d'$ direction vectors. The paper concludes that this makes adapter merging a controllable efficiency–performance spectrum rather than a single fixed operating point.","pith_inferences":["The same shared-subspace reconstruction could transfer to other adapter families whose initialization is asymmetric, provided the common $A$-direction remains stable as task diversity grows.","Because the coefficients are trained only on parameter distance, adding a small calibration set to refine the assignment or the $B'$ matrices is a natural data-driven extension; the paper explicitly leaves that direction open.","Storage savings grow with the number of adapters and exceed 50% when $A$ is larger than $B$, so the method's practical upside is largest in deployments with many adapters rather than just a handful.","The $M$-dial suggests a simple deployment rule the paper does not spell out: choose $M$ by measuring the marginal accuracy gained per additional $B'$ on a validation set, rather than treating the storage budget as a fixed constant."],"forward_implications":["At the same 20% storage as fixed baselines ($M = 1$), HydraOpt performs comparably to Task Arithmetic on English five-way merges, and with just 8% more storage ($M = 2$) it exceeds the best baseline by about 1.5 points on average.","With one $B'$ per task ($M = 5$), average scores approach the unmerged LoRA upper bound on all four tested LLMs, while storing roughly half the parameters.","The same shared-direction scheme works for VeRA adapters, reaching 36.4 average score at 22.7% storage compared with 27.8 for the best fixed-size baseline.","Merging across 40 tasks (5 applications × 8 languages), increasing storage from 20% to 28% raises average performance by about 2 points on Llama-1B and over 3 points on Llama-3B.","Across applications, languages, and the full task grid, the pattern is consistent: HydraOpt matches the best fixed-size method at minimum storage and widens the advantage as $M$ grows."],"supporting_citations":[{"why":"Defines LoRA's low-rank update $\\Delta W = BA$, the objects HydraOpt reconstructs.","marker":"Hu et al., 2022"},{"why":"Supplies the observed asymmetry that $B$ parameters differ across tasks while $A$ parameters stay similar, motivating the shared-$A$ design.","marker":"Zhu et al., 2024"},{"why":"Reports that $A$ parameters converge across separately trained LoRA adapters while $B$ parameters differentiate, the empirical basis for sharing $A'$.","marker":"Tian et al., 2024"},{"why":"Provides the Task Arithmetic baseline that HydraOpt must match or beat at equal storage.","marker":"Ilharco et al., 2023"},{"why":"Provides the TIES baseline, the strongest fixed-size merging competitor in several settings.","marker":"Yadav et al., 2024"},{"why":"Provides the DARE baseline that drops and rescales weight changes, compared at the same storage efficiency.","marker":"Yu et al., 2024"},{"why":"Defines VeRA, the second adapter family used to show HydraOpt generalizes beyond LoRA.","marker":"Kopiczko et al., 2024"},{"why":"Motivates the choice of mean absolute error as a sparsity-inducing distance for the reconstruction loss.","marker":"Bach et al., 2012"}],"fun_headline_variants":["Shared A-matrix cuts adapter storage in half, scores dip <2%","HydraOpt merges LoRAs: half the storage, 1.8% max loss","One shared matrix, many adapters: 52% storage, near-full performance","Tunable adapter merging: 48% smaller, less than 2% worse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small average absolute difference between the original and reconstructed adapter matrices means the merged adapter will still perform each task well; this is tested empirically throughout the paper but not proven analytically.","fun_headline_variants_meta":{"raw":{"variants":["Shared A-matrix cuts adapter storage in half, scores dip <2%","HydraOpt merges LoRAs: half the storage, 1.8% max loss","One shared matrix, many adapters: 52% storage, near-full performance","Tunable adapter merging: 48% smaller, less than 2% worse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":2028,"prompt_tokens":989,"completion_tokens":1039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":948}},"tokens_in":605,"tokens_out":1039,"duration_ms":9789,"temperature":1.0,"reasoning_tokens":948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:18:37.575085+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one task's original LoRA adapter and add random noise to its $B$ matrix with the same L1 norm as HydraOpt's reconstruction error for that task; if the noisy adapter loses far more accuracy than the merged adapter, then parameter-space L1 error is not what preserves task behavior, and the reported accuracy retention would be specific to these tasks rather than a general property. A direct check is whether per-task reconstruction error predicts per-task accuracy loss across the 40-task grid.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TIES baseline, the strongest fixed-size merging competitor in several settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines VeRA, the second adapter family used to show HydraOpt generalizes beyond LoRA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the choice of mean absolute error as a sparsity-inducing distance for the reconstruction loss."}],"review_version":1}