{"id":"a86124b0-17f7-4598-be8e-966abce2fb36","arxiv_id":"2601.13020","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Routing MoE-LoRA experts by the strength of their low-rank activation and regularizing important rank directions reduces forgetting in continual instruction tuning of multimodal LLMs.","lead":"This paper proposes a way for an AI model that keeps learning new image-and-text tasks to avoid forgetting old ones, by choosing and protecting small 'expert' adapters based on which internal pathways they activate. If it works, it would make continual fine-tuning of multimodal assistants cheaper and more stable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 5.1% AP gain may be an artifact of unequal parameter budgets: PASs-MoE removes the MoELoRA router, and the paper does not match total trainable parameters in the comparisons.","rationale":"I identified a different load-bearing concern than the reader. The reader's weakest_assumption concerns the calibration of activation statistics as a proxy for expert compatibility; that is a plausible conceptual issue and is acknowledged in the Limitations. However, the immediate threat to the central empirical claim is that the experimental comparison does not hold total trainable parameters fixed. The proposed method removes the router used in MoELoRA, so the comparison is not apples-to-apples. Since the paper's headline result is a 5.1% AP improvement, and the ablation attributes this to PASs-RW, the router removal is a confound that could explain the entire gain. This is directly testable and should be resolved before the method can be credited with the improvement. Therefore the verdict should remain conditional, but with an additional experimental condition: parameter-matched comparison. I disagree with the reader's weakest_assumption because the proposed test targets this confound, not the calibration of the energy signal.","tokens_in":15680,"tokens_out":10370,"duration_ms":114528,"concrete_test":"Re-run the softmax MoELoRA baseline and PASs-MoE with identical total trainable parameter counts. One way: in MoELoRA, replace the learned router with a fixed (untrained) softmax over the same energy scores and reallocate the removed router parameters to increase the LoRA rank r of all experts (or add an equal-size residual module) such that the parameter totals match. If the AP gap of 5.1% shrinks or vanishes, the reported gain is not attributable to PASs routing. Alternatively, restore a learned router in PASs-MoE by adding a scalar bias per expert to the energy scores (training only those biases) so that the parameter count matches the baseline; if performance drops toward the baseline, the benefit was not from the energy-based routing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—PASs-MoE improves final AP and BWT over MoE-LoRA baselines—is supported by comparisons in Tables 1–3 and ablations in Tables 2 and 7. In all of these, the MoELoRA baselines include a learned router (a linear layer over hidden states), whereas PASs-MoE (PASs-RW) replaces the router with a deterministic softmax over expert activation energies s_e = (1/r)||A_e h||^2 and consequently has no router parameters. The paper states a 'fixed MoE-LoRA parameter budget' (Section 5.3) and 'without increasing model parameters' (abstract), but the budget is matched only at the level of the expert pool (E experts, rank r); the router parameters are not accounted for. Since the router exists in every transformer layer (34+ layers in LLaVA-7B), its removal frees a nontrivial number of parameters. The improvement in AP/BWT could therefore be due entirely to this parameter reduction (e.g., less overfitting or a simpler optimization landscape) rather than the proposed energy-based routing or the rank stabilization. The ablation in Table 2 conflates the removal of the router with the addition of PASs-RW. The Limitations do not mention this confound, despite acknowledging calibration issues with the activation proxy. This makes the empirical support for the central claim unconvincing without a parameter-matched re-evaluation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PASs-MoE, a continual instruction tuning method for MLLMs under a fixed-capacity MoE-LoRA setting. The method defines a Pathway Activation Subspace (PASs) as span(A_e^T) from each LoRA down-projection, and uses the low-rank activation energy s_e=(1/r)||A_e h||^2 to compute routing weights (PASs-RW), replacing the learned router. A second component, PASs-aware Rank Stabilization (PASs-RS), computes rank-level importance I_{e,k}=E[π_e(h)(a_{e,k}^T h)^2] and applies a weighted quadratic penalty on changes of the corresponding LoRA rows. Experiments on MLLM-CTBench with LLaVA-1.5-7B report that the method outperforms MoE-LoRA baselines and standard continual learning methods in both AP and BWT, with a claimed 5.1% AP improvement over the second-best method. Ablations and analyses on routing stability, importance sparsity, and update-vs-importance patterns are provided.","tokens_in":16132,"tokens_out":5048,"duration_ms":58304,"significance":"If the empirical claims hold, the contribution is conceptually interesting and practically useful: routing is tied to the expert's low-rank pathway rather than a separately learned router, and the same activation signal is reused for rank-level regularization. The method is clearly specified, the code is promised, and the paper includes a random-regularization ablation and several diagnostic plots that go beyond a simple accuracy comparison. However, the force of the empirical claim is weakened by the fact that the MoE-LoRA baselines include a learned router while PASs-RW removes it, so the 'fixed MoE-LoRA parameter budget' is not actually matched; by the use of a single random seed; and by hyperparameters λ and E selected on the same benchmark and reported at their best. These issues must be addressed before the claimed improvements can be attributed to the proposed components.","major_comments":[{"comment":"The central comparison is not parameter-matched. The MoELoRA (softmax) baseline contains a learned router linear layer; PASs-RW removes that layer entirely, so the comparison conflates the routing mechanism with a reduction in trainable parameters. The text claims a 'fixed MoE-LoRA parameter budget' without accounting for router parameters. Please either (a) add a baseline that keeps the router (or an equal-size linear layer) while using the PASs energy for reweighting, or (b) increase the expert rank of the MoE-LoRA baseline so that total trainable parameters match PASs-MoE, and re-run the main tables. Without such a matched comparison, the 5.1% AP gain in Table 1 cannot be attributed to the proposed components.","section":"Section 5.3, Tables 2 and 3"},{"comment":"All experiments are reported for a single fixed random seed (42), and no standard deviations or significance tests are given. For a continual learning paper that claims consistent improvements, this is insufficient. Please report mean±std over at least three seeds for the key comparisons (Tables 1, 2, and the Order-B results in Table 6), and state whether the improvements are statistically significant. This is particularly important because the hyperparameters λ and E appear to be tuned on the same benchmark.","section":"Section 5.1, Tables 1–6"},{"comment":"The hyperparameters λ and E are selected on MLLM-CTBench and the paper reports the best configuration (E=6, λ=5e-4). Since Fig. 4 shows a non-monotonic AP curve and Table 3 shows non-monotonic behavior in E, the reported gains may reflect selection on the test set. Please provide a validation-split or a robustness analysis (e.g., a small grid of λ and E) showing that the conclusion is not an artifact of choosing the best values on the test benchmark. At minimum, disclose that these values were chosen on the test set.","section":"Section 5.3.2, Fig. 4 and Table 3"},{"comment":"The evaluation benchmark MLLM-CTBench is cited as Guo et al. 2025b, which includes several of the same authors as this paper (Haiyun Guo, Jinqiao Wang). Evaluating on a self-authored benchmark without external validation is a concern, especially since the paper compares to many methods from other groups. Please either provide the exact splits and evaluation scripts to allow third-party reproduction, or supplement the main evaluation with an external benchmark (e.g., COIN or another established multimodal CIT benchmark) to demonstrate generality. The current paper does not mention this overlap or address potential bias.","section":"Section 5.1 and References (MLLM-CTBench)"}],"minor_comments":[{"comment":"The phrase 'without increasing model parameters' is technically true but potentially misleading: PASs-MoE actually has fewer trainable parameters than MoE-LoRA because the router is removed. Please rephrase to 'with no additional parameters beyond the existing expert pool' or 'with a fixed parameter budget excluding the router', and clarify the router parameter counts.","section":"Abstract and Section 5.2"},{"comment":"The claim 'our final AP exceeds that of the second-best method by 5.1%' refers to an absolute difference of 5.10 percentage points (48.46 vs 43.36). Please use '5.1 points' or '5.1 percentage points' to avoid ambiguity.","section":"Section 5.2"},{"comment":"There is a typo in the Radford et al. reference: 'PmLR' should be 'PMLR'. Also, 'HiDe-llava' in Tables 1 and 6 should be 'HiDe-LLaVA' for consistency.","section":"References"},{"comment":"The caption says 'Best and second-best results for Acc and AP' but the table reports both Acc and Forget. Please clarify which columns the bold/underline refers to (likely Acc and AP).","section":"Table 1 caption"},{"comment":"The stability metric is defined as '−MeanJS ↑' but the precise computation (which layers, which inputs, how the gating distributions are compared) is not fully described in the main text. Please add a short explanation or refer to the appendix.","section":"Fig. 3"},{"comment":"Table 6 presents results under Order-B but has the same caption as Table 1. Please retitle it as 'Results under the alternative task order (Order-B)' to avoid confusion.","section":"Appendix A.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The central technical idea is sound and the paper is clearly written, but the empirical evaluation needs to be substantially strengthened. The parameter-budget confound is the most serious issue: a reviewer cannot currently tell whether the reported gains come from removing the learned router or from the proposed PASs-RW/RS components. Please also note the self-authored benchmark; while that is not disqualifying, the paper should disclose it and ideally add an external benchmark. I recommend major revision and would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and sensible: use the norm of the LoRA down-projection activation as a router signal, and regularize per-rank directions by accumulated activation importance. That combination is not in the cited MoE-LoRA works, and it's cleanly motivated by the factorization of LoRA as B(Ah). The paper does a decent job of showing that both components matter: the ablation in Table 2 and the random-regularization comparison in Table 7 support the importance signal, and the alternative task order in the appendix is a good check. The limitations section is candid about the proxy nature of the activation energy and the noise in importance estimates. That's real credit.\n\nThe soft spots are empirical, not conceptual. The largest is the parameter-budget confound. The MoELoRA baselines include a learned router; PASs-RW removes it. The paper claims a fixed MoE-LoRA parameter budget, but the router parameters are not accounted for. So the comparison is not parameter-matched, and the ablation in Table 2 conflates removing the router with adding the new routing signal. The router is small relative to the LoRA parameters, so the effect is probably modest, but it is a genuine confound and the paper should either match trainable parameters (add a dummy router or bump LoRA rank) or explicitly argue why the router size is negligible. The paper doesn't do that.\n\nSecond, all numbers come from a single seed, with no standard deviations or significance tests. The hyperparameters lambda and E are tuned on the evaluation benchmark itself, and the benchmark is co-authored by the same group. That combination weakens the headline 5.1% AP gain. These are addressable with multiple seeds and ideally an independent benchmark, but as is, the empirical support is thinner than the abstract suggests.\n\nThe conceptual assumption—that activation energy tracks expert specialization rather than input scale or outlier directions—is acknowledged in the limitations, and I buy that it is a real limitation rather than a fatal flaw. The method is internally coherent and the ablations are informative. I don't see a load-bearing mathematical error.\n\nWho this is for: people working on MoE-LoRA for continual instruction tuning. It's a within-subfield improvement, not a paradigm shift. But it deserves a serious referee. I'd send it to peer review with the request that the authors fix the parameter budget comparison and report multiple seeds. The idea is worth engaging with; the current evidential base just needs to be tightened.","headline":"A plausible self-routing MoE-LoRA method with a real confound in the parameter budget and thin statistics; worth refereeing with requests for matched comparisons and multiple seeds.","tokens_in":16520,"tokens_out":2344,"would_cite":true,"duration_ms":27884,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In fixed-capacity MoE-LoRA continual instruction tuning, replacing the learned router with each expert's own low-rank activation energy and stabilizing historically important rank directions reduces forgetting and lifts final average accura","keywords":["continual instruction tuning","mixture of experts","LoRA","pathway activation subspace","catastrophic forgetting","routing","multimodal large language models","rank stabilization"],"falsifier":"Take a trained MoE-LoRA and artificially rescale one expert's A rows or its input so that activation energies rise without changing the expert's output function; if PASs-RW routes more inputs to that expert despite no change in capability, the energy proxy is tracking scale, not specialization. Concretely, compute the correlation between s_e(h) and input norm on held-out examples: if high-energy experts are simply the ones with larger input norms, the assumption fails.","tokens_in":15613,"feed_emoji":"🧠","tokens_out":5432,"duration_ms":58139,"temperature":0.7,"pith_summary":"The paper argues that in fixed-capacity MoE-LoRA continual instruction tuning, the router and experts drift in misaligned ways: the router reroutes old inputs while expert updates change how experts respond, blurring task specialization and worsening forgetting. It proposes to drop the learned router entirely and derive routing from the activation energy of each expert's own low-rank pathway, defined via the pathway activation subspace (PASs) spanned by the LoRA down-projection. A second component, PASs-aware rank stabilization, uses activation-based importance scores to selectively freeze the rank directions that previous tasks depended on, while leaving others free to adapt. On a seven-task multimodal continual benchmark, the method beats the strongest comparison by 5.1% average accuracy and roughly halves forgetting, without adding any parameters. The contribution is a reproducible recipe: one subspace signal for both routing and preservation.","feed_headline":"Router-free LoRA experts cut forgetting by two-thirds","feed_subtitle":"Using each expert's own activation pathway to route and protect rank directions lifts final accuracy by 5.1% without added parameters.","key_machinery":"Pathway activation subspace (PASs): for each LoRA expert e, S_e = span(A_e^T), the row space of the down-projection A_e. The low-rank response z_e = A_e h is an r-dimensional vector of activation coefficients, one per rank direction. From it the paper derives two signals: activation energy s_e = (1/r)||A_e h||^2 for softmax routing (PASs-RW), and rank importance I_{e,k} = E[π_e(h)(a_{e,k}^T h)^2] for weighted stabilization of the rows of A (PASs-RS). The same subspace thus anchors both routing and preservation, eliminating the separate learned router.","core_discovery":"The paper's central claim is that the low-rank activation A_e h of a LoRA expert already encodes enough information to do both routing and forgetting protection, so the separate learned router is not just unnecessary but actively harmful under sequential training. The pathway activation subspace S_e = span(A_e^T) is the row space of the down-projection; the low-rank response z_e = A_e h gives a capability-aligned coordinate system. PASs-RW sets expert mixture weights to softmax of (1/r)||A_e h||^2, tying routing to each expert's functional response. PASs-RS accumulates importance I_{e,k} = E[π_e(h)(a_{e,k}^T h)^2] over past tasks and adds weighted L2 stabilization on rows of A to protect his","pith_inferences":["(Editorial inference) If activation energy genuinely tracks task-relevant specialization, the same signal could serve as a diagnostic: inspecting which experts and rank directions fire on a new task could predict how much forgetting that task will cause before training begins.","(Editorial inference) Router-free routing simplifies deployment because inference no longer needs a separately stored routing network; routing becomes a deterministic function of the expert parameters themselves.","(Editorial inference) The core idea may transfer to other parameter-efficient fine-tuning forms (adapters, prefix tuning) whenever a parameter-induced subspace and activation norm can be defined, though the paper notes that extending it would require redefining the coordinate system.","(Editorial inference) A cleaner ablation would isolate the two components against a single-LoRA baseline and against PASs-RS alone; the paper's ablations always combine both on top of a softmax MoE baseline, so the individual contribution of each to forgetting reduction is not fully separated."],"forward_implications":["Learned routers can be replaced by activation-based self-routing in fixed-capacity MoE-LoRA, removing a source of drift and adding no parameters.","Selective rank-level stabilization guided by activation importance yields better retention than uniform or random regularization of LoRA parameters.","The method keeps the same fixed expert set across all tasks, so model capacity does not grow with the task stream.","The gains are not tied to a single task ordering: a reversed task sequence still improves average performance by 9.46% over the second-best baseline."],"fun_headline_variants":["Pathway activations replace router, cutting forgetting in MoE-LoRA","LoRA expert activation pathways guide routing and preserve old tasks","Use expert activation subspaces to stop co-drift and forgetting","Routing via LoRA activations cuts forgetting and boosts accuracy","PASs-MoE: expert pathways replace router to halt forgetting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Routing and importance both rely on the premise that the magnitude of A_e h reflects which expert (and which rank direction) genuinely specializes for that input — if activation energy is dominated by input scale, task hardness, or outlier directions rather than by learned specialization, both the reweighting and the stabilization will target the wrong places.","fun_headline_variants_meta":{"raw":{"variants":["Pathway activations replace router, cutting forgetting in MoE-LoRA","LoRA expert activation pathways guide routing and preserve old tasks","Use expert activation subspaces to stop co-drift and forgetting","Routing via LoRA activations cuts forgetting and boosts accuracy","PASs-MoE: expert pathways replace router to halt forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2337,"prompt_tokens":835,"completion_tokens":1502,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1423}},"tokens_in":579,"tokens_out":1502,"duration_ms":13004,"temperature":1.0,"reasoning_tokens":1423,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:38:45.784122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained MoE-LoRA and artificially rescale one expert's A rows or its input so that activation energies rise without changing the expert's output function; if PASs-RW routes more inputs to that expert despite no change in capability, the energy proxy is tracking scale, not specialization. Concretely, compute the correlation between s_e(h) and input norm on held-out examples: if high-energy experts are simply the ones with larger input norms, the assumption fails.","supporting_citations":[],"review_version":1}