{"id":"83bc50ac-e00b-41bf-83c2-850c5ad7b5e7","arxiv_id":"2608.01437","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Expert routing in multimodal continual instruction tuning is reformulated as soft task-as-class class-incremental learning, and a new 34-task fingerprint-reduced benchmark shows that plugging CIL classifiers into routers improves both routing quality and downstream scores.","lead":"This paper shows that expert routing in multimodal continual learning is already near-perfect on common benchmarks because models exploit text templates that leak task identity. It introduces a 34-task benchmark that removes those shortcuts and demonstrates that methods from class-incremental learning can improve routing and final accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CIL-router gains may be confounded with logit-scale sharpening; PureLoRA at τ=80 alone reaches M=56.27, above the best PureLoRA+CIL (56.14), so the MCIL attribution is not isolated.","rationale":"The reader's weakest assumption concerns FLEX's fingerprint reduction. The paper addresses this with multiple internal checks: the suffix-injection ablation (Table 3.8) shows that restoring task-specific suffixes restores near-ceiling matching, and the modality-fusion experiment (Table 3.11) shows that the optimal fusion shifts from text-only on FCIT to balanced on FLEX. These results provide direct evidence that the dominant textual shortcut is weakened, making the FLEX-validity concern partially mitigated. In contrast, the calibration confound is not addressed by any comparable control. The paper explicitly demonstrates that scaling the base router's logit scale on PureLoRA produces downstream gains as large as those from CIL transfer, yet this scaled baseline is absent from the main comparison table. This is a more direct threat to the paper's central empirical claim that CIL routers improve downstream performance. The formulation itself (task-as-class MCIL) is conceptually valid, but its empirical validation is undermined if trivial temperature tuning reproduces the gains. Therefore, the most load-bearing concern is the missing tuned-base control, not the residual-fingerprint question. The verdict remains CONDITIONAL: the paper's claims are plausible and well-documented, but require this additional control to be fully convincing.","tokens_in":36632,"tokens_out":6901,"duration_ms":67330,"concrete_test":"Run a logit-scale sweep for the DISCO, SAME, and PureLoRA base routers on FLEX, holding all frozen experts and routing features fixed, with τ (or equivalent temperature) varied over {20, 28, 40, 80, 160}. Report M, G, and R for each setting, and add the best scaled-base row to Table 2 as a comparison. If the best CIL router's MacroScore gain over the base router is within the improvement achieved by scaling alone (e.g., comparable to PureLoRA's +1.67 at τ=80), then the downstream gains attributed to CIL transfer are not established; if the CIL gains clearly exceed the scaled-base gains, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that transferring CIL classifiers to the routing stage improves both routing quality and downstream MacroScore. However, the paper's own soft-routing calibration experiment (Fig. 5) shows that simply increasing the logit scale τ of the PureLoRA base router from 28 to 80 raises MacroScore from 54.60 to 56.27, which is higher than every PureLoRA+CIL variant (best: RanPAC at 56.14). This means that the downstream gain attributed to the MCIL transfer on PureLoRA is fully reproduced by a trivial temperature change, without any change in the router's ranking (R stays 83.77%). The main results table (Table 2) does not include this tuned-base baseline, so the headline improvement of up to 4.6 MacroScore points may overstate the role of the task-as-class MCIL formulation. The concern is not limited to PureLoRA: S2.4 shows that moving from τ=28 to τ=20 moves PureLoRA closer to DISCO's base, indicating that logit scale is a strong confound across frameworks. For DISCO and SAME, no logit-scale sweep is reported, so we cannot determine whether the DDAS/RanPAC gains (e.g., +4.61 and +4.05 MacroScore) exceed what sharpening alone would achieve. Without this control, the empirical support for the 'principled interface' claim is weaker than it appears.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that expert routing in multimodal continual instruction tuning (MCIT) is saturated on existing benchmarks because of textual fingerprints that leak task identity and short 4–10-task sequences with few competing LoRAs. To address this, it introduces FLEX, a 34-task long-horizon benchmark with weakened textual fingerprints, and reformulates progressive-LoRA routing as soft task-as-class Multimodal Class-Incremental Learning (MCIL). Using a controlled PureLoRA baseline, the authors adapt four CIL methods (HC, HC-SOINN, RanPAC, DDAS) as plug-in routers into four MCIT frameworks, reporting improved strict LoRA matching and MacroScore. The appendix contains extensive ablations: fingerprint injection, pool scaling, task order, seed robustness, backbone transfer, and modality fusion.","tokens_in":37065,"tokens_out":6547,"duration_ms":66900,"significance":"If the quantitative claims hold, FLEX is a valuable diagnostic benchmark and the task-as-class MCIL interface is a useful conceptual bridge between expert routing and continual learning. The paper's strengths include a controlled router-replacement protocol, complete per-task results, a public code release, and a thorough supplementary analysis. The routing-quality improvements (R and G) are substantial and not easily explained by confounds. However, the downstream MacroScore improvements attributed to MCIL transfer are currently confounded with logit-scale sharpening, and the main table lacks repeated runs for most configurations. The practical benefit of the MCIL formulation is therefore plausible but not yet established at the claimed level.","major_comments":[{"comment":"The MacroScore gains attributed to CIL routers are not separated from logit-scale sharpening. For PureLoRA, raising only the logit scale τ from 28 to 80 with fixed experts and prototypes gives M=56.27, higher than the best PureLoRA+CIL variant in Table 2 (RanPAC, M=56.14), while keeping R=83.77 unchanged. S2.4 similarly shows that lowering τ from 28 to 20 moves PureLoRA toward DISCO's Base score. Since RanPAC and DDAS are applied with temperatures 0.03 and 0.2 (S2.5), Table 2 mixes router identity with output sharpness. Please add a calibrated logit-scale Base to Table 2 (e.g., PureLoRA at τ=80) and, for DISCO and SAME, sweep τ/temperature to bound how much of the +1.54–+4.61 MacroScore gains is attributable to the CIL classifier rather than to sharpening. Without this control, the headline downstream claim overstates the role of the MCIL formulation.","section":"Fig. 5 and S2.4"},{"comment":"The main results table reports single-run scores for most of the 20 configurations, with no error bars. Gains as small as +0.09 (HiDe-LLaVA+HC-SOINN) and +0.11 (HiDe-LLaVA+HC) are within plausible random variation. The seed studies in S3.4 cover only HiDe-LLaVA training seeds and RanPAC random-projection seeds; they do not cover DISCO, SAME, HC-SOINN, or DDAS, nor the larger gains on those frameworks. Given that the headline claims depend on small MacroScore differences, the paper should provide at least three seeds for each configuration in Table 2 (or a principled subset with justified selection), and report mean±std for M, G, and R. This is needed to determine whether the observed improvements are stable or partly noise.","section":"Table 2 and S3.4"}],"minor_comments":[{"comment":"The text refers to 'FLEXv2' in the t-SNE visualization caption, although the benchmark is called FLEX. Please correct this inconsistency.","section":"S3.1"},{"comment":"The '(fixed)' tag for DISCO and HiDe-LLaVA rows is not explained until S2.1–S2.2. Since these configurations differ from the official released protocols, add a brief footnote at the first occurrence in the main table.","section":"Table 2"},{"comment":"The hardware description '8 T-Head Zhenwu 810E PPUs' is not standard; please specify the GPU/accelerator type and, if possible, include runtime per task for reproducibility.","section":"Experimental Setup"},{"comment":"The statement that tied UCIT tasks under DISCO are 'shown by their 50/50 soft weights' is unclear. Does this mean the softmax weights are exactly 0.5/0.5, or that the matching metric uses a tie-breaking rule? Please clarify.","section":"Fig. 1"},{"comment":"The definition of g_s uses the subscript notation 'α^{(T)}_{s,i,s}' which is easy to misread. Consider rewriting as g_s = (1/N_s) Σ_i α^{(T)}_{s,i,s} to make the indexing clearer.","section":"Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is competently executed and the FLEX benchmark plus the task-as-class MCIL framing are likely useful to the community. The key issue is attribution: the missing calibrated logit-scale baseline undermines the downstream MacroScore claim, and the lack of repeated runs adds uncertainty. These are fixable with additional experiments and a revised presentation, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contributions here are FLEX and the saturation diagnosis. FLEX is a sensible long-horizon benchmark: it weakens textual fingerprints, adds confusable tasks under shared formats, and makes routing genuinely hard. The 2x2 fingerprint-by-sequence ablation and the expert-pool scaling analysis do a good job showing that both factors matter, not just sequence length. The task-as-class MCIL reformulation is also useful: once you view progressive LoRA routing as soft classification over task identities, a large CIL toolbox becomes available as plug-in routers. That framing is conceptually direct and should be cited. The paper is also careful in several ways: the controlled PureLoRA baseline, the seed-robustness checks, the InternVL transfer, the SAME task-order runs, and the explicit '(fixed)' notes for HiDe-LLaVA and DISCO all show honest engagement. The authors do not hide the calibration experiment, but that experiment is the problem. Figure 5 shows that PureLoRA with tau=80 reaches M=56.27, with the same ranking as the base router. The best PureLoRA+CIL variant, RanPAC, gets 56.14. So the downstream gain attributed to the MCIL transfer is fully reproduced by a trivial temperature change. No reranking, no new router, just a sharper softmax. For DISCO and SAME, no logit-scale sweep is reported, so the +4.61 and +4.05 gains are not isolated from sharpening either. The paper should have included tuned-tau baselines in the main table before claiming the CIL transfer is responsible. Other soft spots are minor by comparison: most main-table configurations are single runs without error bars, and the '(fixed)' protocols for HiDe-LLaVA and DISCO make direct comparison with published implementations messy, even though the appendix explains the corrections. The FLEX fingerprint-reduction assumption is reasonably defended through suffix injection and the modality-fusion shift, though an external fingerprint audit would strengthen it. Bottom line: the benchmark and the reformulation deserve serious attention, but the empirical claim about CIL routers improving downstream performance is overstated. A referee should ask for logit-scale-matched baselines and, ideally, error bars on the main table. I would still send this to peer review.","headline":"Solid benchmark and a useful reformulation, but the headline CIL-router gains are undercut by the paper's own logit-scale calibration: on PureLoRA, simply raising tau from 28 to 80 beats every transferred CIL router.","tokens_in":756,"tokens_out":826,"would_cite":true,"duration_ms":25779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Expert routing in multimodal continual instruction tuning is a soft class-incremental learning problem; reusing continual-learning classifiers as routers improves strict LoRA matching by up to 16.3 percentage points.","keywords":["multimodal continual instruction tuning","expert routing","LoRA","class-incremental learning","task-as-class","benchmark","textual fingerprints","routing saturation"],"falsifier":"Take FLEX and randomly permute the shared answer-format suffixes across tasks, then train and evaluate a text-only router on the same split; if strict matching stays above roughly 90%, residual textual cues rather than the normalized templates are carrying task identity, and the benchmark would not be testing what the paper claims.","tokens_in":36571,"feed_emoji":"🧠","tokens_out":9476,"duration_ms":81547,"temperature":0.7,"pith_summary":"The paper argues that expert routing in multimodal continual instruction tuning is not actually solved: standard benchmarks let routers guess the task from textual fingerprints and short task sequences, so routing looks saturated. To expose this, it builds FLEX, a 34-task benchmark that shares answer-format templates across confusable tasks and weakens identity leakage. It then reframes progressive-LoRA routing as soft task-as-class multimodal class-incremental learning, where each task is a class and the full score distribution sets the LoRA mixture weights. With that interface, four continual-learning classifiers can be plugged into existing routers without touching trained experts, improving strict LoRA matching by up to 16.3 percentage points and MacroScore by up to 4.6 points.","feed_headline":"Routing looks solved only because benchmarks leak task identity","feed_subtitle":"FLEX strips the fingerprints; treating routing as incremental classification lifts expert matching by up to 16.3 points.","key_machinery":"The load-bearing object is the task-as-class MCIL reformulation. In a progressive LoRA pool, task $k$ becomes routing class $k$: the target response is discarded, and every input $x_k^i=(v_k^i,q_k^i)$ is relabelled $c_k^i=k$, turning routing into an expanding $t$-way classification problem. The router output $s_t(x)\\in\\mathbb{R}^t$ is converted into LoRA mixture weights by the temperature-scaled softmax $\\alpha^{\\mathrm{soft}}_{t,k}(x)=\\exp(\\tau s_{t,k}(x))/\\sum_{j=1}^t \\exp(\\tau s_{t,j}(x))$, with hard routing as the $\\arg\\max$ special case. This identity makes 'which LoRA should process this input' the same decision as 'which class does this input belong to', and lets any CIL/MCIL scorer t","core_discovery":"The paper's central claim is that progressive-LoRA expert routing is an expanding task-identification problem, not a solved subproblem. Concretely, each incoming task $k$ defines a routing class $\\widetilde{D}_k=\\{(x_k^i, c_k^i)\\}$ with $c_k^i=k$ (Eq. 3), so after $t$ tasks the router faces a $t$-way classification problem over an expanding class set. The router's score vector is converted into LoRA mixture weights by $\\alpha^{\\mathrm{soft}}_{t,k}(x)=\\exp(\\tau s_{t,k}(x))/\\sum_{j=1}^t \\exp(\\tau s_{t,j}(x))$ (Eq. 7), with hard routing as the $\\arg\\max$ special case. Existing frozen-CLIP prototype routers are, under this view, just nearest-class-mean classifiers over frozen features. On the 34","pith_inferences":["A consequence the paper leaves implicit: if routing is task-as-class MCIL, benchmark difficulty can be engineered independently of task content---adding near-duplicate interface groups or adversarial suffix permutations could make a routing stress test even sharper than FLEX.","The saturation finding implies that some previously reported MCIT gains may reflect better shortcut recovery rather than better continual learning; re-running established methods on fingerprint-reduced sequences could re-rank them.","The task-as-class mapping treats each task as a flat class; a natural next step is hierarchical routing---first choose an interface group (classification, captioning, multiple-choice, VQA), then a task inside it---which would shrink the effective expert pool and may scale better than a 34-way softmax.","Because soft routing weights every expert, routing calibration and cross-task LoRA compatibility could be disentangled by training deliberately confusable task pairs and testing whether a calibrated router learns to share compatible LoRAs."],"forward_implications":["Existing 4–10-task MCIT benchmarks overstate router quality: once textual fingerprints are weakened and the expert pool grows, task-agnostic routing falls far short of oracle routing, so routing should be evaluated on long, interface-shared sequences.","Any CIL/MCIL method that yields comparable scores over an expanding task set can be dropped into an MCIT router; the transferred classifiers here recover 28.3–50.2% of the Base-to-oracle MacroScore gap on the routing-sensitive frameworks.","Soft routing is not just a compromise: with fixed experts and prototypes, a calibrated soft mixture (logit scale ×80) outperforms hard top-1 selection by 2.01 MacroScore points at the same top-1 accuracy, so composition matters after identification.","Routing-quality metrics and downstream scores co-move mainly through top-1 identification (Spearman ρ=0.71), but the cost of a wrong expert depends on cross-task LoRA compatibility; forced transfer shows most confusions hurt, while one (GQA→VQAv2) helps."],"supporting_citations":[{"why":"Defines the progressive task-specific LoRA pool where each incoming task adds one expert, the setting the task-as-class mapping relies on.","marker":"(Yu et al. 2025)"},{"why":"Provides DISCO, a text-only prototype router whose near-saturated matching motivates FLEX and whose first-stage router is later replaced.","marker":"(Guo et al. 2025b)"},{"why":"Provides HiDe-LLaVA, the hierarchical-decoupling framework whose top-layer soft routing and remain-layer fusion are analyzed and enhanced.","marker":"(Guo et al. 2025a)"},{"why":"Provides SAME, a stabilized mixture-of-experts framework whose near-ceiling matching on short benchmarks demonstrates routing saturation.","marker":"(Xie et al. 2026)"},{"why":"Supplies the nearest-class-mean classifier that the paper identifies as equivalent to existing frozen-CLIP prototype routing.","marker":"(Mensink et al. 2013)"},{"why":"Supplies RanPAC, a closed-form random-projection continual classifier adapted as one of the four plug-in routers.","marker":"(McDonnell et al. 2023)"},{"why":"Supplies DDAS, a reconstruction-error-based continual classifier adapted as one of the four plug-in routers.","marker":"(Yu et al. 2024)"},{"why":"Supplies HC and HC-SOINN, prototype-graph class-incremental classifiers adapted as two of the four plug-in routers.","marker":"(Yi et al. 2026)"},{"why":"Supplies temperature-scaled softmax, the mechanism that converts task scores into soft LoRA mixture weights.","marker":"(Guo et al. 2017)"}],"fun_headline_variants":["Routing looks solved only because benchmarks leak task ID","FLEX strips task fingerprints; routing as incremental classes boosts 16.3%","Long-horizon routing is class-incremental; FLEX unmasks 34 tasks","Benchmark leaks fake routing success; FLEX exposes true challenge","Progressive LoRA routing = expanding classification; FLEX proves it"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"FLEX's ability to expose routing difficulty rests on the assumption that normalizing answer-format templates and grouping by interface removes textual task-identity shortcuts without stripping task-necessary content, leaving no residual cues that still leak task identity.","fun_headline_variants_meta":{"raw":{"variants":["Routing looks solved only because benchmarks leak task ID","FLEX strips task fingerprints; routing as incremental classes boosts 16.3%","Long-horizon routing is class-incremental; FLEX unmasks 34 tasks","Benchmark leaks fake routing success; FLEX exposes true challenge","Progressive LoRA routing = expanding classification; FLEX proves it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00044,"raw_usage":{"total_tokens":2149,"prompt_tokens":904,"completion_tokens":1245,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1150}},"tokens_in":648,"tokens_out":1245,"duration_ms":8930,"temperature":1.0,"reasoning_tokens":1150,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:09:35.433857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take FLEX and randomly permute the shared answer-format suffixes across tasks, then train and evaluate a text-only router on the same split; if strict matching stays above roughly 90%, residual textual cues rather than the normalized templates are carrying task identity, and the benchmark would not be testing what the paper claims.","supporting_citations":[],"review_version":1}