{"id":"008bff80-13eb-4363-bb90-ee48b21333bb","arxiv_id":"2508.02978","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Shared and domain-specific LoRAs are constrained to the column and left null spaces of pretrained weights, but experimental benefits are mixed.","lead":"A team proposes splitting multi-domain adapters into two orthogonal subspaces: shared LoRAs update along the pretrained weights' main directions, while domain-specific LoRAs update in the leftover low-variance directions. In tests on three action recognition datasets the separation is mathematically exact but only gives small, inconsistent accuracy gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed left-null-space guarantee is not exact: pretrained ViT weights are full-rank, so the 'left null space' is a truncated tail of W, not a true null space; non-interference is an empirical assumption, not a construction.","rationale":"The reader's weakest_assumption already identifies the truncation that manufactures a left null space and the empirical risk that low-variance directions may be important. My analysis sharpens this into a mathematical point: because the pretrained weights are full rank, the true left null space is empty or contains no used directions, so the phrase 'left null subspace of the pre-trained weights' is not literally correct. The method instead constructs an approximate tail subspace, and the claimed guarantee of non-interference with pretrained knowledge is an empirical spectral assumption. This is the most load-bearing concern because it targets the paper's central theoretical contribution rather than its empirical evaluation. The empirical section is honestly reported as mixed, and the method remains well-defined if reframed as an SVD-tail constraint, so the concern does not by itself warrant rejection. It does reinforce the conditional verdict: the paper should be revised to state the approximation explicitly, justify the threshold with quantitative residual checks, and test whether the observed gains depend on the tail being treated as null.","tokens_in":11114,"tokens_out":9311,"duration_ms":117809,"concrete_test":"For each linear layer that receives the proposed constraint (projection, MLP Linear1, MLP Linear2), compute R = ||W U_n||_F / ||W||_F, where U_n is the basis of the 'left null space' determined by the 95% cumulative-contribution threshold. If R is not close to 0 (e.g., >1%), the domain-specific LoRA is not in a null space of W, so the non-interference statement is unsupported. Report R per layer and re-run Table 1 with a threshold chosen so that R is below a small tolerance (or with the true Ker(W^T) for layers where it is nonempty) to see whether the observed gains survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract and Section 3.2) is that shared LoRAs live in col(W) and domain-specific LoRAs live in Ker(W^T), which 'by construction' prevents interference. But W is a full-rank pretrained weight; for square or underdetermined layers, Ker(W^T) is {0}, and for all layers the true left null space contains no information-bearing directions. The method in Section 3.1.2 therefore truncates the SVD at a 95% cumulative contribution threshold and renames the tail span U_n as the left null space. These tail directions are not null: W U_n has nonzero (if small) singular values, so a domain-specific LoRA confined to P_n still changes the model's output along directions the pretrained network actively uses. The claimed guarantee that domain-specific adaptation 'does not affect the features obtained from W' holds only to the extent that the discarded 5% of squared singular-value energy is semantically irrelevant. The paper itself notes results are threshold-dependent (Section 4.2.3, Table 3), with the lowest threshold best for HMDB51, which is hard to reconcile with the assertion that larger column spaces better preserve pretrained knowledge. Thus the central theoretical virtue is not established by construction; it rests on an unvalidated spectral assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for multi-domain learning with LoRA adapters that explicitly separates shared and domain-specific information by constraining shared LoRA updates to the column space of each pretrained weight matrix W and domain-specific LoRA updates to a subspace treated as the left null space of W, using projection matrices derived from SVD. It further introduces two auxiliary losses (Lorth and Lss) to encourage orthonormality and separation among domain-specific LoRA bases. The method is evaluated on action recognition across UCF101, Kinetics400, and HMDB51 with a frozen ImageNet-21K-pretrained ViT, using two integration structures (upper-heavy and all-flat). Results show small and inconsistent accuracy differences compared to unconstrained variants, and an analysis of the effective dimensions of LoRA weights is presented.","tokens_in":11443,"tokens_out":4032,"duration_ms":41423,"significance":"The core idea is attractive: if shared and domain-specific LoRA updates could be confined to mutually orthogonal subspaces by construction, it would provide a principled way to prevent interference in multi-domain PEFT. The gradient projection derivation in Eqs. (5)-(7) is correct, and the projection matrices are derived from the frozen pretrained weights (not learned), which is a strength. However, the paper's central 'by construction' claim is not exact, because the pretrained weights are full-rank and the method actually uses a truncated SVD tail rather than a true left null space. Moreover, the experimental evidence is not yet sufficient to establish effectiveness: no multiple seeds or variance are reported, differences are within a few tenths of a point with mixed signs, and no existing multi-domain PEFT methods are included as baselines. The paper is honest about these limitations (Sections 3.1.1 and 4.2.3), but as it stands the main theoretical virtue is an unvalidated spectral assumption and the empirical support is fragile.","major_comments":[{"comment":"The central claim that domain-specific LoRAs are confined to Ker(W^T) and therefore 'do not affect the features obtained from W' is not exact as stated. For full-rank pretrained weight matrices, the true left null space is trivial (for square or overdetermined layers), so the method in Section 3.1.2 truncates the SVD at a 95% cumulative contribution threshold and renames the tail span U_n as the left null space. The discarded directions have nonzero singular values, so W U_n is not zero and a domain-specific LoRA in U_n does change the model output along directions the pretrained network actively uses. The claimed non-interference guarantee holds only to the extent that the discarded 5% of squared singular energy is semantically irrelevant, which is an empirical assumption, not a construction. The paper's own Table 3 shows that for HMDB51 the lowest threshold (0.80) gives the best performance, which is hard to reconcile with the rationale that a higher threshold better preserves pretrained knowledge. The authors should either prove a bound on the interference, empirically validate the spectral assumption, or re-frame the contribution as an approximate separation rather than an exact one.","section":"Section 3.1.2 and Section 3.2"},{"comment":"The empirical support for effectiveness is weak and inconsistent. In Table 1, with the upper-heavy structure, adding subspace constraints improves UCF101 by only +0.19 and leaves HMDB51 unchanged, while degrading Kinetics400 by -0.29; adding both constraints improves HMDB51 by +0.40 but degrades UCF101 by -0.95 and Kinetics400 by -0.33. In the all-flat structure, the proposed constraints mostly hurt performance. No multiple seeds, standard deviations, or significance tests are reported, so these differences may be within noise. Furthermore, there is no comparison to MTLoRA, VMT-Adapter, or any other existing multi-domain PEFT method; the paper only compares variants of its own method. Without stronger experiments, the claim that the proposed separation is effective in practice is not established.","section":"Section 4.2, Tables 1-3"},{"comment":"The separation loss Lss = - (1/sqrt(2)) sum_{i<j} ||B_i B_i^T - B_j B_j^T||_F is proposed to push domain-specific subspaces apart, but the paper provides no analysis of how this loss interacts with the orthonormality loss Lorth or with the scale of the B_i matrices. Since ||B_i B_i^T||_F is not normalized, the negative distance could be trivially affected by scaling, and the reported configuration (lambda1=1, lambda2=1e-7) is the only one tested. Table 1 does not isolate the effect of Lss from the subspace projection, so it is unclear whether the separation loss contributes to the observed results. A sensitivity study or a proof that the loss has the intended geometric effect is needed before the domain-separation claim can be accepted.","section":"Section 3.3, Eq. (9)"}],"minor_comments":[{"comment":"The sentence 'Then we introduce constrains the LoRA weights...' contains a grammatical error; 'constrains' should be 'constraints'.","section":"Section 3.2"},{"comment":"The text after Eq. (9) refers to 'Bd1 and Bd2' but the formula uses 'Bi and Bj'; please unify the notation.","section":"Section 3.3, Eq. (9)"},{"comment":"The text says 'The curve colors show three domains (UCF100, Kinetics400, and HMDB51)' but the dataset is UCF101; please correct the typo.","section":"Section 4.2.4"},{"comment":"Reference [10] has 'yelong shen' in lowercase; please fix the capitalization to 'Yelong Shen'.","section":"References"},{"comment":"The text states 'In general, performance improves as r increases' but Table 2 shows HMDB51 peaking at r=16 and declining at r=32; please clarify this statement to account for the non-monotonic behavior.","section":"Section 4.2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is early-stage and the authors are transparent about its limitations (e.g., Section 3.1.1 acknowledges the left-null-space approximation, and Section 4.2.3 flags the threshold anomaly). However, the abstract and Section 3.2 present the non-interference property as a construction guarantee, which is not accurate under the truncated-SVD implementation. I recommend major revision with a request to either prove a rigorous bound or soften the claim, and to strengthen the empirical evaluation with multiple seeds and at least one existing PEFT baseline. The idea has potential, but in its current form the evidence does not meet the bar for the central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new subspace-orthogonality idea for multi-domain LoRA, presented honestly, but the central 'by construction' claim is approximate and the experiments are too thin to carry it. I'd send it out, but expect heavy revision.\n\nThe new thing is the constraint itself: project the shared LoRA onto the column space of the pretrained weight W and each domain-specific LoRA onto the complementary tail of the left singular vectors, then add a TesNet-style orthogonality loss between domain subspaces. The gradient derivation (Eqs. 5–7) checks out, and because P_m and P_n are orthogonal projections, the updates do stay in their respective subspaces. That's a clean, testable mechanism that I haven't seen in MTLoRA or VMT-Adapter. The paper also deserves credit for looking at the effective rank of the LoRA weights (Fig. 4) rather than just accuracy.\n\nThe main soft spot is that the 'left null space' isn't one. The pretrained ViT weights are full rank, so the true Ker(W^T) is either {0} or contains nothing of interest. The method truncates the SVD at 95% cumulative energy and renames the tail U_n as the left null space. Those tail directions still have nonzero singular values, so a domain-specific LoRA confined to them still changes the output of W, just less than the principal directions do. The paper actually acknowledges this in a parenthetical in Sec. 3.1.1, but the abstract and Sec. 3.2 state a hard guarantee. That overstatement matters because the threshold-dependence results (Table 3) show HMDB51 does best with the smallest column space, which cuts against the story that the column space holds the domain-agnostic knowledge. The authors flag this as needing investigation, which is good, but it means the central interpretability claim is not yet supported.\n\nThe experiments are the other weak point. Gains in Table 1 are ~0.2–0.4 accuracy points on UCF and HMDB, with a ~2-point drop on Kinetics. No multiple seeds, no error bars, no comparison to a real MTLoRA baseline (the 'upper-heavy, no constraints' row is not equivalent to MTLoRA as published), and no external state-of-the-art numbers. The hyperparameters are tuned on the same benchmarks. The paper is transparent about all this, but as a result it reads as an idea paper with a proof-of-concept, not a validated method.\n\nWho should read this: anyone working on multi-domain PEFT or on interpretable adapter subspaces. It's a plausible building block. I'd send it to reviewers, but the revision needs to fix the overclaim, add parameter-matched baselines and variance, and sort out the threshold behavior.","headline":"Clean subspace-orthogonality idea for multi-domain LoRA, but the 'left null space' guarantee is approximate and the empirical support is too thin for full acceptance.","tokens_in":11934,"tokens_out":3859,"would_cite":false,"duration_ms":41680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that forcing the shared LoRA into the column space of the pretrained weights and each domain-specific LoRA into the left null space guarantees the two kinds of adapters learn different information, with small and mixed…","keywords":["multi-domain learning","low-rank adaptation","LoRA","subspace separation","column space","left null space","truncated SVD","action recognition"],"falsifier":"Train the same upper-heavy multi-domain model with the projections swapped: domain-specific LoRAs confined to the strong column-space directions of $W$ and the shared LoRA confined to the left null space, then compare UCF101, Kinetics400, and HMDB51 accuracy against the paper's configuration. If the swapped assignment does not clearly degrade accuracy, the paper's claim that shared knowledge lives in $\\mathrm{col}(W)$ and domain-specific knowledge in $\\mathrm{Ker}(W^T)$ is unsupported.","tokens_in":10933,"feed_emoji":"🧩","tokens_out":12023,"duration_ms":117984,"temperature":0.7,"pith_summary":"This paper tries to establish that in multi-domain fine-tuning with LoRA, the shared and domain-specific adapters can be made structurally distinct rather than only trained on different data. The proposed method pins the shared LoRA updates to the column space of each frozen pretrained weight and pins each domain's LoRA updates to the corresponding left null space, obtained by truncating the singular value decomposition at a 95% cumulative contribution threshold. On action recognition with a frozen ImageNet-pretrained Vision Transformer, the constraints give small gains in some configurations (UCF101 92.69 to 92.88 with subspace constraints; HMDB51 66.05 to 66.45 with subspace constraints plus domain separation) while slightly lowering accuracy on other datasets. The authors present the result as a structural guarantee worth further investigation, not as a new accuracy record.","feed_headline":"Shared and domain LoRAs are forced into separate subspaces","feed_subtitle":"Truncated SVD at 95% routes shared LoRA to the column space and domain LoRA to the left null space.","key_machinery":"The central object is the truncated singular value decomposition of each frozen pretrained linear weight $W = U\\Sigma V^{\\top}$, with the cutoff chosen so that the top singular vectors accumulate 95% of the squared singular values and form the column space $\\mathrm{col}(W)$; the remaining singular vectors span the manufactured left null space $\\mathrm{Ker}(W^{\\top})$. Projection matrices $P_m = U_m U_m^{\\top}$ and $P_n = U_n U_n^{\\top}$ are inserted into the forward pass, so that a layer computes $h = Wx + P_m\\Delta W x + P_n\\Delta W_i x$. Because the gradient of each LoRA weight is multiplied by the same symmetric projection, every parameter update stays inside its assigned subspace. A second mechanism is a pair of losses on each domain-specific factor $B_i$: an orthonormality loss $\\lVert B_i^{\\top}B_i - I_r\\rVert_F^2$ and a subspace-separation loss that maximizes the Frobenius distance between $B_iB_i^{\\top}$ and $B_jB_j^{\\top}$, pushing different domains onto different bases within the left null space.","core_discovery":"The central claim is that the ambiguity of shared-plus-domain-specific LoRA architectures—where a shared adapter trains on all domains and a domain-specific adapter trains on one domain, without any explicit difference in what they are allowed to represent—can be removed by construction. The paper claims that restricting the shared LoRA $\\Delta W$ to the column space $\\mathrm{col}(W)$ of the pretrained weight and each domain-specific LoRA $\\Delta W_i$ to the left null space $\\mathrm{Ker}(W^T)$ makes the two types of updates mutually orthogonal, so the shared module cannot silently absorb domain-specific information and the domain-specific module cannot overwrite pretrained knowledge. Because the pretrained weights are full rank, the left null space is manufactured by truncated SVD: singular directions below a 95% cumulative contribution are treated as zero and assigned to the domain-specific side. A further pair of losses pushes each domain's LoRA basis toward orthonormality and away from every other domain's basis. The empirical claim is that this enforced separation helps in some action-recognition configurations and hurts in others, and that the LoRA dimension analysis confirms the separation changes how the adapters use their rank.","pith_inferences":["The same subspace-splitting recipe could be applied to other parameter-efficient adapters, such as residual adapters or prefix vectors, whenever a pretrained weight supplies a column space; the paper does not test these settings.","A direct check of the load-bearing assumption would be to swap the projection: confine domain-specific LoRAs to the top principal directions and the shared LoRA to the low-variance directions; if accuracy does not clearly fall, the paper's identification of which subspace carries shared knowledge is wrong.","The static-bias motivation suggests a testable extension: measure whether domain-specific LoRAs confined to the left null space reduce a model's reliance on static scene features in action recognition compared with unconstrained LoRAs.","As the number of domains grows, the fixed left null space dimension will eventually saturate; the paper's own dimension curves imply that the useful width of each domain-specific LoRA shrinks with more domains, so a scaling test with ten or more domains would clarify the approach's practical ceiling."],"forward_implications":["If the orthogonality guarantee holds, adding or updating a domain-specific LoRA cannot change the shared representation, which should reduce interference in multi-domain training.","The cumulative-contribution analysis of LoRA weights gives a diagnostic: with subspace constraints alone, domain-specific LoRAs concentrate in fewer effective dimensions, and adding the separation losses spreads them across all 32 rank dimensions.","The 95% truncation threshold becomes a tunable hyperparameter with measurable consequences; on HMDB51, threshold 0.80 outperforms 0.95 by more than one accuracy point.","The method scales spatially: with the left null space roughly 20–80% of each layer's dimension and about 600 dimensions available at rank 32, three domains are far from exhausting the space, so more domains could be separated without extra parameters beyond one LoRA per domain.","The accuracy gains reported are small and inconsistent, so the paper's contribution is best read as a structural separation guarantee plus an analysis method, not as a large performance improvement over prior multi-domain fine-tuning."],"supporting_citations":[{"why":"supplies the shared-plus-domain-specific LoRA structure that the proposed constraints extend.","marker":"[1]"},{"why":"defines the low-rank adapter form that the method constrains and projects.","marker":"[10]"},{"why":"provides the orthonormality and subspace-separation losses used to separate LoRAs of different domains.","marker":"[33]"},{"why":"supplies the truncated SVD technique used to manufacture a left null space from the full-rank pretrained weights.","marker":"[40]"},{"why":"provides the Vision Transformer architecture whose frozen linear layers are decomposed.","marker":"[5]"},{"why":"supplies the ImageNet21K pretraining of the backbone used in all experiments.","marker":"[25]"},{"why":"one of the three action-recognition evaluation datasets, Kinetics400.","marker":"[12]"},{"why":"one of the three action-recognition evaluation datasets, HMDB51.","marker":"[14]"},{"why":"one of the three action-recognition evaluation datasets, UCF101.","marker":"[28]"}],"fun_headline_variants":["Shared and domain LoRAs get orthogonal subspaces","Truncated SVD splits LoRA subspaces","LoRAs forced into column and null subspaces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that the singular directions of the pretrained weight $W$ below the 95% cumulative contribution cutoff carry no useful pretrained knowledge, and that all domain-agnostic information lives in the strong column space; if either part fails, the projection discards useful adaptations instead of separating them.","fun_headline_variants_meta":{"raw":{"variants":["Shared and domain LoRAs get orthogonal subspaces","Truncated SVD splits LoRA subspaces","LoRAs forced into column and null subspaces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2949,"prompt_tokens":887,"completion_tokens":2062,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":503,"tokens_out":2062,"duration_ms":17848,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:45:25.090535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same upper-heavy multi-domain model with the projections swapped: domain-specific LoRAs confined to the strong column-space directions of $W$ and the shared LoRA confined to the left null space, then compare UCF101, Kinetics400, and HMDB51 accuracy against the paper's configuration. If the swapped assignment does not clearly degrade accuracy, the paper's claim that shared knowledge lives in $\\mathrm{col}(W)$ and domain-specific knowledge in $\\mathrm{Ker}(W^T)$ is unsupported.","supporting_citations":[{"cited_title":"MT- LoRA: Low-rank adaptation approach for efficient multi- task learning","cited_arxiv_id":null,"evidence_quote":"supplies the shared-plus-domain-specific LoRA structure that the proposed constraints extend."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"defines the low-rank adapter form that the method constrains and projects."},{"cited_title":"Interpretable image recognition by constructing transparent embedding space","cited_arxiv_id":null,"evidence_quote":"provides the orthonormality and subspace-separation losses used to separate LoRAs of different domains."},{"cited_title":"Tensors for Data Processing: Theory, Methods, and Applications","cited_arxiv_id":null,"evidence_quote":"supplies the truncated SVD technique used to manufacture a left null space from the full-rank pretrained weights."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"provides the Vision Transformer architecture whose frozen linear layers are decomposed."},{"cited_title":"Imagenet-21k pretraining for the masses","cited_arxiv_id":null,"evidence_quote":"supplies the ImageNet21K pretraining of the backbone used in all experiments."},{"cited_title":"Poggio, and Thomas Serre","cited_arxiv_id":null,"evidence_quote":"one of the three action-recognition evaluation datasets, HMDB51."}],"review_version":1}