The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

Abigail Woodring; Adrian Chan; Chau-Wai Wong; Rana Muhammad Shahroz Khan; Sukwon Yun; Tianlong Chen

REVIEW 4 major objections 5 minor 17 references

A one-time LoRA patch stays effective across ten continual-pretraining updates of the base model, and the paper attributes this portability to near-orthogonality between pretraining and fine-tuning gradients in high-dimensional parameter sp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:13 UTC pith:EXOYUU7O

load-bearing objection The empirical portability result is real and useful; the theoretical explanation leans on an unmeasured quasi-orthogonality assumption, so the mechanism claim is a hypothesis, not a proof. the 4 major comments →

arxiv 2607.20301 v1 pith:EXOYUU7O submitted 2026-07-22 cs.LG cs.CL

The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

Abigail Woodring , Adrian Chan , Rana Muhammad Shahroz Khan , Sukwon Yun , Chau-Wai Wong , Tianlong Chen This is my paper

classification cs.LG cs.CL

keywords temporal portabilitycontinual pretrainingLoRA patchesPortLLMnear-orthogonalityhigh-dimensional geometryloss landscapeparameter-efficient fine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that temporal portability—the practice of taking a LoRA patch fine-tuned once on an initial base model and applying it, unchanged, to a base model that has since undergone repeated continual pretraining—works over the long term, and that the explanation is geometric rather than content-based. Empirically, the authors show that PortLLM patches trained at time step 0 remain competitive with stepwise fine-tuning across ten continual-pretraining steps on three base models (Mistral, Gemma, Qwen) and three repetitions, clearly beating no patching and keeping performance approximately flat across time. Theoretically, they present two analyses: one slices the fine-tuning loss along a 1-D segment between patches and shows that the cost, benefit, and variation of portability are single inner products; the other iterates a first-order Taylor expansion along continual pretraining steps to bound the variation by a term proportional to the quasi-orthogonality constant between pretraining and fine-tuning gradients. Both point to the same conclusion: in billion-dimensional parameter space, the pretraining update direction is nearly orthogonal to the fine-tuning gradient direction, so a patch trained on the old base model still lands in a flat region of the loss landscape on the new base model.

Core claim

The central claim is that the well-known PortLLM result—reusing a LoRA patch fine-tuned on the original base model with no training and no data—extends to ten continual-pretraining updates, and that the root cause is near-orthogonality in high-dimensional space. In the paper's terms, the cost of PortLLM relative to stepwise fine-tuning, the benefit relative to no patching, and the variation across time steps each reduce to a single inner product; these inner products are small because the relevant vectors (e.g., the pretraining gradient and the fine-tuning gradient at the patched model) are nearly orthogonal, and because the loss landscape is flat around fine-tuned models. The paper formaliz

What carries the argument

The central objects are ε-quasi-orthogonality, defined as two vectors whose cosine similarity magnitude is at most ε, and the displaced-gradient–displacement product F(γΔx)=⟨∇f(x0+γΔx),γΔx⟩, which captures how a displacement from a reference point aligns with the gradient at the displaced point. The 1-D-slice analysis shows that each performance metric (C_PortLLM, B_PortLLM, V_PortLLM) equals such an inner product, whose magnitude is small when alignment is small (near-orthogonality) and the landscape is flat (small spectral norm of the Hessian). The iterative analysis uses a first-order Taylor expansion of the fine-tuning loss along each continual-pretraining gradient step, combined with th

Load-bearing premise

Assumption 1—that every pretraining gradient ∇L_pt(θ) and the corresponding PortLLM-adapted fine-tuning gradient ∇L_ft(θ+Δθ_0) have cosine similarity bounded by a small ε_orth along the entire continual-pretraining trajectory—is the load-bearing premise; if it is violated, the Theorem 12 bound is vacuous and the near-orthogonality explanation reduces to an unverified analogy.

What would settle it

Run a continual-pretraining trajectory (e.g., the paper's 10 Fineweb steps on Mistral) and record, at many iterates, the cosine similarity between ∇L_pt(θ) and ∇L_ft(θ+Δθ_0). If the maximum observed cosine similarity is not small (say >0.1), or if a downstream task whose fine-tuning data is heavily aligned with the pretraining data (forcing high cosine similarity) eliminates PortLLM's portability, the central claim fails. Conversely, if PortLLM remains portable in a setting where the measured ε_orth is large, then near-orthogonality is not the operative cause.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Repeated fine-tuning is unnecessary when the base model is periodically updated: a patch trained once on the initial base model can be carried forward across at least ten continual-pretraining steps without retraining.
PortLLM is a practical alternative to stepwise fine-tuning: across Mistral, Gemma, and Qwen on WinoGrande, BoolQ, ARC-Easy, and ARC-Challenge, PortLLM matches stepwise fine-tuning and beats no patching by 4–11 percentage points at t=10.
PortLLM performance is approximately constant across time steps: a statistically significant linear trend appears in only 3 of 12 evaluation cases, with slopes on the order of 10^-3.
The two theoretical analyses apply beyond PortLLM to other LoRA variants and to comparing arbitrary LoRA patches on a given task, since the definitions of cost, benefit, and variation as inner products are generic.
When base-model performance degrades during continual pretraining, PortLLM mirrors that degradation, implying that careful, stable continual pretraining is needed to preserve portability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if near-orthogonality is the mechanism, portability should be fragile when the continual-pretraining data is close to the downstream task, since then the pretraining gradient will align with the fine-tuning gradient; this suggests a testable boundary condition the paper does not explore.
Editorial inference: the paper's bound suggests a practical pre-deployment diagnostic—measure the cosine similarity between the pretraining gradient and the fine-tuning gradient before relying on PortLLM—since the bound's usefulness depends entirely on ε_orth being small.
Editorial inference: the analysis assumes stable continual pretraining with bounded gradient norms; under aggressive updates, sharp minima, or gradient explosion, both the flatness and quasi-orthogonality premises break. The paper's own Appendix F hints that when base-model performance falls, PortLLM falls with it.
Editorial inference: because the proof only uses bounded update norms and quasi-orthogonality, the portability result likely generalizes to other low-rank PEFT updates (e.g., DoRA-style magnitude-direction decompositions) and to other forms of base-model evolution, such as pruning or quantization, not just continual pretraining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The empirical portability result is real and useful; the theoretical explanation leans on an unmeasured quasi-orthogonality assumption, so the mechanism claim is a hypothesis, not a proof.

read the letter

The solid part of this paper is the empirical study. Ten continual-pretraining steps across Mistral-7B, Gemma-12B, and Qwen-0.5B, three repetitions, several benchmarks, and statistical tests for trend lines: that is a real extension of Khan et al.'s at-most-4-step result, and it supports the practical claim that a LoRA patch fine-tuned once can be reused across base-model updates with little loss. The comparison to stepwise fine-tuning is the kind of head-to-head the field needs, and the finding that PortLLM is roughly flat across steps is convincing, with the caveat that 7B/12B pretraining is approximated with high-rank LoRA and Qwen full pretraining is only 0.5B. That caveat is worth stating, but it doesn't undermine the main empirical conclusion.

The 1-D slice analysis is a clean way of writing performance gaps as inner products, and the empirical plots showing that the inner product is much smaller than the norm product do tell you that near-orthogonality, not small norms, is behind the small cost CPortLLM. That is a nice observation. Lemma 1, which the reader flagged, is actually fine: the vector Taylor step is legitimate for a twice-differentiable loss, and the spectral-norm bound is standard. Theorems 4 and 10 are tautological, as the reader says—they define epsilon as the max of the derivative and then recover the trivial bound |f(1)-f(0)|≤max|f'|. That's harmless as a formal device, but the explanatory weight is in the empirical smallness of epsilon, not in the bound itself.

The bigger soft spot is Assumption 1 in Section 4.3. The whole iterative analysis for RQ3 assumes pretraining and fine-tuning gradients are quasi-orthogonal with an unknown epsilon_orth, and the paper never measures that quantity. What Figure 4 measures is orthogonality between the fine-tuning gradient along the slice and the patch difference—a different vector pair. So the headline claim that near-orthogonality explains temporal portability is supported for the cost metric, but for the variation RQ3 it rests on an unverified assumption. The paper even argues the assumption is plausible via random-vector intuition, which is fine as motivation, but it's not evidence about actual language-modeling gradients. That makes the root-cause statement too strong for the current evidence.

The paper also ships no code or checkpoints, which hurts reproducibility, and the self-citations are appropriate given the prior work. Overall, this is a useful empirical contribution with a theoretical framework that is honest but incomplete. It deserves a serious referee, not a desk rejection. I would want the revision to either measure epsilon_orth or soften the mechanism claim to a hypothesis. If the authors can close that gap, this becomes a significantly stronger paper.

Referee Report

4 major / 5 minor

Summary. The paper investigates long-term temporal portability of LoRA patches (PortLLM) across repeated continual pretraining updates. Empirically, the authors fine-tune a LoRA patch at time 0 on three base models (Mistral-7B, Gemma3-12B, Qwen2-0.5B), then apply it after 10 continual pretraining steps, comparing against no patching and stepwise fine-tuning with three repetitions. They find that PortLLM roughly matches stepwise fine-tuning, clearly outperforms no patching, and stays approximately constant across time steps. Theoretically, the paper offers two analyses: a 1-D slice of the loss landscape yielding inner-product characterizations of the cost/benefit/variation metrics, and an iterative optimization-step analysis with quasi-orthogonality assumptions leading to a bound on the temporal variation. The authors conclude that near-orthogonality of high-dimensional vectors is a key justification for temporal portability.

Significance. If the claims hold, the work has clear practical value: downstream developers could reuse a LoRA patch across many base-model updates without retraining, and the geometric mechanism would provide a principled explanation. The empirical study is a solid contribution: it extends the temporal horizon of PortLLM from 1-4 steps to 10 steps, covers three architectures and multiple benchmarks, includes repetitions, and applies statistical tests. The theoretical portion, however, is not yet at the level claimed: the main theorem for RQ3 depends on an unmeasured quasi-orthogonality assumption, two of the theorems are tautological, and a key lemma relies on a mathematically questionable vector-valued Taylor step. The empirical findings are independent of these issues, but the 'blessing of dimensionality' explanation is currently supported by assumed rather than measured properties of the gradients.

major comments (4)

[Section 4.2, Lemma 1 (Eq. 5) and Appendix C] The vector-valued Taylor expansion ∇f(x0+γΔx)=∇f(x0)+γ∇²f(x0+β'γΔx)Δx assumes a single β' for all N components of the gradient. For a scalar-valued f, each component of ∇f has its own Lagrange remainder point, so this is generally not valid. The subsequent bound and the 'alignment/sharpness' interpretation in Remarks 3, 7, and 9 rely on this step. Please derive the bound using an integral remainder and a supremum of ∥∇²f∥ over the segment, or explicitly label this as a heuristic requiring additional conditions.
[Section 4.2, Theorems 4 and 10] These theorems are tautological. Since ε is defined as the maximum absolute derivative of the 1-D loss slice over α∈[0,1], and Lemma 2/8 shows the cost/variation equals that derivative at some α', the inequalities |C|≤ε and |V|≤ε follow immediately. No independent estimate of ε is provided, so the theorems do not explain why the quantities are small; they merely restate the quantities. To support the 'key justification' claim, the authors need to bound ε by measurable quantities (e.g., quasi-orthogonality constants and Hessian norms) or measure ε directly on the models.
[Section 4.3, Assumption 1 and Theorem 12 (Eq. 11)] The assumption that ∇L_pt(θ) and ∇L_ft(θ+Δθ0) are ε_orth-quasi-orthogonal for every θ is load-bearing for the RQ3 bound and for the abstract's root-cause claim. It is never measured. Figure 4 and Appendix G measure a different vector pair—the fine-tuning gradient along the φ-slice and the patch displacement Δ_t^(ft)—which does not address Assumption 1. The random-vector heuristic in Appendix D concerns independent Gaussian vectors, not structured language-modeling gradients that may share systematic next-token prediction components. The authors should measure the cosine in Assumption 1 on the actual models (or measure the inner product in Lemma 8). Otherwise the bound (11) is vacuous if ε_orth is not small.
[Section 4.2, Lemma 8 and note 8] The paper states that compute constraints prevent calculating the derivative g'_v(α) for V_PortLLM, i.e., the inner product ⟨∇_θ L_ft(θ(α)), Δ_t^(pt)⟩. This is precisely the quantity that would provide direct evidence for near-orthogonality as the mechanism of temporal portability (RQ3). Without it, the explanation for the flat RQ3 curves relies on Assumption 1, which is unverified. Given that the models are already available and the quantity involves one gradient computation per α, this measurement should be feasible and is necessary to substantiate the central claim.

minor comments (5)

[Section 4.2, Remark 7] The phrase 'for α≤0≤1' is a typographical error; should be 'α∈[0,1]'.
[Section 3.2, Table 1] The caption says 'Repetition Slope (×10^-4)' but the test is described as intercept vs. linear; please clarify how the p-values are computed and whether any multiple-testing correction is applied across the 12 cases.
[Figures 2-10] Error bars are 95% confidence intervals from 3 repetitions; with 2 degrees of freedom these are very wide. Consider overlaying individual repetition markers to aid interpretation.
[Section 4.2, Eq. (8a)] The superscript on Δ_t^(ft) is used in Eq. (8a) but not defined until later; please define it at first use.
[Section 4.3, Theorem 12 statement] Typo: 'time stemt' should be 'time step'.

Circularity Check

2 steps flagged

Theorems 4 and 10 bound each metric by the maximum of the very derivative that, by Taylor's theorem, exactly equals the metric; these bounds are tautological. The empirical portability measurements are independent and non-circular.

specific steps

self definitional [Section 4.2, Lemma 2 and Theorem 4]
"Lemma 2 There exists an α′ ∈ [0,1] such that C_PortLLM(t)=⟨∇_φ ℓ_{θ_t}(φ(α′)), Δ_t^{(ft)}⟩ ... Theorem 4 Let ϵ= max_{α∈[0,1]} |⟨∇_φ ℓ_{θ_t}(φ(α)), Δ_t^{(ft)}⟩| be the maximum derivative magnitude ... Then the cost of using PortLLM instead of stepwise fine-tuning is bounded as |C_PortLLM(t)| ≤ ϵ."

By the Taylor step preceding Lemma 2, C_PortLLM(t)=g_c(1)−g_c(0)=g'_c(α′)=⟨∇_φ ℓ_{θ_t}(φ(α′)), Δ_t^{(ft)}⟩ at one particular α′. Theorem 4 then defines ε as the maximum of the absolute value of exactly that same inner product over all α∈[0,1]. Therefore |C|≤ε is true by construction for any differentiable function; it cannot fail and it uses no property of the loss landscape. The proof says 'Taking the maximum over α∈[0,1] upper bounds the cost,' confirming that the bound reduces to the definition of a maximum. The empirical measurement of the derivative in Figure 4, not the theorem, carries the explanatory weight.
self definitional [Section 4.2, Lemma 8 and Theorem 10]
"Lemma 8 There exists an α′ ∈ [0,1] such that the change in PortLLM performance at time step t is given by V_PortLLM(t)=⟨∇_θ L_ft(θ(α′)), Δ_t^{(pt)}⟩ ... Theorem 10 Let ϵ= max_{α∈[0,1]} |⟨∇_θ L_ft(θ(α)), Δ_t^{(pt)}⟩| be the maximum derivative magnitude ... Then the change in performance is bounded as |V_PortLLM(t)| ≤ ϵ."

The derivation of Lemma 8 gives V_PortLLM(t)=g_v(1)−g_v(0)=g'_v(α′)=⟨∇_θ L_ft(θ(α′)), Δ_t^{(pt)}⟩ at a fixed α′. Theorem 10 defines ε as the maximum absolute value of the same inner product over the whole slice, so |V|≤ε is a tautology: it is true by the definition of a maximum for every loss and every pair of base models. It does not independently prove that V is small or that near-orthogonality is the mechanism; the paper's RQ3 explanation rests on the empirical flatness of the 1-D slice, not on this bound.

full rationale

The core empirical contribution—long-term PortLLM portability across 10 continual-pretraining steps on Mistral, Gemma, and Qwen with three repetitions—is based on direct benchmark measurements (Figures 2, 7–10 and Table 1), so that claim is not circular. Self-citations to Khan et al. (2025, 2026) introduce the PortLLM method and prior short-horizon results, but the present experiments independently re-run and extend those measurements; the self-citations are not load-bearing in the paper's new empirical or theoretical argument. The main circularity is in the theoretical framing: Theorem 4 and Theorem 10 are definitional bounds because each defines ε as the maximum of the derivative that, by the accompanying Lemma and Taylor's theorem, exactly equals the quantity being bounded. They reduce to |C|≤max|C-slice derivative| and |V|≤max|V-slice derivative|, which cannot be false and therefore do not by themselves establish near-orthogonality as an explanation. The empirical 1-D slice measurements in Figure 4 and Appendix G provide non-circular evidence that the relevant inner products are small relative to the norm products, and this is the real support for the root-cause claim. Assumption 1 in Section 4.3 (ε_orth-quasi-orthogonality of pretraining and fine-tuning gradients) is unmeasured and justified only by high-dimensional random-vector intuition; that is a correctness/evidential gap, not a circular reduction, because Theorem 12 honestly states a conditional bound with ε_orth as an explicit input. Overall score 4: the central empirical claim is independent and directly measured, while the theoretical explanation is partially definitional.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The theoretical explanation rests on several unmeasured or assumed quantities: ε_orth, ℓ_pt, ℓ_ft, ε_η, and the rank-64/128 LoRA proxy for true continual pretraining. The paper supplies no direct measurement of pretraining-vs-fine-tuning gradient cosine similarity, and Lemma 1's justification relies on a questionable Taylor step. The empirical portability result, however, does not depend on these theoretical constructs.

free parameters (5)

ε_orth (quasi-orthogonality constant)
Assumption 1, Section 4.3. The paper assumes pretraining and fine-tuning gradients are ε_orth-quasi-orthogonal for all θ but does not measure this quantity on actual model gradients.
ℓ_pt (pretraining gradient norm bound)
Assumption 3. Assumed stable gradient norm during continual pretraining; no numeric estimate provided.
ℓ_ft (fine-tuning gradient norm bound)
Assumption 4, Eq. (9). Assumed bounded fine-tuning gradient in a neighborhood of pretraining iterates; no numeric estimate provided.
ε_η (Taylor remainder bound)
Assumption 5, Eq. (10). Bounds the first-order Taylor error; characterized in Remark 16 as O(η²e²ℓ_pt²Σσ_j) but never estimated numerically.
LoRA rank for approximate pretraining = r=64 (Mistral-7B), r=128 (Gemma3-12B)
Section 3.1. Continual pretraining for the two largest models is approximated by training high-rank LoRA patches and merging them. This is a modeling choice that defines the regime actually tested.

axioms (6)

domain assumption Pretraining gradient ∇L_pt(θ) and PortLLM-adapted fine-tuning gradient ∇L_ft(θ+Δθ0) are ε_orth-quasi-orthogonal for all θ.
Assumption 1, Section 4.3. This is the core premise of Theorem 12; justified only by analogy to random high-dimensional vectors, with no measurement on actual LLM gradients.
domain assumption Pretraining gradient norm is bounded: ∥∇L_pt(θ_t^(i))∥₂ ≤ ℓ_pt for all iterations.
Assumption 3. Stability, i.e., no gradient explosion during continual pretraining; plausible with gradient clipping but not derived.
domain assumption Fine-tuning gradient norm is bounded in a neighborhood of pretraining iterates, and ∥Δθ0∥₂ ≤ d.
Assumption 4, Eq. (9). Assumes pretrained models are well-conditioned starting points for fine-tuning.
domain assumption First-order Taylor expansion of fine-tuning loss along pretraining updates has small error R2 with |R2| ≤ ε_η.
Assumption 5, Eq. (10). Needed for the iterative bound; justified by small learning-rate steps.
standard math Single-β' Taylor remainder for the gradient vector, as used in Lemma 1.
Appendix C. The paper writes ∇f(x0+γΔx)=∇f(x0)+γ∇²f(x0+β'γΔx)Δx. This is not generally valid for vector-valued gradients because componentwise intermediate points differ. The inequality can be salvaged by choosing β' at the max spectral norm along the segment, but the stated Taylor argument is false.
standard math Random high-dimensional vectors are approximately orthogonal.
Appendix D and Table 2. Used to justify Assumption 1. True for independent random vectors, but actual model gradients are not shown to be random or independent.

pith-pipeline@v1.3.0-alltime-deepseek · 28934 in / 20194 out tokens · 170027 ms · 2026-08-01T10:13:08.785360+00:00 · methodology

0 comments

read the original abstract

Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computational costs. PortLLM is a training-free and data-free scheme used to adapt LLMs after continual pretraining. Although the initial PortLLM results show that LoRA patches exhibit short-term temporal portability, the long-term performance of PortLLM across several updates of continual pretraining remains underexplored. Furthermore, the intriguing effectiveness of PortLLM is not well understood from a theoretical standpoint. We address these two open questions by (1) performing an extensive empirical study of the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen; and (2) offering two theoretical analyses to explain our observation that the simple PortLLM method achieves competitive performance. We find empirically that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated. We find theoretically that near-orthogonality of high-dimensional vectors is a key justification for temporal portability. Our analyses also demonstrate a geometric perspective of the loss landscape in facilitating the theoretical comparison of different adaptation options.

Figures

Figures reproduced from arXiv: 2607.20301 by Abigail Woodring, Adrian Chan, Chau-Wai Wong, Rana Muhammad Shahroz Khan, Sukwon Yun, Tianlong Chen.

**Figure 1.** Figure 1: Comparison of three possible patching strategies represented by the three rows of patches: (bottom) stepwise fine-tuning (SWFT) finetunes a new patch ∆θi at each time step, (middle) PortLLM retains ∆θ0 trained at time step 0, and (top) no patching uses the unadapted base model. RQ1: Cost of PortLLM compared to stepwise fine-tuning. To aid in relatively highresource scenarios, we ask the research ques… view at source ↗

**Figure 2.** Figure 2: Temporal portability results for continually pretraining Mistral-7B-v0.1 on Fineweb data and downstream fine-tuning on (a) WinoGrande, (b) BoolQ, (c) ARC-Easy, and (d) ARC-Challenge. Average performance across 3 repetitions for PortLLM patching, stepwise fine-tuning, and no patching performance are shown, with the error bars indicating the 95% confidence interval. It can be observed that (RQ1) the cost of… view at source ↗

**Figure 3.** Figure 3: Geometric interpretation for analyzing the cost of employing PortLLM instead of stepwise fine-tuning, CPortLLM(t). The PortLLM patch ϕ0 and the stepwise finetuned patch ϕt are shown as blue arrows in unexpanded LoRA parameter space on the left. The downstream testing loss on a 1-D slice between the two patches is shown on the right. The loss, shown as a blue solid line, is approximately constant so that t… view at source ↗

**Figure 4.** Figure 4: Empirical results for cost (top row) and benefit (bottom row) metrics on 1-D slice for t = 2. We shown (a) and (e) WinoGrande (b) and (f) BoolQ (c) and (g) ARC-Easy and (d) and (h) ARC-Challenge. The derivative g ′ c (α) is small in most cases, indicating that the cost of using PortLLM instead of stepwise fine-tuning is small. The derivative g ′ c (α) = ⟨∇ϕℓθt (ϕ (α) ), ∆ (ft) t ⟩ is much smaller than the … view at source ↗

**Figure 5.** Figure 5: Illustration of analysis for the benefit of employing PortLLM instead of no patching, BPortLLM(t). The PortLLM patch ϕ0 is shown as blue arrows in unexpanded LoRA parameter space on the left. The downstream testing loss on a 1-D slice between the PortLLM patch and no patching is shown on the right. The loss is shown as a blue solid line while the derivative of the loss is shown as a red dashed line. The d… view at source ↗

**Figure 6.** Figure 6: Geometric interpretation for the variation of PortLLM performance, VPortLLM(t). The initial PortLLM parameters θ0 + ∆θ0 and the PortLLM parameters at time t > 0, θt + ∆θ0 are shown as blue arrows in expanded LoRA parameter space on the left. The downstream testing loss on a 1-D slice between the two parameters is shown on the right. The loss, shown as a blue solid line, is approximately constant so that th… view at source ↗

**Figure 7.** Figure 7: Temporal portability results for Mistral-7B-v0.1 pretrained on Fineweb (Penedo et al., 2024) and downstream fine-tuned on additional benchmarks. Testing results are shown for (a) MNLI (b) SST2 and (c) GSM8k. For each benchmark, reported results are a sample average across 3 repetitions. The error bars shown 95% confidence interval. lowing order: auto math text, wikihow, openstax, stanford, stories, web sa… view at source ↗

**Figure 8.** Figure 8: Temporal portability results for continually pretraining Gemma3-12B on Fineweb data and downstream fine-tuning on (a) WinoGrande, (b) BoolQ, (c) ARC-Easy, and (d) ARC-Challenge. Average performance across 3 repetitions for PortLLM patching, stepwise fine-tuning, and no patching performance are shown. The error bars indicate the 95% confidence interval. No patching PortLLM patching Stepwise fine-tuning No p… view at source ↗

**Figure 9.** Figure 9: Temporal portability results for continually pretraining Qwen2-0.5B on Fineweb data and downstream fine-tuning on (a) WinoGrande, (b) BoolQ, (c) ARC-Easy, and (d) ARC-Challenge. Average performance across 3 repetitions for PortLLM patching, stepwise fine-tuning, and no patching performance are shown. The error bars indicate the 95% confidence interval. F.2 Gemma3-12B Pretrained on Fineweb Temporal portabil… view at source ↗

**Figure 10.** Figure 10: Temporal portability results for continually pretraining Mistral-7B-v0.1 on Cosmopedia data and downstream fine-tuning on (a) WinoGrande, (b) BoolQ, (c) ARC-Easy, and (d) ARC-Challenge. Average performance across 3 repetitions is plotted. The error bars indicate the 95% confidence interval. approximately constant on WinoGrande, and degrades slightly on BoolQ and ARC-Easy. This indicates that, if continua… view at source ↗

**Figure 11.** Figure 11: WinoGrande empirical results for RQ1. The plots include the loss for downstream testing data gc(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt (ϕ (α) )∥∥∆ (ft) t ∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 0.00 0.25 0.50 0.75 1.00 0 … view at source ↗

**Figure 12.** Figure 12: WinoGrande empirical results for RQ2. The plots include the loss for downstream testing data gb(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt ((1 − α)ϕ0)∥∥ϕ0∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 27 [PITH_FULL_IMAGE:figures/fu… view at source ↗

**Figure 13.** Figure 13: BoolQ empirical results for RQ1. The plots include the loss for downstream testing data gc(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt (ϕ (α) )∥∥∆ (ft) t ∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 0.00 0.25 0.50 0.75 1.00 0 10 0 2… view at source ↗

**Figure 14.** Figure 14: BoolQ empirical results for RQ2. The plots include the loss for downstream testing data gb(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt ((1−α)ϕ0)∥∥ϕ0∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 28 [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗

**Figure 15.** Figure 15: ARC-Easy empirical results for RQ1. The plots include the loss for downstream testing data gc(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt (ϕ (α) )∥∥∆ (ft) t ∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 0.00 0.25 0.50 0.75 1.00 0 2 4… view at source ↗

**Figure 16.** Figure 16: ARC-Easy empirical results for RQ2. The plots include the loss for downstream testing data gb(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt ((1−α)ϕ0)∥∥ϕ0∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 29 [PITH_FULL_IMAGE:figures/full_fi… view at source ↗

**Figure 17.** Figure 17: ARC-Challenge empirical results for RQ1. The plots include the loss for downstream testing data gc(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt (ϕ (α) )∥∥∆ (ft) t ∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 0.00 0.25 0.50 0.75 1.00… view at source ↗

**Figure 18.** Figure 18: ARC-Challenge empirical results for RQ2. The plots include the loss for downstream testing data gb(α) on a 1-D slice between the PortLLM patch and the stepwise fine-tuning patch (dashed green line), the derivative of the 1-D slice (solid red line), and the norm product ∥∇ϕℓθt ((1 − α)ϕ0)∥∥ϕ0∥ (dotted blue line). We show t ∈ {2, 4, 6, 8, 10} and three repetitions on each plot. 30 [PITH_FULL_IMAGE:figures… view at source ↗

**Figure 19.** Figure 19: 1-D slice of the loss landscape in a random direction. We select the direction as a Gaussian random vector and rescale it to have a norm equal to the distance between the PortLLM and stepwise fine-tuning tasks. Results are for t = 2 and (a) WinoGrande, (b) BoolQ, (c) ARC-Easy, and (d) ARC-Challenge. Three repetitions are shown for each plot. We also show 10 random direction for BoolQ for a fixed training … view at source ↗

discussion (0)

Reference graph

Works this paper leans on

17 extracted references · 3 canonical work pages

[6]

doi: 10.1145/3711875.3729128. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized llms. InNeural Information Processing Systems, volume 36,

arXiv
[7]

32 The Blessing of Dimensionality Y

doi: 10.18653/v1/2021.acl-long.378. 32 The Blessing of Dimensionality Y. Guo, J. Fu, H. Zhang, D. Zhao, and Y. Shen. Efficient continual pre-training by miti- gating the stability gap.arXiv preprint arXiv:2406.14833,

Pith/arXiv arXiv 2021
[8]

Hayou, N

S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354,

Pith/arXiv arXiv
[9]

doi: 10.18653/v1/2021.acl-long.353. S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. DoRA: Weight-decomposed low-rank adaptation. InInternational Conference on Machine Learning,

doi:10.18653/v1/2021.acl-long.353 2021
[11]

doi: 10.1145/3474381. S. Shukla, A. Sriram, M. K. Narayanaswamy, and H. Jain. qa-FLoRA: Data-free query- adaptive fusion of LoRAs for LLMs. InAAAI Conference on Artificial Intelligence, volume 40,

doi:10.1145/3474381
[13]

doi: 10.18653/v1/2025.repl4nlp-1.11. Y.-L. Sung, V. Nair, and C. Raffel. Training neural networks with fixed sparse masks. In Neural Information Processing Systems,

doi:10.18653/v1/2025.repl4nlp-1.11 2025
[14]

K. Svozil. The geometric part of decoherence: Quasi-orthogonality in high-dimensional hilbert spaces.arXiv preprint arXiv:2605.03807,

Pith/arXiv arXiv
[15]

doi: 10.52202/079017-3801. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Ba...

doi:10.52202/079017-3801
[17]

L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

Pith/arXiv arXiv
[1989]

doi: https://doi.org/10.1016/0047-259X(89)90039-0. G. Penedo, H. Kydl ´ ıˇ cek, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V. Werra, and T. Wolf. The Fineweb datasets: Decanting the web for the finest text data at scale. In Neural Information Processing Systems Datasets and Benchmarks Track,

work page doi:10.1016/0047-259x(89)90039-0
[2000]

doi: 10.1109/ 18.841175. C ¸ . Yıldız, N. K. Ravichandran, N. Sharma, M. Bethge, and B. Ermis. Investigating con- tinual pretraining in large language models: Insights and implications.arXiv preprint arXiv:2402.17400,

Pith/arXiv arXiv
[2018]

Cobbe, V

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Pith/arXiv arXiv
[2019]

Clark, I

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457v1,

Pith/arXiv arXiv
[2021]

Das and S

A. Das and S. Dutta. Weight-decay turns transformer loss landscapes Villani: Functional-analytic foundations for optimization and generalization.arXiv preprint arXiv:2605.06599,

Pith/arXiv arXiv
[2023]

PMID: 37439496

doi: 10.1021/acs.chemrestox.3c00042. PMID: 37439496. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InConference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,

work page doi:10.1021/acs.chemrestox.3c00042
[2024]

H. Chen, Y. Dong, Z. Wei, Y. Huang, Y. Zhang, H. Su, and J. Zhu. Unveiling the basin-like loss landscape in large language models.arXiv preprint arXiv:2505.17646,

arXiv
[2025]

doi: 10.18653/v1/2025.emnlp-industry.164. Y. Deng, Z. Qiao, Y. Zhang, Z. Ma, Y. Liu, and J. Ren. CrossLM: A data-free collaborative fine-tuning framework for large and small language models. InAssociation for Computing Machinery, MobiSys ’25,

work page doi:10.18653/v1/2025.emnlp-industry.164 2025

[1] [6]

doi: 10.1145/3711875.3729128. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized llms. InNeural Information Processing Systems, volume 36,

arXiv

[2] [7]

32 The Blessing of Dimensionality Y

doi: 10.18653/v1/2021.acl-long.378. 32 The Blessing of Dimensionality Y. Guo, J. Fu, H. Zhang, D. Zhao, and Y. Shen. Efficient continual pre-training by miti- gating the stability gap.arXiv preprint arXiv:2406.14833,

Pith/arXiv arXiv 2021

[3] [8]

Hayou, N

S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354,

Pith/arXiv arXiv

[4] [9]

doi: 10.18653/v1/2021.acl-long.353. S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. DoRA: Weight-decomposed low-rank adaptation. InInternational Conference on Machine Learning,

doi:10.18653/v1/2021.acl-long.353 2021

[5] [11]

doi: 10.1145/3474381. S. Shukla, A. Sriram, M. K. Narayanaswamy, and H. Jain. qa-FLoRA: Data-free query- adaptive fusion of LoRAs for LLMs. InAAAI Conference on Artificial Intelligence, volume 40,

doi:10.1145/3474381

[6] [13]

doi: 10.18653/v1/2025.repl4nlp-1.11. Y.-L. Sung, V. Nair, and C. Raffel. Training neural networks with fixed sparse masks. In Neural Information Processing Systems,

doi:10.18653/v1/2025.repl4nlp-1.11 2025

[7] [14]

K. Svozil. The geometric part of decoherence: Quasi-orthogonality in high-dimensional hilbert spaces.arXiv preprint arXiv:2605.03807,

Pith/arXiv arXiv

[8] [15]

doi: 10.52202/079017-3801. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Ba...

doi:10.52202/079017-3801

[9] [17]

L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. MetaMath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

Pith/arXiv arXiv

[10] [1989]

doi: https://doi.org/10.1016/0047-259X(89)90039-0. G. Penedo, H. Kydl ´ ıˇ cek, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V. Werra, and T. Wolf. The Fineweb datasets: Decanting the web for the finest text data at scale. In Neural Information Processing Systems Datasets and Benchmarks Track,

work page doi:10.1016/0047-259x(89)90039-0

[11] [2000]

doi: 10.1109/ 18.841175. C ¸ . Yıldız, N. K. Ravichandran, N. Sharma, M. Bethge, and B. Ermis. Investigating con- tinual pretraining in large language models: Insights and implications.arXiv preprint arXiv:2402.17400,

Pith/arXiv arXiv

[12] [2018]

Cobbe, V

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Pith/arXiv arXiv

[13] [2019]

Clark, I

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457v1,

Pith/arXiv arXiv

[14] [2021]

Das and S

A. Das and S. Dutta. Weight-decay turns transformer loss landscapes Villani: Functional-analytic foundations for optimization and generalization.arXiv preprint arXiv:2605.06599,

Pith/arXiv arXiv

[15] [2023]

PMID: 37439496

doi: 10.1021/acs.chemrestox.3c00042. PMID: 37439496. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InConference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,

work page doi:10.1021/acs.chemrestox.3c00042

[16] [2024]

H. Chen, Y. Dong, Z. Wei, Y. Huang, Y. Zhang, H. Su, and J. Zhu. Unveiling the basin-like loss landscape in large language models.arXiv preprint arXiv:2505.17646,

arXiv

[17] [2025]

doi: 10.18653/v1/2025.emnlp-industry.164. Y. Deng, Z. Qiao, Y. Zhang, Z. Ma, Y. Liu, and J. Ren. CrossLM: A data-free collaborative fine-tuning framework for large and small language models. InAssociation for Computing Machinery, MobiSys ’25,

work page doi:10.18653/v1/2025.emnlp-industry.164 2025