{"id":"d484ef50-1390-4121-8c75-97efd6d515d3","arxiv_id":"2608.05253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AuroOFT attaches a zero-initialized gated low-rank nonlinear residual to QOFT layers and reports improved mathematical reasoning accuracy over matched QOFT on 1.5B/3B Qwen2.5 models, with larger gains over QLoRA at lower parameter cost.","lead":"AuroOFT adds a zero-start, gated nonlinear branch next to quantized orthogonal fine-tuning (QOFT), letting each adapted layer learn input-dependent corrections while the QOFT path stays unchanged. On Qwen2.5 1.5B/3B math benchmarks the paper reports Macro-6 gains of 1.30 to 2.70 points over matched QOFT and larger gains over QLoRA with fewer trainable parameters, though 7B results are mixed and error bars are absent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing linear-residual control confounds expressivity claim; abstract's 1.30–2.70 range omits 3B-it delta of 0.38","rationale":"The reader's single-seed concern is valid, but the more fundamental identification problem is the lack of a linear-capacity control. Even with multiple seeds, comparing AuroOFT to bare QOFT cannot attribute the gain to nonlinearity; it may simply reflect extra parameters. This is the load-bearing issue because the paper's stated purpose is to test whether nonlinear corrections expand QOFT beyond rotations, and the introduction explicitly says the matched QOFT comparison is meant to isolate the residual branch. The reporting inconsistency (omitting the 0.38 delta for 3B-it from the abstract's claimed range) further overstates the headline. The proposed control experiment is cheap and decisive: it directly measures whether nonlinearity, rather than capacity, drives the improvement. A verdict of conditional is appropriate because the architectural idea and zero-start equivalence are sound, but the central empirical attribution requires this additional condition.","tokens_in":10970,"tokens_out":5920,"duration_ms":52418,"concrete_test":"Add a control: QOFT + linear LoRA-style residual with the same zero-initialized up projection, branch LR/clipping, adapter placement, and a parameter-matched rank (e.g., rank 4 to match AuroOFT rank 2) under the paper's matched protocol, with at least 3 seeds and 95% confidence intervals on Macro-6. If the linear control matches AuroOFT's gains over QOFT, the nonlinearity claim is not supported; if AuroOFT significantly exceeds it, the claim is supported. Also report the 3B-it delta explicitly and correct the abstract's improvement range to include 0.38.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the nonlinear residual branch supplies useful expressivity beyond the QOFT carrier. Table 1 compares AuroOFT (QOFT + nonlinear branch, 16.74M params) against bare QOFT (13.00M) and QLoRA (24.73M), but never against QOFT plus a linear low-rank residual of matched parameter count. The +1.30 to +2.70 Macro-6 deltas in Figure 4 could therefore be caused by the 3.74M additional parameters or by the residual pathway's added capacity in general, not by nonlinearity. Table 2's ablation varies only among nonlinear bases (Lite, SplineNorm, Enhanced), so it cannot separate nonlinear expressivity from capacity. Proposition 2 and Remark 1 correctly state containment but do not address this confound. A second issue: the abstract's '1.30–2.70 points' range omits the 3B-instruct setting, where the Macro-6 delta is 0.38 per Table 1. With one of four Qwen2.5 settings near zero and no linear control, the structural-expressivity claim is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AuroOFT, a parameter-efficient fine-tuning method that augments Quantized Orthogonal Fine-Tuning (QOFT) with a zero-start, gated, low-rank nonlinear residual branch. The main branch remains the QOFT rotation before frozen NF4 weights; the residual branch applies a down projection, an adaptive nonlinear layer, a zero-initialized up projection, and a gate, and adds the result to the QOFT output. The authors prove zero-start equivalence with QOFT, show that AuroOFT contains the QOFT function family, and present experiments on Qwen2.5 models of 1.5B, 3B, and 7B scale, reporting Macro-6 averages across six mathematical reasoning benchmarks. They claim that under matched data, optimization, decoding, and parser protocols, AuroOFT improves over matched QOFT by 1.30 to 2.70 points in the 1.5B/3B settings, exceeds QLoRA by 6.52 to 10.62 points, and saves 32.3 to 44.7 percent of trainable parameters relative to QLoRA.","tokens_in":11225,"tokens_out":2512,"duration_ms":24905,"significance":"If the claimed improvements over matched QOFT are robust, AuroOFT would be a useful contribution: it offers a clean architectural separation between a stable orthogonal backbone and an expressive nonlinear residual, and the paper is unusually careful in matching protocols across methods, in treating SAT Math only as a diagnostic, and in stating the boundary of what the structural results do and do not prove. The explicit zero-start equivalence, the containment proposition, the leakage-audited proxy selection, and the availability of code are all strengths. However, the central expressivity claim is currently supported only by single-seed point estimates and lacks a linear-residual control, so the significance of the method over QOFT is not yet established at the level the paper claims.","major_comments":[{"comment":"The structural-expressivity claim is confounded by parameter count. In Table 1, AuroOFT adds 2.31M parameters over QOFT at 1.5B scales, 3.74M at 3B scales, and 5.05M at 7B scales. Table 2 ablates only the nonlinear-branch configuration (Lite, SplineNorm, Enhanced) and never compares against QOFT plus a linear low-rank residual of matched parameter count, such as a LoRA-style BAx branch with the same rank, gate, and scaling. Without this control, the positive Macro-6 deltas in Figure 4 could be caused by the added residual capacity in general, not by nonlinearity. Please add a matched linear-residual baseline with approximately the same trainable parameter count as AuroOFT and report its Macro-6 performance in Table 1 or Table 2.","section":"Table 1, Table 2, Figure 4"},{"comment":"All reported Macro-6 deltas are single-seed point estimates with no error bars or repeated runs. The Limitations section states that broader validation should test more seeds, but this does not resolve the central claim. The 3B-instruct delta is 0.38 points, and the 7B-instruct and 7B deltas are -5.63 and -0.33 points, so run-to-run variance is a plausible explanation for the 1.30 to 2.70 point improvements in the 1.5B/3B settings. Please report results across multiple seeds (at least three) with means and standard deviations, or otherwise justify statistically why single-seed point estimates are sufficient for the main claim.","section":"Table 1, Figure 4, Limitations"}],"minor_comments":[{"comment":"The abstract's stated '1.30–2.70 points' range omits the 3B-instruct setting, whose Macro-6 delta is 0.38 per Table 1; please use the full range (0.38–2.70) or explicitly qualify that the range excludes one protocol-sensitive setting.","section":"Abstract vs. Table 1"},{"comment":"The table caption says only AuroOFT-specific residual-branch components are varied, but the Enhanced-r4 row changes the rank and increases parameters from 16.74M to 20.49M; please clarify that this row is a capacity ablation and state explicitly how rank variation is disentangled from nonlinearity.","section":"Table 2"},{"comment":"The gate notation G_l in Eq. (5) is overloaded among scalar, bounded scalar, and token-dependent forms; Eqs. (14)-(16) define these variants, but a sentence in the experimental setup identifying which gate type is used in each main-table configuration would improve reproducibility.","section":"Method, Eq. (14)-(16)"},{"comment":"Line 8 writes the residual as g_l * (alpha/r) * B_l * u_l, but Eq. (5) places the dropout after the nonlinearity and before the up projection; please align the algorithm's notation with the equation for the exact computation order.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well structured and honest about its boundaries, but the missing linear-residual control and the absence of error bars are load-bearing for the central claim that nonlinearity, rather than added capacity, explains the gains. These are fixable by additional experiments and do not require rethinking the method, so I see this as a major revision rather than a rejection. The abstract's selective reporting of the delta range should also be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AuroOFT is a sensible, well-scoped method paper. The core idea is new as a combination: keep QOFT as the stable quantized orthogonal branch, attach a zero-initialized gated low-rank nonlinear residual in parallel, and let the residual start exactly at zero so the initialized model is function-equivalent to QOFT. The containment proposition is correct, and the paper is honest that containment does not by itself prove accuracy gains. The matched evaluation protocol—same data, optimizer, decoding, parser—is a real strength, and the authors are appropriately cautious about SAT Math and the small exam-style set. The code release and leakage-audited proxy selection also count in their favor.\n\nThe soft spots are real but addressable. The main one is the missing control: Table 1 compares AuroOFT (QOFT + nonlinear branch) against bare QOFT with fewer parameters, but never against QOFT plus a linear low-rank residual of matched parameter count. So the +1.30 to +2.70 Macro-6 gains could come from added capacity or the residual pathway in general, not from nonlinearity. The ablation in Table 2 only varies among nonlinear bases, so it cannot separate nonlinear expressivity from capacity. That is a genuine confound for the paper's central structural claim.\n\nSecond, the headline numbers rest on single-seed runs with no error bars. The deltas over QOFT are small, and the 7B settings are actually worse (Macro-6 -5.63 on 7B-it). The abstract's \"1.30–2.70\" range quietly omits the 3B-it delta of 0.38. These are not fatal—the paper acknowledges some of this in the Limitations—but they mean the empirical evidence is thinner than the prose suggests.\n\nWho is this for? Researchers working on orthogonal PEFT or quantized adapters will find the architecture and the matched-evaluation template useful. The paper deserves serious peer review: the method is coherent, the write-up is honest, and the missing pieces are standard referee requests rather than signs of a broken core. My recommendation: send it out, but ask for a matched linear-residual baseline and multiple seeds before accepting. The central idea is plausible; it just is not yet proven.","headline":"AuroOFT is a clean, honestly-scoped extension of QOFT with a genuinely new zero-start nonlinear residual branch, but the empirical case needs a matched linear-capacity control and repeated seeds before the expressivity claim holds.","tokens_in":11762,"tokens_out":1605,"would_cite":true,"duration_ms":17401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AuroOFT adds a zero-start gated nonlinear residual beside QOFT's rotation branch and reports up to 2.70-point Macro-6 gains on 1.5B/3B Qwen2.5 math reasoning.","keywords":["parameter-efficient fine-tuning","quantized orthogonal fine-tuning","low-rank adaptation","nonlinear residual","mathematical reasoning","large language models","zero-start initialization","Qwen2.5"],"falsifier":"Run the matched 1.5B and 3B Qwen2.5 protocols with AuroOFT and QOFT across at least five seeds under identical data order, optimizer, decoding, and parser settings, and compute the Macro-6 delta with confidence intervals. If the average delta is near zero or its interval overlaps zero, the central claim that the nonlinear residual expands QOFT's useful expressivity is not supported.","tokens_in":10728,"feed_emoji":"🧮","tokens_out":8623,"duration_ms":72966,"temperature":0.7,"pith_summary":"This paper argues that quantized orthogonal fine-tuning (QOFT) is stable but expressively limited because its task-specific updates are restricted to input-side orthogonal linear transformations. To fix this, it introduces AuroOFT, which keeps the QOFT branch intact and attaches a parallel zero-start gated low-rank nonlinear residual to each adapted layer. At initialization the residual is exactly zero, so AuroOFT computes the same function as QOFT; during training the residual can learn token-dependent nonlinear corrections. Under matched data, optimizer, decoding, and parser protocols, the paper reports Macro-6 (the average of six non-SAT math benchmarks) gains of 1.30–2.70 points over QOFT on 1.5B and 3B Qwen2.5 models, and improvements of 6.52–10.62 points over QLoRA while using 32.3–44.7 percent fewer trainable parameters.","feed_headline":"A small gated branch lifts QOFT math scores by 2.7 points","feed_subtitle":"A zero-start gated residual adds input-dependent corrections to QOFT, beating QLoRA with fewer trainable parameters.","key_machinery":"The central object is the AuroOFT layer, a wrapped QOFT module augmented by a parallel gated low-rank nonlinear branch. The branch maps the same activation $x$ through a down projection $A_l\\in\\mathbb{R}^{r\\times d_{\\mathrm{in}}}$, an RMS-normalized compact latent space, an adaptive nonlinear layer $\\phi_l$ (Lite tanh, bounded B-spline SplineNorm, or a dual tanh/SwiGLU ANL), and a zero-initialized up projection $B_l$, scaled by $\\alpha/r$ and a scalar, bounded, or token-dependent gate $G_l$. The zero initialization of $B_l$ is what makes the layer start exactly as QOFT; the gate and scaling regulate how much nonlinear correction enters during training. This parallel placement isolates expressivity as an attribution variable: ablations vary only the residual-branch components while the QOFT carrier stays fixed.","core_discovery":"AuroOFT's central claim is that orthogonality should serve as a stable quantized backbone, not as a constraint on the function the adapter can learn. The combined layer keeps the QOFT rotation before frozen 4-bit quantized weights and adds a nonlinear residual $\\Delta h_l = G_l(\\tilde z_l) \\frac{\\alpha}{r} B_l \\mathrm{Drop}(\\phi_l(\\tilde z_l))$ with zero-initialized up projection $B_l$, making the initial model exactly equivalent to QOFT. The paper proves this zero-start equivalence, shows the QOFT function family is contained in the AuroOFT family, and states clearly that only the main branch is orthogonal while the full mapping is nonlinear and generally non-mergeable. Empirically, under matched conditions the residual branch improves Macro-6 by 1.30–2.70 points over QOFT in the 1.5B/3B settings, and the method exceeds QLoRA by 6.52–10.62 points while saving 32.3–44.7 percent of QLoRA's trainable parameters; results at 7B are mixed, and the small SAT set is explicitly treated as a protocol-sensitivity diagnostic rather than structural evidence.","pith_inferences":["The paper does not report multi-seed variance; a natural extension is to rerun the matched 1.5B/3B protocols over several seeds and check whether the 1.30–2.70 point deltas exceed run-to-run noise.","Because the residual is token-dependent, one testable prediction is that its gains concentrate on multi-step arithmetic or on examples where a local symbolic correction changes the answer; per-example error analysis could verify this.","The non-mergeability of the residual implies AuroOFT's practical value depends on whether the accuracy gain outweighs added latency; a latency-constrained deployment study would settle that trade-off.","The containment result suggests the method's theoretical advantage is expressivity, not optimization; a further study could test whether the residual branch also improves optimization stability by measuring loss landscapes or early-training gradients."],"forward_implications":["Since AuroOFT starts as QOFT, any divergence during training is attributable to the nonlinear residual branch, giving a controlled test of whether extra expressivity helps.","Improvements at 1.5B and 3B indicate that input-dependent nonlinear corrections matter for mathematical reasoning even when a stable orthogonal quantized path is present.","Beating QLoRA with fewer trainable parameters suggests the residual branch uses the parameter budget more efficiently than a pure low-rank additive adapter.","Because the residual is nonlinear, it cannot be merged into frozen weights, so AuroOFT carries extra inference cost per token; the paper flags this non-mergeability explicitly.","The mixed 7B results and the protocol-sensitive SAT diagnostic imply the method is not a universal improvement, and matched-protocol evaluation is required before attributing gains to the residual."],"supporting_citations":[{"why":"Provides QOFT/OFTv2, the quantized orthogonal carrier that AuroOFT wraps and the matched baseline the structural gains are measured against.","marker":"Qiu et al. 2025"},{"why":"Supplies the adaptive nonlinear layer principle (tanh and B-spline bases) that AuroOFT adapts as a residual branch.","marker":"Dong et al. 2025"},{"why":"Demonstrates nonlinear low-rank adaptation as a way to go beyond linear LoRA, motivating the nonlinear correction family.","marker":"Li, Song, and Hou 2024"},{"why":"Defines QLoRA, the parameter-efficiency baseline that AuroOFT claims to beat while saving 32.3–44.7 percent of trainable parameters.","marker":"Dettmers et al. 2023"},{"why":"Defines the LoRA two-projector linear update whose linearity both AuroOFT and its nonlinear relatives aim to escape.","marker":"Hu et al. 2022"},{"why":"Introduces orthogonal fine-tuning, the rotation-based adaptation idea that QOFT and AuroOFT build on.","marker":"Qiu et al. 2023"}],"fun_headline_variants":["AuroOFT: nonlinear bonus lifts QOFT up to 2.7 points","Zero-start gating adds expressive power to QOFT","Beyond rotations: AuroOFT beats QLoRA with 44% fewer params","AuroOFT: rotations for stability, nonlinearity for accuracy","Nonlinear residual: QOFT becomes more expressive, beats QLoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported Macro-6 gains over matched QOFT are single-seed point estimates, and the paper's structural-expressivity claim assumes that 1.30–2.70 point deltas are larger than run-to-run variation.","fun_headline_variants_meta":{"raw":{"variants":["AuroOFT: nonlinear bonus lifts QOFT up to 2.7 points","Zero-start gating adds expressive power to QOFT","Beyond rotations: AuroOFT beats QLoRA with 44% fewer params","AuroOFT: rotations for stability, nonlinearity for accuracy","Nonlinear residual: QOFT becomes more expressive, beats QLoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001484,"raw_usage":{"total_tokens":6022,"prompt_tokens":1070,"completion_tokens":4952,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":4852}},"tokens_in":686,"tokens_out":4952,"duration_ms":36391,"temperature":1.0,"reasoning_tokens":4852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:01:07.118263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the matched 1.5B and 3B Qwen2.5 protocols with AuroOFT and QOFT across at least five seeds under identical data order, optimizer, decoding, and parser settings, and compute the Macro-6 delta with confidence intervals. If the average delta is near zero or its interval overlaps zero, the central claim that the nonlinear residual expands QOFT's useful expressivity is not supported.","supporting_citations":[{"cited_title":"2024 , url =","cited_arxiv_id":null,"evidence_quote":"Demonstrates nonlinear low-rank adaptation as a way to go beyond linear LoRA, motivating the nonlinear correction family."}],"review_version":1}