{"id":"1ed7bcfd-3b86-470c-ad76-5099059e9618","arxiv_id":"2608.11342","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WFT approximates supervised fine-tuning for LLM personalization by transporting supervised residuals from author history to the current prompt through a dropout-estimated logit-space operator, without updating any weights.","lead":"Weightless Fine-Tuning personalizes LLMs by adjusting the model's output scores at generation time, using dropout-based estimates to move a correction from an author's past writing to the current prompt. It claims to match per-author fine-tuning quality on three LaMP benchmarks while using far less compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix B's coordinate-wise update is not the Section 3.1 transport operator; the experiments test a different method, so the central mechanism is unsupported.","rationale":"I read the paper in good faith. The method is clearly described, the empirical comparisons are internally consistent, and the formal appendix attempts to ground the operator in NTK theory, which is a strength. However, the central claim hinges on the cross-prefix transport operator being the mechanism that transfers supervised residuals to the prompt, and the implementation diverges from that operator in a way that is not analyzed. This is not a disagreement with external consensus; it is an internal gap between the stated theory and the evaluated algorithm. The reader's weakest assumption identified both the unverified isotropy condition and the Appendix B approximation; I agree with the latter as the more load-bearing issue, because it affects the interpretation of every experimental result, including the ablation and the logit-similarity analysis. If the authors can show on a small vocabulary that the coordinate-wise update closely tracks the full operator, or re-derive the method from the actual implementation, the concern would be resolved. Until then, the claim that WFT works by transporting residuals through a dropout-estimated cross-prefix transport operator is not supported by the experiments, and the verdict should remain conditional pending this check.","tokens_in":16574,"tokens_out":11922,"duration_ms":109013,"concrete_test":"Run both updates on identical dropout samples with a small transformer (e.g., GPT-2, vocabulary truncated to roughly 256 tokens): compute Δz_full = η Σ_s Σ_{t,s}(Σ_{s,s}+λI)^{-1} r_s and the Appendix B update Δz_app on the same prompt/training prefixes. Report the cosine similarity and relative L2 error between Δz_full and Δz_app. If they are not close (e.g., cosine below 0.9), the implemented method is not the Section 3.1 operator, and the central mechanism claim must be revised or the coordinate-wise approximation must be derived from the operator. This single check settles whether the experiments test the proposed transport mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is that the implemented WFT update is not the transport operator defined in Section 3.1. Section 3.1 defines the correction as Δz_t = η Σ_s M_{t←s} r_s with M_{t←s} = Σ_{t,s}(Σ_{s,s}+λI)^{-1}. In coordinates, the full operator update is (M_{t←s} r_s)_v = 1/(K-1) Σ_k t_k(v) Σ_{v'} U_s(k,v') w_s(v'), where w_s = (Σ_{s,s}+λI)^{-1} r_s. Appendix B instead computes Δz_t(v) = η/(S(K-1)) Σ_s Σ_k t_k(v) U_s(k,v) w_s(v), keeping only the v'=v term and averaging over source positions inside the same sum. No derivation shows that off-diagonal vocabulary transport is negligible; the paper only calls it a 'dropout-factored approximation.' Consequently, the results in Tables 1-3, the ablation that removes M, and the cosine-similarity analysis in Section 6.2 all exercise a diagonal rescaling of logits, not the cross-prefix transport operator invoked by the central claim. Separately, Proposition 1's Condition 2 (Σθ = σ²I) is not merely unverified but inconsistent with actual dropout: dropout masks rescale rows/columns of weight matrices, yielding a sparse, low-rank, anisotropic parameter covariance. The NTK-convergence statement in Proposition 1(b) therefore does not follow for real dropout. The empirical results may support a useful heuristic, but they do not establish the claimed transport mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Weightless Fine-Tuning (WFT), a decoding-time method that aims to approximate the effect of per-author supervised fine-tuning (SFT) on next-token distributions without updating the model weights. The method computes supervised residuals on an author's training sequence and transports them to the current prompt prefix through a cross-prefix transport operator estimated from dropout-augmented forward passes, motivated by an NTK-style local-linearization argument. Experiments on three LaMP benchmarks with Qwen3-8B and Llama-3.1-8B-Instruct report that WFT achieves the best average ROUGE scores, matches or exceeds SFT on individual tasks, and does so at a fraction of the effective compute. Additional results include a budget-controlled SFT comparison, ablations, a human evaluation, and a logit-space similarity analysis against one-step SFT.","tokens_in":16923,"tokens_out":4753,"duration_ms":43501,"significance":"If the transport mechanism were established, WFT would be a practically valuable method for per-author personalization, since it avoids per-author weight storage and training while operating through forward passes only. The paper contains several constructive elements: a clearly motivated problem, a budget-controlled comparison, an ablation study, and a human evaluation. However, the current manuscript does not substantiate the central mechanistic claim. The theory in Proposition 1 requires assumptions that are not met by real dropout, and, more importantly, the algorithm actually implemented in Appendix B is not the transport operator defined in Section 3.1. The reported experiments therefore support a diagonal logit-rescaling heuristic rather than the cross-prefix logit-space transport that the paper claims. These gaps are load-bearing and need to be resolved before the paper's central claim can be accepted.","major_comments":[{"comment":"The implemented update is not the Section 3.1 transport operator. Section 3.1 defines the correction as Delta z_t = eta * sum_s M_{t<-s} r_s, with M_{t<-s} = Sigma_{t,s} (Sigma_{s,s} + lambda I)^{-1}; in coordinates the full operator acts as (M_{t<-s} r_s)_v = (1/(K-1)) * sum_k t_k(v) * sum_{v'} U_s(k,v') w_s(v'). Appendix B instead computes Delta z_t(v) = eta * sum_j (1/(K-1)) * sum_k t_k(v) * B^{(j)}_k(v), where B^{(j)}_k(v) = (1/S) * sum_s U_s(k,v) w^{(j)}_s(v). This retains only the v'=v term and averages over source positions inside the coefficient matrix. No derivation is provided to show that off-diagonal vocabulary transport is negligible; the paper only calls this a 'vocabulary-coordinate-wise dropout-factored approximation.' Consequently, Tables 1-3, the ablation that removes M, and the cosine-similarity analysis in Section 6.2 exercise a diagonal rescaling of logits, not the cross-prefix transport operator invoked by the central claim.","section":"Appendix B vs. Section 3.1"},{"comment":"Proposition 1's Condition 2, Sigma_theta = sigma^2 I, is not merely unverified but inconsistent with the dropout mechanism actually used in the experiments. Dropout masks set individual neurons to zero and rescale the remaining entries, producing sparse, low-rank, anisotropic parameter covariance; they do not yield isotropic perturbations. Therefore the convergence result in part (b), M_hat_{t<-s} -> Theta_{t,s} (Theta_{s,s} + lambda' I)^{-1}, does not follow for the dropout-perturbed forward passes used to estimate the covariances. To make the theory applicable, the paper needs either to model the actual dropout-induced covariance and re-derive the estimator, or to provide a direct empirical test that the dropout-estimated operator matches the NTK action on a setting where the Jacobian can be computed exactly.","section":"Appendix A, Proposition 1"},{"comment":"The ablation that sets M to the identity does not test the cross-prefix transport operator, because the implemented method already replaces the operator with a coordinate-wise approximation. Setting M=I in the implemented algorithm only removes the dropout-factored reweighting w_s and leaves the diagonal, per-token action intact. The observed degradation therefore cannot be attributed to 'structured transport' or to the cross-prefix coupling that the paper's central claim relies on; it is equally consistent with the hypothesis that ridge-regularized, dropout-normalized token rescaling is what helps.","section":"Section 5.3, Table 3"},{"comment":"The cosine-similarity analysis of 0.875 is computed on the top-10 tokens ranked by Delta z_WFT, which is a post-hoc selected subset. Such selection can produce high similarity even when the two updates are only weakly aligned in general, because the subset is chosen to maximize the apparent agreement. The analysis is also based on a single randomly sampled author, so it provides no measure of variability across authors or contexts. The claim that 'WFT captures the distributional effect of supervised adaptation' would require either reporting similarity over the full vocabulary or with a proper baseline, and averaging over multiple authors.","section":"Section 6.2"}],"minor_comments":[{"comment":"The title and abstract contain a typo: 'Fine-T uning' should be 'Fine-Tuning'.","section":"Title/Abstract"},{"comment":"The main results are reported as point estimates without confidence intervals or significance tests, and the number of test authors is only 50; it would be helpful to report per-author variance and pairwise significance for the WFT-versus-SFT comparisons.","section":"Tables 1-2"},{"comment":"The human evaluation uses two annotators on 50 examples and reports only the aggregate preference count; inter-annotator agreement (e.g., Cohen's kappa) should be reported to gauge reliability.","section":"Section 5.4"},{"comment":"The complexity comparison assumes that K and E are of the same order and sets c >= 2 in T_b = c T_f; these choices are stated but not justified, and the 'effective budget B of WFT is approximately 100' is not derived rigorously from the algorithm's cost expression.","section":"Section 6.1"},{"comment":"The time-homogeneous approximation M^{(j)}_{t<-s} = M_{t<-s} is introduced without a bound on the approximation error or a discussion of when update sizes remain small enough for the first-order argument to hold.","section":"Section 3.2"},{"comment":"The column header 'Full WFT Component Steps Data' is confusing; it should be split into separate headers for 'Full WFT', 'w/o M', '1/2 steps', '1/4 steps', '1/2 data', and '1/4 data'.","section":"Table 3 header"}],"recommendation":"major_revision","confidential_remarks":"The reader's central concern is accurate: the implementation in Appendix B is not the Section 3.1 operator, and the paper does not bridge that gap. I would require the authors to either (a) implement and evaluate the full transport operator, or (b) provide a rigorous derivation of the coordinate-wise approximation and validate that it preserves the key cross-prefix behavior, before the paper can be accepted. The Proposition 1 assumptions also need to be reconciled with the actual dropout mechanism. If these points are addressed, the empirical study could support a useful heuristic; as it stands, the claimed mechanism is unsupported. There is no indication of circularity or data leakage in the evaluation setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is worth reading but not in its current form. The proposed WFT method sounds great on paper—estimate a cross-prefix transport operator from dropout covariance and apply SFT-style residuals in logit space—and the experiments on three LaMP tasks are reasonably careful. But the algorithm actually implemented in Appendix B is a coordinate-wise diagonal approximation that drops all off-diagonal vocabulary transport, and the paper never justifies that approximation. So the results in Tables 1–3, the ablation, and the cosine-similarity analysis test something different from the Section 3.1 operator. That's a load-bearing mismatch, not a nit.\n\nWhat's genuinely new: the specific idea of using dropout-induced cross-covariance to approximate the cross-prefix NTK action for personalization. I haven't seen that exact recipe in the cited work, and the KL-proximal NGD framing is a nice way to motivate logit-space corrections. The authors also get credit for running real 8B models on three datasets, doing an ablation, a budget-controlled SFT comparison, and a small human eval. The budget analysis, while rough, at least tries to compare end-to-end cost.\n\nWhere it falls down: first, the theory-implementation gap. Proposition 1 proves convergence of the full operator, but the implementation uses a diagonal coordinate-wise update. No argument shows off-diagonal transport is negligible, and the paper just calls it a 'dropout-factored approximation.' Second, Proposition 1's isotropic dropout assumption (Σθ = σ²I) is inconsistent with how dropout actually perturbs weights—dropout masks produce structured, low-rank, anisotropic noise. So the formal justification is doubly removed from the algorithm. Third, the empirical reporting is thin: no error bars or variance across author subsamples, no code release, and only 50 test authors per dataset. The human eval is only 50 examples with two annotators, so it's suggestive at best.\n\nDo I think the authors are dishonest? No. They admit the approximation in Section 3.2 and Appendix B. But the central claim is unsupported until they either implement the full operator or reframe the method as a diagonal heuristic and provide a separate justification.\n\nWho should read this: people working on decoding-time adaptation and personalization, and anyone interested in NTK approximations for LLMs. It's a legitimate research question with a novel recipe and fair empirical effort. I'd send it to peer review—it deserves referee time—but the recommendation should be major revision, not acceptance.\n\nFor my own work: I wouldn't cite it as a reliable method yet, but I'd keep an eye on a revised version.","headline":"A clever training-free personalization idea whose central transport mechanism never actually gets tested in the experiments.","tokens_in":17443,"tokens_out":3325,"would_cite":false,"duration_ms":29536,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The distributional effect of supervised fine-tuning can be reproduced at decoding time without updating any model weights.","keywords":["personalization","weightless fine-tuning","logit-space transport","cross-prefix transport operator","dropout covariance","neural tangent kernel","decoding-time adaptation","large language models"],"falsifier":"On a small model where the full Jacobian can be computed, compare the dropout-estimated $\\widehat M_{t\\leftarrow s}$ against the true cross-prefix NTK action $\\Theta_{t,s}\\Theta_{s,s}^{-1}$ as $K$ grows; if the two do not converge for residuals in the column span of $\\Theta_{s,s}$, then the isotropic dropout assumption is false and WFT's transport estimate is not an NTK approximation.","tokens_in":16378,"feed_emoji":"🧠","tokens_out":8755,"duration_ms":84893,"temperature":0.7,"pith_summary":"This paper proposes Weightless Fine-Tuning (WFT), a decoding-time method that aims to reproduce the effect of supervised fine-tuning (SFT) on a language model's next-token distribution without changing any weights. WFT computes supervised residuals on an author's training text—the difference between the model's predicted next-token distribution and the ground-truth token—and transports those residuals to the current prompt through a cross-prefix operator estimated from dropout-perturbed forward passes. The authors report that on three LaMP personalization tasks WFT matches or exceeds per-author SFT on average, using less than 7% of the effective computation, and that its induced logit shifts have cosine similarity 0.875 with SFT's shifts over 95% of the probability mass. If this holds, personalization no longer requires per-author fine-tuning runs, weight storage, or retraining when new author data arrives.","feed_headline":"Logit transport matches supervised fine-tuning at 7% of the cost","feed_subtitle":"Dropout-measured transport moves author residuals to the prompt, matching SFT with no per-author weights.","key_machinery":"The load-bearing object is the cross-prefix transport operator $M_{t\\leftarrow s}$, a $V\\times V$ map that converts a supervised residual computed at training position $s$ into a logit correction at prompt position $t$. It is estimated from the empirical cross-covariance $\\Sigma_{t,s}$ and self-covariance $\\Sigma_{s,s}$ of dropout-perturbed logits as $\\widehat M_{t\\leftarrow s}=\\Sigma_{t,s}(\\Sigma_{s,s}+\\lambda I)^{-1}$, which Proposition 1 identifies as a ridge-regularized approximation to the cross-prefix NTK action $\\Theta_{t,s}\\Theta_{s,s}^{-1}$. The theoretical mechanism that makes this possible is that a KL-proximal natural-gradient step is multiplicative in probability and additive in logits ($q\\propto p\\,e^{\\eta g}$), so a weight update can be replaced by adding a logit shift; the operator supplies that shift directly, and the multi-step version iterates residual recomputation on the training side while keeping the operator fixed.","core_discovery":"The paper's central claim is that the distributional effect of a supervised natural-gradient update can be moved from the training context to a different prompt context in logit space, without ever updating the weights. Concretely, the logit shift at prompt prefix $x_{1:t}$ induced by one supervised step is written as $\\Delta z_t = \\eta \\sum_{s=1}^S M_{t\\leftarrow s} r_s$, where $r_s = e_{\\tilde{x}_{s+1}} - p(\\cdot|\\tilde{x}_{1:s})$ is the supervised residual at training position $s$, and $M_{t\\leftarrow s}$ is a cross-prefix transport operator. WFT estimates this operator as $\\widehat M_{t\\leftarrow s} = \\Sigma_{t,s}(\\Sigma_{s,s}+\\lambda I)^{-1}$ from paired dropout-perturbed forward passes, and shows it converges to a ridge-regularized cross-prefix neural tangent kernel action $\\Theta_{t,s}(\\Theta_{s,s}+\\lambda' I)^{-1}$. The paper argues this transported residual approximates the logit shift one-step SFT would produce, and supports the claim with benchmark results and logit-level similarity between WFT and SFT shifts.","pith_inferences":["Inference: the method's theoretical guarantees require local linearity of logits in weights and isotropic dropout noise; on very large models where these fail, WFT's alignment with SFT should degrade, so the 0.875 similarity is likely an upper bound for harder reasoning-heavy tasks.","Inference: the implementation replaces the full $V\\times V$ operator with a vocabulary-coordinate-wise approximation that the paper does not derive from Section 3.1's operator; if the approximation rather than the full operator drives the empirical gains, the NTK interpretation may be stronger than necessary.","Inference: WFT's transport operator is estimated per author from that author's own text, so its quality should depend on how much author history is available; with very little history the covariance estimate will be rank-deficient and the ridge regularization will dominate, which should predict a sharp performance floor.","Inference: the same residual-transport idea could be tested for cross-task or cross-author transfer, where residuals from one author or task are transported to another prompt; the paper does not evaluate this, but the operator formalism does not restrict it to same-author data."],"forward_implications":["Per-author personalization becomes an inference-time operation: no per-author weight copies are stored, and incorporating new author data only requires recomputing residuals and covariance estimates rather than retraining.","At the reported budget, WFT reaches SFT-comparable quality with less than 7% of the effective computation, so serving thousands of authors becomes feasible on fixed hardware.","Because the base model weights are never modified, WFT avoids catastrophic forgetting and leaves the model's general capabilities intact for non-personalized tasks.","WFT operates through stochastic forward passes and output logits, so it can personalize models accessed through inference-only APIs where gradients and weight access are unavailable.","The high cosine similarity between WFT and SFT logit shifts suggests WFT captures the same distributional direction as supervised adaptation over the tokens that dominate next-token probability."],"supporting_citations":[{"why":"Supplies the LaMP benchmark and the three personalization tasks on which WFT is evaluated against SFT and baselines.","marker":"Salemi et al., 2024"},{"why":"Establishes per-author SFT as the strong quality baseline that WFT aims to approximate in distribution space.","marker":"Chakrabarty et al., 2025"},{"why":"Provides the LoRA setup used for the SFT baseline and for the budget-controlled training comparisons.","marker":"Hu et al., 2022"},{"why":"Gives the matrix identity that lets WFT invert the V by V covariance through a K by K dual Gram matrix, making the implementation tractable.","marker":"Woodbury, 1950"},{"why":"Used in the complexity analysis to estimate the backward-pass constant that underlies the claim that WFT uses less than 7% of SFT's compute.","marker":"Hoffmann et al., 2022"}],"fun_headline_variants":["Zero-weight fine-tuning matches SFT at 7% compute","No weight updates: logit transport achieves SFT quality","Personalize LLMs without training, at 7% of SFT cost","Logit transport replaces gradient updates for personalization","Match SFT performance with 7% compute and zero weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole construction stands on the assumption that dropout-perturbed forward passes reveal the same cross-prefix propagation that a real supervised weight update would cause, which requires the model's logits to be locally linear in the weights and the dropout perturbations to be isotropic; if that fails, the estimated transport operator no longer tracks SFT.","fun_headline_variants_meta":{"raw":{"variants":["Zero-weight fine-tuning matches SFT at 7% compute","No weight updates: logit transport achieves SFT quality","Personalize LLMs without training, at 7% of SFT cost","Logit transport replaces gradient updates for personalization","Match SFT performance with 7% compute and zero weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000999,"raw_usage":{"total_tokens":4272,"prompt_tokens":1032,"completion_tokens":3240,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":3155}},"tokens_in":648,"tokens_out":3240,"duration_ms":25021,"temperature":1.0,"reasoning_tokens":3155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:41.559392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small model where the full Jacobian can be computed, compare the dropout-estimated $\\widehat M_{t\\leftarrow s}$ against the true cross-prefix NTK action $\\Theta_{t,s}\\Theta_{s,s}^{-1}$ as $K$ grows; if the two do not converge for residuals in the column span of $\\Theta_{s,s}$, then the isotropic dropout assumption is false and WFT's transport estimate is not an NTK approximation.","supporting_citations":[],"review_version":1}