{"id":"5cf3d508-184a-46f9-b659-687aca9cf9ae","arxiv_id":"2505.14826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A greedy token-level Fisher information data selection method that reports improved sample efficiency for GPT-2 supervised fine-tuning on Shakespeare text relative to uniform, density, and AskLLM baselines.","lead":"FisherSFT selects the most informative training sentences for fine-tuning a language model, scoring each sentence by how much it increases a crude measure of what the model already knows. If the method holds up, fine-tuning an LLM on a new domain could need far fewer examples, which means lower compute costs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.1 rests on an unsatisfiable matrix assumption: diag(p)-pp^T is rank-deficient (all-ones vector is a null eigenvector), so no γ>0 exists; the determinant identity in its proof also misapplies det(A⊗B), swapping the powers of d and L.","rationale":"The reader's weakest_assumption identifies exactly the false matrix inequality in Lemma 3.1, and my independent check confirms it is unsatisfiable: diag(p)-pp^T is the covariance of a categorical draw and is always singular along the all-ones vector. This is not a disagreement with a modeling convention but a mathematical impossibility, so the lower bound motivating the greedy objective is not established. The paper also contains a determinant-algebra error in the same proof, swapping powers of d and L; even a repaired lower-bound assumption would yield a different (L-scaled) bound, not the stated d log det. Both issues directly undermine Theorem 4.3, not merely its constants. The empirical evaluation is suggestive but lacks code, error bars, and training details; however, the central empirical hypothesis (greedy log-det selection helps SFT) is not disproven by the theoretical failure. A CONDITIONAL verdict is appropriate: either prove a corrected lower bound (e.g., restricted to the identifiable subspace orthogonal to the all-ones direction, with a different objective or with Θ1 fixed and the corresponding row removed), empirically validate or replace Assumption 4.2, and provide reproducibility details. I agree with the reader that this is the load-bearing weakness; no additional concern is needed.","tokens_in":19640,"tokens_out":2910,"duration_ms":26554,"concrete_test":"Independently verify Lemma 3.1 numerically: (1) For any L>1 and any p on the simplex, compute the eigenvalues of diag(p)-pp^T; the all-ones vector gives eigenvalue exactly 0, so the smallest eigenvalue is 0 and no γ>0 satisfies the stated inequality. (2) For L=3, d=2, choose a random symmetric positive-definite B and compute det(I_3⊗B) numerically; compare with det(B)^3 (correct) versus det(B)^2 (as claimed in the proof). Either check settles whether Lemma 3.1 can be repaired as written.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central theoretical justification for greedy log-det maximization is Lemma 3.1. The lemma assumes diag(p(yi,j|xi,j;Θ)) - p(yi,j|xi,j;Θ)p(yi,j|xi,j;Θ)^T ⪰ γI_L for γ>0. This is impossible: for any probability vector p, (diag(p)-pp^T)1 = p - p(p^T1) = 0, so the matrix has a zero eigenvalue in the all-ones direction. Hence A ⪰ γI_L cannot hold for any positive γ, and the proof's step ∇²LS(Θ) ⪰ (γ/n) Σ_i Σ_j (I_L ⊗ xi,j xi,j^T) is invalid. Additionally, the determinant identity is misstated: Section 3.3 writes det(I_L ⊗ B) = det(I_L)^L det(B)^d, but Proposition 7.1.11 of Bernstein gives det(A⊗B) = det(A)^q det(B)^p for A∈R^{p×p}, B∈R^{q×q}; with p=L, q=d this is det(B)^L, not det(B)^d. Even under the (false) lower-bound assumption, the claimed d log det would be L log det. The singular direction corresponds to the translation-invariance of the softmax likelihood, which is not addressed by restricting Θ1=0 in Assumption 4.1. Because Lemma 3.1 is the sole bridge from the true Hessian to the computationally tractable objective log det(Σ_i xi,j xi,j^T), the theoretical guarantee for FisherSFT's selection rule is unsupported. The empirical results may still be valid, but the paper's claim of an analyzable, information-theoretically motivated method is not backed by the presented proof.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FisherSFT, a data-selection method for supervised fine-tuning of LLMs. It formulates the last layer of an autoregressive model as a multinomial logistic regression, approximates the Hessian of the negative log-likelihood by a per-token covariance matrix, and greedily selects sentences that maximize the log-determinant of the accumulated embedding covariance. The authors present a fast lazy-greedy implementation, a theoretical O(1/sqrt(n)) maximum-prediction-error bound (Theorem 4.3), and experiments on synthetic data, word2vec embeddings, and GPT-2 fine-tuning on TinyShakespeare, where FisherSFT outperforms uniform, density, clustering, sentence-level optimal design, and AskLLM baselines. The central theoretical justification for the selection objective is Lemma 3.1, which is used to lower-bound the log-determinant of the true Hessian by the log-determinant of the tractable design matrix.","tokens_in":20034,"tokens_out":6050,"duration_ms":56352,"significance":"If the empirical findings are robust, the paper contributes a simple, computationally efficient connection between LLM fine-tuning data selection and D-optimal experimental design. The synthetic experiments are well designed because the ground-truth softmax parameter is known, and they provide direct evidence that the greedy log-det rule is a reasonable heuristic at fixed sample budgets. The use of a held-out LLM judge for the GPT-2 experiments is a useful complement to the synthetic metrics. The method has no fitted selection constants apart from the ridge regularization sigma0, and the lazy-greedy acceleration is a practical contribution. However, the advertised theoretical guarantee is currently unsupported: the key matrix inequality in Lemma 3.1 is unsatisfiable, and the determinant identity used in its proof is misstated. The empirical value of the heuristic is not necessarily affected by these issues, but the claim that the method is 'analyzable' with a valid information-gain bound is not established by the manuscript as written.","major_comments":[{"comment":"The assumption diag(p(y|xi,j;Theta)) - p(y|xi,j;Theta)p(y|xi,j;Theta)^T >= gamma I_L for gamma > 0 is unsatisfiable for any probability vector p: multiplying by the all-ones vector gives (diag(p)-pp^T)1 = p - p(p^T1) = 0. The matrix is always singular, so the inequality cannot hold. This invalidates the step in Section 3.3 that concludes Hessian >= (gamma/n) sum_i sum_j I_L (x) xi,j xi,j^T. Because this step is the sole bridge from the true Hessian to the tractable objective log det(sum xi,j xi,j^T), the theoretical motivation for the selection rule is not established. Restricting attention to the subspace {Theta : Theta 1 = 0} may salvage a positive-definiteness statement, but that restriction is not used in Lemma 3.1.","section":"Lemma 3.1 / Section 3.3"},{"comment":"The proof of Lemma 3.1 misapplies the Kronecker determinant identity. For A in R^{p x p} and B in R^{q x q}, det(A (x) B) = det(A)^q det(B)^p. With A = I_L and B = (gamma/n) sum xi,j xi,j^T, the correct lower bound is det(B)^L, not det(B)^d. The manuscript writes det(I_L)^L det(B)^d and concludes log det(Hessian) >= d log det((gamma/n) sum xi,j xi,j^T). Since L is the vocabulary size and generally L >> d, the stated lower bound does not follow; the proof would yield an L log det bound, which is a different and much larger quantity. This is a second load-bearing error in the derivation of the objective.","section":"Section 3.3, determinant identity"},{"comment":"Theorem 4.3 depends on Assumption 4.2, a per-step diversity ratio kappa that is not verified in the experiments or argued from the greedy choice. As stated in the main text, the assumption quantifies over i in S_{t-1}, which appears to be a typo for the unselected set; either way, it is not justified. The greedy rule only guarantees that the selected sentence has the largest log-det increment among the remaining sentences at each step, so a version of the inequality with kappa = 1 holds for unselected sentences by definition. If the intended assumption is different, the manuscript needs to state it correctly and justify why kappa is bounded. Without this, the displayed O(1/sqrt(n)) bound is not a standalone guarantee on the algorithm's behavior.","section":"Assumption 4.2 and Theorem 4.3"}],"minor_comments":[{"comment":"The input line writes {x_i = (x_{i,j})_{N_i}_{j=1}}_i, but the length of sentence i is denoted M_i in the body; the notation should be made consistent.","section":"Algorithm 1"},{"comment":"The statement of Lemma 4.4 omits the regularization term: it defines Sigma_S as a sum of xi,j xi,j^T only, while Assumption 4.1 and the proof use Sigma_S = sigma_0 I + sum xi,j xi,j^T. This should be corrected.","section":"Lemma 4.4"},{"comment":"The LLM-judge results are reported as point estimates without confidence intervals, and one entry is 0.52 with a reported position bias of 0.54. A short discussion of statistical significance or a bias-corrected estimate would make the comparison more convincing.","section":"Table 1"},{"comment":"The legend labels are inconsistent between the two figures, e.g., 'Greedy Sentence-OD' versus 'SentenceOD' and 'DensitySampling' versus 'Density Sampling'. Please unify the naming in the figures and the text.","section":"Figures 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is plausible and the synthetic evaluation is well designed, but the theoretical claim is not currently valid as stated. If the proof issues in Lemma 3.1 and the determinant identity cannot be repaired, the authors should downgrade the theoretical contribution to a heuristic motivation and state the per-step approximation assumption as an empirical condition. The paper is a reasonable candidate for major revision rather than rejection because the heuristic itself is sensible and the experimental evidence is supportive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about arXiv:2505.14826. First, the core idea is sensible and potentially useful: treat SFT data selection as a D-optimal design problem on token-level embeddings, and greedily maximize the log-det of the accumulated covariance. The fast lazy-greedy implementation in Algorithm 2, with cached gains and parallel batch updates, is a genuine practical contribution. Second, the theoretical results as stated are not valid. Lemma 3.1 assumes diag(p) - pp^T ⪰ γI_L for γ > 0. That matrix is the covariance of a single categorical draw and always has the all-ones vector as a null eigenvector, so no positive γ exists. The proof also misapplies the Kronecker determinant: for A ∈ R^{L×L} and B ∈ R^{d×d}, det(A⊗B) = det(A)^d det(B)^L, not det(A)^L det(B)^d. So the lower bound leading to the log-det objective is unproven, and the error bound in Theorem 4.3 rests on it, plus the unverified per-step approximation ratio Assumption 4.2.\n\nWhat is actually new: token-level greedy log-det selection under a softmax approximation, extending earlier sentence-level optimal design and D-optimal design for RLHF reward models. The fast cached implementation is new and, by itself, worth having.\n\nEmpirically, the synthetic and word2vec experiments are clean and show FisherSFT beating uniform, density, clustering, and sentence-level OD on both maximum and mean prediction error. The GPT-2 results with an LLM judge are suggestive but under-reported: no code, no error bars, no fine-tuning hyperparameters, and the judge shows a 0.54 first-position bias that is not accounted for. None of this disproves the heuristic — the greedy D-optimal idea is known to work in linear models — but the paper does not provide enough detail to replicate the real-world claims.\n\nBottom line: the methodological core is a useful heuristic, the fast algorithm is a real contribution, and the empirical evidence is consistent with the heuristic. The theory is a load-bearing wall that is not standing, and the authors should be asked to fix or explicitly downgrade Lemma 3.1, verify or replace Assumption 4.2, and release code with proper statistical reporting. This deserves a serious referee — the topic is timely and the failure modes are fixable — but as-is it should not be accepted.","headline":"Useful heuristic with a genuinely fast lazy-greedy implementation, but the theory as stated does not hold; treat this as an empirical paper with a broken Theorem.","tokens_in":20549,"tokens_out":3843,"would_cite":false,"duration_ms":33957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","68T07","90C27"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes FisherSFT, a greedy sentence-selection method that maximizes a log-determinant surrogate for the Fisher information of the softmax log-likelihood, and claims lower prediction error and better generated text at fixed…","keywords":["data-efficient fine-tuning","supervised fine-tuning","Fisher information","optimal design","submodular maximization","log-determinant","data selection","large language models"],"falsifier":"Take any vocabulary size $L$ and any softmax vector $p$ with equal entries, $p=(1/L,\\dots,1/L)$. A direct eigenvalue calculation shows that $\\mathrm{diag}(p)-pp^\\top$ has a zero eigenvalue with eigenvector $(1,\\dots,1)$, so Lemma 3.1's $\\gamma$ condition is violated at the very point where the softmax is most uniform; hence the stated lower bound cannot be established by that lemma. An empirical check would be to run FisherSFT on a dataset whose token embeddings are all identical, where the log-det objective gives every sentence the same score; if the $O(1/\\sqrt{n})$ error bound still holds in that setting, some other mechanism must be responsible.","tokens_in":19454,"feed_emoji":"🎯","tokens_out":8067,"duration_ms":82324,"temperature":0.7,"pith_summary":"Supervised fine-tuning of a language model costs roughly in proportion to the number of training sentences, so choosing which sentences to spend that budget on is a direct path to cheaper adaptation. This paper proposes FisherSFT, which treats the fine-tuning loss as a product of multinomial logistic regressions over the model's pre-logit embeddings and selects sentences that maximize a tractable surrogate for the Fisher information of those regressions. The surrogate is the log determinant of the accumulated token-embedding covariance matrix, and the selection is done greedily, exploiting the monotone submodularity of log determinant for speed. The paper reports that, at the same budget, models fine-tuned on FisherSFT-selected sentences have lower maximum and mean prediction error than uniform, density, clustering, and sentence-level optimal-design baselines on synthetic and word2vec tasks, and that GPT-2 fine-tuned on its selections produces text an LLM judge prefers. If the claim holds, fine-tuning budgets can be spent on fewer, more informative sentences without sacrificing quality.","feed_headline":"Greedy log-det selection needs fewer fine-tuning samples","feed_subtitle":"At equal budgets, models trained on FisherSFT-selected sentences beat uniform, density, and cluster baselines.","key_machinery":"The workhorse is the design matrix $V = I_d + \\sum_{(i,j) \\in \\text{selected sentences}} x_{i,j}x_{i,j}^\\top$, built from pre-logit embeddings $x_{i,j}$ of each token. Greedily choosing the sentence that maximally increases $\\log \\det V$, with $V$ updated after each pick, is a monotone submodular maximization, so a caching scheme (Algorithm 2) can skip recomputing gains that cannot improve and evaluate candidate sentences in parallel batches. The log-determinant criterion is the volume of the ellipsoid defined by $V$, so selection favors sentences whose tokens explore new embedding directions and disfavors redundancy. The theory attaches to this procedure a maximum-prediction-error bound via a strong-convexity argument that uses the softmax Hessian's Kronecker form and a concentration bound on the score gradient.","core_discovery":"The central claim is that information gain for supervised fine-tuning can be measured almost entirely through the covariance of token embeddings in the last layer. The paper shows that the Hessian of the softmax negative log-likelihood has a Kronecker structure, with one factor depending on softmax probabilities and the other on the outer products $x_{i,j}x_{i,j}^\\top$; Lemma 3.1 asserts that, under a uniform positive-definiteness condition on the probability factor, the log determinant of the full Hessian is bounded below by $d$ times the log determinant of the averaged embedding covariance. Maximizing that lower bound reduces to a $d\\times d$ greedy optimal-design problem, over sentences rather than individual tokens. The paper's Theorem 4.3 then bounds the maximum prediction error of the maximum-likelihood estimator on the selected sentences by $\\tilde{O}(dL/\\sqrt{n})$ up to problem constants, under Assumption 4.2's per-step approximation ratio. Empirically, the method outperforms the compared sampling baselines on synthetic data, on word2vec features, and in a GPT-4o-judged text-generation comparison, where human-visible samples show less repetitive output than uniform sampling.","pith_inferences":["A testable extension, not pursued in the paper, is to apply the same greedy log-det selection to embeddings extracted at earlier layers or from adapter weights; the selection objective is layer-agnostic, though the theory's proof would need re-checking for that setting.","Because the selection score uses only embedding geometry and never the label tokens, the method's predicted advantage is largest when last-layer softmax structure dominates the task; domains where reasoning happens in earlier layers may show smaller gains.","The proof's $O(e^{2L})$ vocabulary factor is an artifact of a uniform strong-convexity bound; a corrected analysis on the $L-1$-dimensional quotient space after removing the all-ones logit direction could plausibly replace it with a factor of $L-1$.","An empirical check that would sharpen the practical claim is to compare FisherSFT against random selection on datasets with many near-duplicate sentences; the log-det objective should be robust to such redundancy, which would confirm that the gain comes from information content rather than coverage."],"forward_implications":["At a fixed budget of $n$ sentences, FisherSFT spends the budget on sentences whose token embeddings explore new directions, so redundant sentences do not consume fine-tuning compute.","The maximum prediction error of the selected-data estimator is proved to decay as $O(1/\\sqrt{n})$ up to log factors, matching the rate of recent optimal-design bounds.","On the synthetic and word2vec tasks, FisherSFT reaches the best baseline's error with fewer sentences; in the synthetic setup, the best baseline's error at $n=2000$ is matched by FisherSFT at $n=1000$.","In the GPT-2 Shakespeare experiments, an LLM judge preferred text from models fine-tuned on FisherSFT-selected data over uniform, density, and AskLLM baselines at every tested budget from 100 to 5000 sentences.","Algorithm 2 produces the same selected subset as the naive greedy Algorithm 1 while exploiting monotone submodularity and batch parallelism to cut computation."],"supporting_citations":[{"why":"Provides the greedy approximation guarantee for monotone submodular functions, which justifies the greedy sentence-selection algorithm.","marker":"Nemhauser et al., 1978"},{"why":"Supplies the optimal-design framework linking Fisher information matrices to log-determinant selection criteria.","marker":"Pukelsheim, 2006"},{"why":"Underpins the confidence-interval view of the Hessian that motivates maximizing the design matrix.","marker":"Abbasi-Yadkori et al., 2011"},{"why":"Supplies the Kronecker determinant identity used to reduce the $dL \\times dL$ Hessian to a $d\\times d$ objective in Lemma 3.1.","marker":"Bernstein, 2009"},{"why":"Provides the per-token strong-convexity claim used in Lemma 4.5 to lower-bound the loss gap.","marker":"Hajek et al., 2014"},{"why":"Defines the DensitySampling and AskLLM baselines and the data-selection framework the experiments compare against.","marker":"Sachdeva et al., 2024"},{"why":"Introduces the GPT-2 architecture used in the real-world fine-tuning experiments.","marker":"Radford et al., 2019"},{"why":"Supplies the pretrained word2vec embeddings used in the real-feature experiments.","marker":"Mikolov et al., 2013"}],"fun_headline_variants":["FisherSFT selects the most informative examples for fine-tuning","Information gain via last-layer Hessian trims SFT data","Greedy log-det picks beat uniform sampling for SFT","Fewer fine-tuning samples with Fisher information selection","Data-efficient SFT: pick examples maximizing Fisher info"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that the greedy log-determinant objective is a valid proxy for Fisher information rests on a per-token assumption that the matrix $\\mathrm{diag}(p)-pp^\\top$ is uniformly positive definite, but for a softmax probability vector this matrix always has a zero eigenvalue along the all-ones direction, so the assumed constant cannot exist.","fun_headline_variants_meta":{"raw":{"variants":["FisherSFT selects the most informative examples for fine-tuning","Information gain via last-layer Hessian trims SFT data","Greedy log-det picks beat uniform sampling for SFT","Fewer fine-tuning samples with Fisher information selection","Data-efficient SFT: pick examples maximizing Fisher info"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2803,"prompt_tokens":929,"completion_tokens":1874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1794}},"tokens_in":545,"tokens_out":1874,"duration_ms":13942,"temperature":1.0,"reasoning_tokens":1794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:13.492306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any vocabulary size $L$ and any softmax vector $p$ with equal entries, $p=(1/L,\\dots,1/L)$. A direct eigenvalue calculation shows that $\\mathrm{diag}(p)-pp^\\top$ has a zero eigenvalue with eigenvector $(1,\\dots,1)$, so Lemma 3.1's $\\gamma$ condition is violated at the very point where the softmax is most uniform; hence the stated lower bound cannot be established by that lemma. An empirical check would be to run FisherSFT on a dataset whose token embeddings are all identical, where the log-det objective gives every sentence the same score; if the $O(1/\\sqrt{n})$ error bound still holds in that setting, some other mechanism must be responsible.","supporting_citations":[{"cited_title":"L., Wolsey, L","cited_arxiv_id":null,"evidence_quote":"Provides the greedy approximation guarantee for monotone submodular functions, which justifies the greedy sentence-selection algorithm."},{"cited_title":"Optimal Design of Experiments, volume 50 of Classics in Applied Mathematics","cited_arxiv_id":null,"evidence_quote":"Supplies the optimal-design framework linking Fisher information matrices to log-determinant selection criteria."},{"cited_title":"Improved algorithms for linear stochastic bandits","cited_arxiv_id":null,"evidence_quote":"Underpins the confidence-interval view of the Hessian that motivates maximizing the design matrix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kronecker determinant identity used to reduce the $dL \\times dL$ Hessian to a $d\\times d$ objective in Lemma 3.1."},{"cited_title":"Minimax-optimal Inference from Partial Rankings","cited_arxiv_id":"1406.5638","evidence_quote":"Provides the per-token strong-convexity claim used in Lemma 4.5 to lower-bound the loss gap."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Introduces the GPT-2 architecture used in the real-world fine-tuning experiments."},{"cited_title":"S., and Dean, J","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained word2vec embeddings used in the real-feature experiments."}],"review_version":1}