{"id":"7a6e463d-7919-4c75-bb40-9ae72a72b104","arxiv_id":"2601.22068","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training only per-member singular values of pretrained weight matrices, while freezing singular vectors, produces an implicit ensemble with calibration near deep ensembles at <1% parameter overhead.","lead":"This paper introduces a cheap way to build a model ensemble for uncertainty estimates: freeze the 'knowledge directions' of a pretrained model's weights and only train a small set of scaling factors per ensemble member. The goal is to give small teams deep-ensemble-level calibration for less than 1% extra parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SVE's central claim depends on member diversity, yet no quantitative diversity/collapse metric is reported; with shared frozen U,V and 1% init noise, members may converge to identical singular values, voiding the uncertainty estimate.","rationale":"The paper makes a clear, plausible proposal and provides broad empirical evidence across vision and NLP (22M–7B backbones, calibration, OOD, distribution shift, efficiency). The efficiency calculation (Eq. 7) is transparent, and the M-ablation (Table 6) shows monotone gains with ensemble size, which is indirect evidence that members are not entirely degenerate. However, the mechanism of the method—diversity through per-member singular values—is never directly measured. The qualitative Appendix B trajectories are not a substitute for a collapse statistic. This is the same weak point the reader flagged, and it is the most load-bearing because if members collapse, the method's uncertainty estimates are not epistemically meaningful, regardless of ECE numbers. The 'state-of-the-art calibration' overclaim in the contributions is secondary and already tempered by the paper's own tables. I therefore agree with the reader's CONDITIONAL verdict: accept the contribution as a promising baseline, conditional on releasing code and adding a quantitative diversity/collapse analysis and matched baselines.","tokens_in":18182,"tokens_out":5769,"duration_ms":67546,"concrete_test":"Run a heads-only control: train SVE as usual, but after training compute (i) mean pairwise cosine distance / relative L2 distance between learned Σ^(m) across all layers, and (ii) mean pairwise KL divergence between members' predictive distributions on the ID test set and on OOD (e.g., CIFAR-100→CIFAR-10). Compare to the same quantities for an ablation in which all M members are initialized with identical Σ (ε=0) and only the classification heads differ. If the standard SVE's diversity metrics are not significantly above the heads-only control, or if the control reproduces SVE's ECE and OOD AUROC, the diversity premise is falsified; if SVE shows clearly larger disagreement and better calibration than the control, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the M members produce genuinely different predictive functions, so their average captures epistemic uncertainty. Equations 4–5 instantiate members by freezing U,V and training only Σ^(m), initialized as Σ⊙(1+ε^(m)) with ε ~ N(0, 0.01²). The objective in Eq. 6 is a sum of per-member cross-entropy terms; there is no coupling or diversity penalty between members. Under gradient descent with shared mini-batches, the expected gradient for each member is the same function of the data, and the only symmetry-breaking mechanisms are the small multiplicative init noise and stochastic batch order. Nothing prevents the members from collapsing to nearly identical Σ^(m), or from differing only in the separate classification heads. If they collapse, the ensemble reduces to a single feature extractor with M heads, and the reported improvements in ECE/OOD AUROC should not be attributed to singular-value diversity. Appendix B is the paper's only evidence for diversity, and it is qualitative: it plots relative changes of top singular values from pretrained values, but does not report pairwise distances among members' Σ vectors or prediction disagreement on ID/OOD inputs. The manuscript's Limitation section also does not discuss collapse. Since the central claim of accessible epistemic uncertainty relies on diversity, this missing quantitative validation is the weakest load-bearing link.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Singular Value Ensemble (SVE), an implicit ensembling method for fine-tuned foundation models. For each pretrained weight matrix W, SVE computes the SVD, freezes the singular vectors U and V, and trains M copies of the singular values Σ^(m), initialized with small multiplicative noise. Members share the same singular-vector \"knowledge basis\" but differ in how strongly each direction is used, and their softmax outputs are averaged at test time. The paper argues this creates a diverse ensemble with less than 1% parameter overhead. Experiments cover vision (Flowers102, CIFAR-100, DTD, Oxford Pets, CIFAR-100-C, OOD detection) and NLP (SST-2, ARC-Easy with LLaMA-2-7B), comparing against deep ensembles, LoRA-Ensemble, MC Dropout, Batch Ensemble, and Bayesian LoRA variants.","tokens_in":18520,"tokens_out":8919,"duration_ms":91228,"significance":"The core idea is simple and potentially valuable: if singular vectors of pretrained weights are meaningful knowledge directions, then rescaling singular values is a very cheap way to generate ensemble members. The parameter-count derivation (Eq. 7) is explicit, and the experimental sweep from BERT-base to LLaMA-2-7B is a strength. However, the central mechanism—diversity of per-member singular values—is only supported by qualitative plots, and the strongest calibration comparisons on ARC-Easy rely on imported baselines. If the diversity concern is addressed with direct quantitative metrics and the baseline comparison is made controlled, the paper would be a solid contribution to efficient uncertainty quantification.","major_comments":[{"comment":"The central mechanism is member diversity, but the paper provides no quantitative evidence that members do not collapse. Eq. (5) initializes each Σ^(m) with a 1% multiplicative Gaussian perturbation; Eq. (6) trains all members jointly with no diversity penalty or coupling. With shared mini-batches, the expected gradient is the same for all members, so nothing in the objective prevents convergence to a common Σ; the ensemble would then reduce to one feature extractor with M heads. Appendix B shows only relative changes of top singular values from pretrained values, not pairwise distances among Σ^(m) or prediction disagreement on ID/OOD inputs. Please report quantitative diversity metrics (e.g., average pairwise cosine distance of trained Σ^(m), softmax disagreement between members) and relate them to the ECE/OOD improvements; otherwise the reported gains cannot be attributed to the singul","section":"§4, Eqs. (5)-(6), Appendix B"},{"comment":"The headline ARC-Easy calibration results are not obtained under a controlled comparison. Baselines such as Deep Ensemble (M=3), Ckpt Ensemble (M=3), Bayes-LoRA, Blob (N=5/10), and LoRA-Ensemble (M=5) are imported from Yang et al. (2024) and Wang et al. (2023), with different training budgets and ensemble sizes, while SV-Ensemble uses M=16 under the authors' own pipeline. This mixes the method with the training budget and makes the claim that SV-Ensemble 'approaches' Blob's calibration incomparable. Please re-run the baselines with the same protocol, or clearly mark the imported rows and limit the conclusions to the rows that are directly compared.","section":"Table 3 (ARC-Easy)"},{"comment":"The contributions claim 'state-of-the-art calibration' and the abstract claims SVE 'performs comparable to an explicit ensemble.' In Table 1 (CIFAR-100), LoRA-Ensemble achieves ECE 1.1±0.2 vs SV-Ensemble 2.1±0.2 and better NLL; in Table 4 (OOD), LoRA-Ensemble has higher AUROC/AUPRC and lower FPR@95. The claims should be qualified to name LoRA-Ensemble as a stronger method on these benchmarks. As written, the abstract and contribution list overstate the empirical standing of SVE.","section":"Abstract and §1 Contributions"}],"minor_comments":[{"comment":"Backbone entries are inconsistent: Table 1 headers say Flowers102 uses 'DINO ViT-S/14' while Table 7 lists 'DINO ViT-S/16'; there is also a typo 'DINOv ViT-S/16'. Please harmonize the naming.","section":"Appendix C, Table 7"},{"comment":"The text says σ_init < 1 'guarantees positivity', but 1+ε can be negative for ε < -1, even if unlikely for σ_init=0.01. Rephrase as 'with high probability' and state how negative values are handled during training.","section":"§4, Eq. (5)"},{"comment":"The main text says σ_init=0.01 in most experiments, but Table 7 lists 0.005 for Flowers102, CIFAR-100, and DTD. Please reconcile these values.","section":"§4 and Appendix C"},{"comment":"The reference to 'Bommasani, R. (2021)' should be 'Bommasani et al.' for the Foundation Models report, which is a multi-author document.","section":"References"},{"comment":"The manuscript is marked 'Preliminary work' and code is promised only 'at a later stage.' Adding a reproducibility statement or releasing code would materially strengthen the paper's claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The diversity-collapse concern is the main technical risk. I would not require new theory, but a quantitative diversity analysis (pairwise Σ distances and prediction disagreement) is necessary before the mechanism can be accepted. The ARC-Easy baseline comparison also needs to be made controlled or clearly relabeled. The central idea is promising and the experiments are broad; with those additions the paper could be a good contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SVE. The core idea is clean and the experiments are broad; I think the paper is worth a serious referee, with revisions.\n\nWhat's new: train per-member singular values on top of frozen singular vectors, sharing the basis across members. That's a natural extension of SVF/SVFit and LoRA-Ensemble, but it hasn't been done before, and the parameter-count claim (under 1%) is real. The empirical coverage is genuinely useful: DINO/BERT/LLaMA across calibration, OOD, corruption shift. Best or second-best on most of their own tables is accurate; on Flowers, DTD, Pets it leads, on SST-2 it has the best ECE, and on CIFAR-100 LoRA-Ensemble beats it. The paper would be stronger if they tempered the 'state-of-the-art calibration' line to match their own data.\n\nThe weak spots are not fatal but are real. The diversity premise—members converging to different rescalings—is the load-bearing assumption, and the only evidence is qualitative plots of singular value trajectories in Appendix B. No pairwise distance between members, no prediction disagreement, no quantitative collapse check. Given that the training objective has no diversity penalty, the worry about collapse is legitimate, and a paragraph or a figure would answer it. The ARC-Easy table imports baselines from other papers with different training setups; that's a mismatch for a comparison table. The code isn't out yet. Inference FLOPs scale with M, as the paper honestly says in Limitations, so the 'accessible' claim is about parameters and memory, not compute.\n\nOn citations: the main comparison, LoRA-Ensemble, is the authors' own prior work, but they compare it fairly and report it losing in some settings, so I don't see a problem there. The knowledge-directions hypothesis is borrowed from prior interpretability work, and they say so.\n\nWho it's for: anyone working on parameter-efficient fine-tuning or UQ for foundation models. It's a solid baseline, not a breakthrough. I'd send it to a venue where empirical UQ papers get a sharp review—the diversity question and the baseline matching need to be addressed, but the method itself deserves referee time.","headline":"A useful, well-tested parameter-efficient ensemble idea whose claims outrun its evidence in two places—a missing diversity check and imported baselines—but worth serious review.","tokens_in":19040,"tokens_out":2576,"would_cite":true,"duration_ms":28427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an ensemble of foundation-model adapters can be built by freezing each weight matrix's singular vectors and training only per-member singular values, matching deep-ensemble calibration with under 1% parameter overhead","keywords":["singular value decomposition","implicit ensembling","uncertainty quantification","calibration","parameter-efficient fine-tuning","foundation models","deep ensembles","expected calibration error"],"falsifier":"Train SVE with M=16 and measure average pairwise disagreement among members on test inputs (e.g., fraction of inputs with differing argmax, or variance of softmax probabilities), then compare with a 16-member deep ensemble. If SVE members disagree far less yet show similar ECE, the claimed diversity mechanism is not what drives the calibration. A second check: randomize the frozen singular vectors (replace U,V with random orthogonal matrices) and retrain — if calibration still holds, the knowledge-basis assumption is not load-bearing; if it fails, freezing meaningful directions is essential.","tokens_in":18086,"feed_emoji":"🧩","tokens_out":5921,"duration_ms":59262,"temperature":0.7,"pith_summary":"This paper tries to make epistemic-uncertainty estimation practical for large pretrained models. Its core move is to treat the singular vectors of each pretrained weight matrix as a shared 'knowledge basis' and create ensemble members that differ only in the singular values, i.e., in how strongly each direction contributes. The paper claims that training several copies of the singular values — with small random perturbations and ordinary mini-batch sampling — produces functionally diverse members, and that averaging their predictions yields calibration comparable to a deep ensemble while adding under 1% to the parameter count. Support comes from fine-tuning experiments on vision and NLP benchmarks with backbones from 22M to 7B parameters, including out-of-distribution detection and corrupted-input stress tests. A sympathetic reader would care because it suggests that calibrated uncertainty need not cost an extra copy of the model.","feed_headline":"Cut ensemble overhead below 1% by training only singular values","feed_subtitle":"A new method matches deep-ensemble calibration on vision and NLP by reweighting each layer's frozen singular directions.","key_machinery":"The central object is a singular value decomposition used as an ensembling parameterization. For each target weight matrix, the pretrained singular vectors U and V are frozen and shared by all members; they encode the paper's hypothesized 'knowledge directions.' The only trainable per-member parameters are the diagonal singular-value vectors Σ(m), one per member, so the overhead per layer is the length of that diagonal instead of a full low-rank adapter matrix pair. Diversity is meant to arise from (i) multiplicative Gaussian perturbation at initialization and (ii) random mini-batch order during joint training, which makes members land on distinct rescalings of the same basis. This is also w","core_discovery":"The central claim is that repeated rescaling of one orthonormal singular-vector basis is enough to generate ensemble diversity. Concretely, SVE decomposes each pretrained weight matrix W as UΣV⊤, freezes U and V, replaces Σ with M trainable copies Σ(m) initialized as Σ⊙(1+ε(m)) with ε(m) small Gaussian noise, and reconstructs member weights as W(m)=UΣ(m)V⊤. Members share every layer's singular vectors and differ only in the per-member diagonal rescaling; separate classification heads are added per member. The paper reports that with M=4–16 members, SVE matches or beats deep ensembles on calibration metrics (ECE, NLL, Brier) on most tested settings, stays competitive in accuracy, and does so","pith_inferences":["Editorial: A quantitative diversity check is the natural next step. The paper only shows singular-value trajectories; if pairwise prediction disagreement among SVE members is much lower than among deep-ensemble members, the reported calibration gains may come from averaging artifacts rather than genuine epistemic diversity.","Editorial: The knowledge-direction premise suggests a geometric reading of uncertainty: a model is uncertain when members strongly disagree about how to reweight shared directions. That could connect uncertainty estimates to interpretability tools that examine a layer's singular vectors.","Editorial: Because the trainable parameters are just one vector per layer, SVE members can be stored as tiny deltas on top of the frozen base, enabling on-device uncertainty with essentially no extra weight memory — an extension the paper does not quantify.","Editorial: A testable extension is to learn which singular directions matter for a task and perturb only those, instead of all directions uniformly, potentially improving diversity and calibration further."],"forward_implications":["Calibrated epistemic uncertainty becomes affordable for single-GPU fine-tuning of large models, not just for well-resourced labs.","On ARC-Easy with a 7B-parameter model, SVE at M=16 reaches ECE 3.8%, nearly matching the Bayesian BLoB method (3.6%) while remaining far simpler and cheaper, suggesting implicit ensembles can rival explicit Bayesian approximations on calibration.","The advantage of SVE grows with the strength of the pretrained representation: with a strong self-supervised backbone it can beat a deep ensemble, which implies the method will keep improving as foundation models improve.","SVE keeps the frozen singular vectors intact, so it can be composed with other parameter-efficient adapters or distilled into a single network to cut inference FLOPs — directions the paper names as future work."],"fun_headline_variants":["Match deep ensembles with <1% extra parameters","Singular value ensembles: calibration at <1% overhead","Reweight frozen singular vectors for ensemble diversity","Implicit ensembles by rescaling singular values"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that joint training with small random perturbations and mini-batch sampling really makes members diverge to meaningfully different singular-value vectors; the paper offers no quantitative diversity metric, only qualitative trajectories, and if members collapse to similar vectors the ensemble degenerates to a single model with no epistemic uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Match deep ensembles with <1% extra parameters","Singular value ensembles: calibration at <1% overhead","Reweight frozen singular vectors for ensemble diversity","Implicit ensembles by rescaling singular values"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4027,"prompt_tokens":800,"completion_tokens":3227,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3168}},"tokens_in":544,"tokens_out":3227,"duration_ms":21476,"temperature":1.0,"reasoning_tokens":3168,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:42:31.686382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SVE with M=16 and measure average pairwise disagreement among members on test inputs (e.g., fraction of inputs with differing argmax, or variance of softmax probabilities), then compare with a 16-member deep ensemble. If SVE members disagree far less yet show similar ECE, the claimed diversity mechanism is not what drives the calibration. A second check: randomize the frozen singular vectors (replace U,V with random orthogonal matrices) and retrain — if calibration still holds, the knowledge-basis assumption is not load-bearing; if it fails, freezing meaningful directions is essential.","supporting_citations":[],"review_version":1}