{"id":"cf6e6db0-17cd-47b6-bda1-b00229961ef0","arxiv_id":"2412.08592","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A Gaussian graphical model with non-convex group sparsity selects a small set of singular components of a pretrained model to fine-tune, achieving competitive GLUE performance with 0.084M trainable parameters.","lead":"The paper proposes a new way to fine-tune large language models by selecting a small set of important parameters using a statistical model of their interactions. The method is tested on four language tasks and matches common baselines while training fewer parameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GGM sample covariance is never specified; with typical node counts exceeding the number of early training steps, the precision-matrix objective is ill-posed and the reported 0.084M selection is unverifiable.","rationale":"The reader's rejection was driven by an invalid code link, missing hyperparameters, partial GLUE coverage, and no error bars. I agree with the rejection, but I want to identify a sharper, more technical soft spot in the core algorithm itself. The paper's entire contribution is the GGM-based node selection, yet the sample covariance that feeds the GGM is specified only as 'samples during the initial stages' with no K and no node count. In the realistic high-dimensional regime (n far exceeding K), \\hatΣ is singular, log det Ω is not finite, and the BCD solver's initialization fails. Even in lower dimensions, the selected node set may be highly sensitive to K and to the unstated shrinkage/regularization, making the reported 0.084M parameter count an artifact of undocumented choices. A simple rank and stability check across K would settle whether this concern lands. Until that is tested, the central claim of competitive performance with dramatically fewer trainable parameters is unsupported. I therefore retain the reader's REJECT verdict; the concern reinforces it rather than moving it to a different outcome.","tokens_in":8061,"tokens_out":5447,"duration_ms":58636,"concrete_test":"Run Ours-2 on RoBERTa-base with all weight matrices included, fixing r as reported (once known). Compute the node count n and record the importance-score trajectories. For K = 10, 20, 50, 100, and 300 early steps, compute \\hatΣ, its rank, its condition number, and the final selected node set. Then rerun the full fine-tuning with the selected nodes for each K, applying diagonal shrinkage if \\hatΣ is singular. If the set of trainable nodes (and the resulting GLUE score) changes significantly across K, or if \\hatΣ is singular for any K used in the paper, the GGM-selection step is not a stable basis for the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the GGM step in Algorithm 1 (steps 3–5). Section 2.1 defines node values v^(k)(·) for each training step k and Section 2.2 builds a sample covariance \\hatΣ from these values, but the manuscript never states how many early steps K are used, how many nodes n enter the model, or whether any shrinkage is applied. For RoBERTa-base, selecting r principal components for each layer and each relevant weight matrix (query/key/value/output/intermediate) yields n on the order of hundreds to thousands, while a 'few early steps' would plausibly be K ≤ 100. In that regime \\hatΣ is rank-deficient. The objective (6) contains log det Ω, which is finite only for positive definite Ω; the BCD initialization Ω^(0)=diag(\\hatΣ)^{-1} in Algorithm 2 also requires strictly positive diagonal entries, which is not guaranteed since many importance scores are initially zero. Without a full-rank covariance estimate or an explicit regularization, the precision matrix Ω* is not well-defined, and the subsequent selection of nodes with large ∥Ω*_i∥_2 is an artifact of unspecified numerical choices. Thus the reported 0.084M trainable parameters and the 85.1 GLUE average are not reproducible from the paper as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a parameter-efficient fine-tuning (PEFT) method based on a Gaussian Graphical Model (GGM). The method reparameterizes each pre-trained weight matrix via SVD, treats the top-r principal components plus the bias as nodes, computes node importance values from the PLATON-style scores during early training, fits an ℓ2,g-regularized precision matrix to the sample covariance of these values, and then fine-tunes only the nodes selected by large sample means and large ℓ2 norms of precision-matrix columns. A block coordinate descent (BCD) algorithm is presented for the non-convex optimization problem. Experiments on four GLUE tasks with RoBERTa-Base report 0.084M trainable parameters and an average score of 85.1, compared with 84.9 for LoRA.","tokens_in":8376,"tokens_out":4305,"duration_ms":48890,"significance":"If the results are reproducible and the comparisons are statistically meaningful, the paper would make a genuinely novel contribution: it is, to my knowledge, the first application of GGM-based structural sparsity to PEFT, and the reported parameter count (0.084M) is substantially lower than that of LoRA and AdaLoRA. The idea of using a precision matrix to capture interactions among SVD components is plausible and worth pursuing. However, the contribution is currently conditional: the surrogate function g is never fixed, the covariance estimation and node-selection thresholds are unspecified, and the experimental section does not report variability or hyperparameter settings. The paper also provides no formal convergence or consistency guarantees for the proposed BCD scheme.","major_comments":[{"comment":"The surrogate function g is never specified for the proposed method. Section 2.3 defines the ℓ2,g regularization through an arbitrary non-negative, increasing, concave g, and Table 1 lists several examples, but the manuscript does not state which g is used in the experiments, nor the value of its internal hyperparameter (e.g., γ). Algorithm 3 explicitly depends on g'(x), and the selected node set depends on the resulting Ω*, so the reported 0.084M parameter selection is not reproducible without this information.","section":"Section 2.3 and Table 1 (second table)"},{"comment":"The sample covariance \\widehatΣ is not defined in terms of the number of early training steps K or the number of nodes n, and no shrinkage or regularization is described. With n on the order of hundreds to thousands and K plausibly small, \\widehatΣ is rank-deficient; then log det Ω in problem (6) is only finite for positive definite Ω, and the BCD initialization Ω^(0) = diag(\\widehatΣ)^{-1} in Algorithm 2 requires strictly positive diagonal entries, which is not guaranteed because many importance scores are initially zero. The authors should specify K, n, any shrinkage (e.g., adding εI), and a well-defined initialization.","section":"Section 2.2 and Algorithm 2"},{"comment":"The BCD algorithm's convergence is not analyzed. Problem (6) is non-convex, and the paper neither proves convergence of the BCD iterates to a stationary point nor states conditions under which such convergence holds. The statement that Δ converges to Ω as λ → +∞ concerns the reformulation, not the convergence of the iterates, and the claimed superlinear convergence in Algorithm 3 applies only to the scalar proximal subproblem. A convergence or stationarity guarantee is needed to support the claim that the optimization problem is 'efficiently solved.'","section":"Section 3"},{"comment":"The central empirical claim—competitive performance with 0.084M trainable parameters—is not supported with statistical evidence. The results are averaged over five seeds, but no standard deviations, confidence intervals, or significance tests are reported, so the 85.1 vs. 84.9 average difference from LoRA is within the range of typical seed noise. Additionally, the experimental configuration is missing: the rank r, the regularization parameters τ and λ, the momentum parameters β1 and β2, the number of initial training steps K, the threshold for selecting the 'important nodes' I, and the chosen surrogate g are all absent. Without these, Table 2 cannot be reproduced or meaningfully compared with prior work.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The definition of the 'important nodes' set I is incomplete: the manuscript says I is determined by selecting 'the nodes with the highest values in the sample mean,' but it does not specify how many nodes are selected or what threshold is used; this should be made explicit.","section":"Section 2.2"},{"comment":"There are two tables labeled 'Table 1': one is the symbol list and the other lists surrogate functions of the ℓ0 norm; renumber the tables to avoid confusion.","section":"Throughout"},{"comment":"Figure 2 is referred to as 'Figure 2.1' in the text; the citation should be to Figure 2.","section":"Section 2.1"},{"comment":"Step 3 says 'Collect samples during the initial stages of the training process,' but the number of samples K is never stated; this should be a concrete hyperparameter in the experiments.","section":"Algorithm 1"},{"comment":"The paper says 'GLUE benchmark' but evaluates only four of the eight GLUE tasks; the abstract and introduction should state this limitation explicitly.","section":"Section 4"},{"comment":"The code URL in the abstract contains a space ('Course projects.git') and appears not to be a standard repository path; a stable, accessible link should be provided.","section":"Abstract"},{"comment":"The ablation study in Table 3 reports results for only three tasks and a single seed; reporting the same error bars as in Table 2 would strengthen the comparison between Ours-1 and Ours-2.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be at an early stage: it states 'This work is still ongoing,' the code link is unusual, and several core algorithmic ingredients (the surrogate g, the number of early training steps, and the covariance shrinkage) are left unspecified. These are fixable in a revision, but as written the central claims are not verifiable. The authors should also be asked to clarify the relationship between this submission and the existing GAI algorithm of Zhang et al. (2023), since Algorithm 3 appears to be a direct reuse of that work without a self-contained derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a PEFT paper that grafts a Gaussian graphical model onto the SVD-principal-component scheme from PiSSA, using PLATON importance scores as node values. The genuinely new bit is the ℓ2,g nonconvex group-sparsity penalty on the precision matrix, which is meant to select nodes that interact strongly with a shortlist of \"important\" nodes. That is a reasonable extension, and the BCD derivation is formally laid out, though it relies on Algorithm 3 from the authors' earlier TPAMI paper rather than being self-contained.\n\nWhat the paper does well: the node definition is practical, the method honestly aims to reduce trainable parameters while keeping accuracy, and the reported numbers on four GLUE tasks (85.1 average with 0.084M parameters vs LoRA's 84.9 with 0.3M) are the kind of result worth checking. The ablation Ours-1 vs Ours-2 shows the \"important nodes\" term does something.\n\nThe soft spots are large. The stress-test worry about the GGM step is legitimate. Nowhere does the paper say how many early training steps K are used to form the sample covariance, how many nodes n enter the model, or whether any shrinkage is applied to \\hatΣ. For the objective (6), log det Ω is only finite for positive definite Ω, and the BCD initialization Ω⁰=diag(\\hatΣ)^{-1} requires strictly positive diagonal entries. If n > K or any node's importance is identically zero in the sampled steps, the problem is ill-posed and the selected 0.084M is an artifact of unspecified numerical choices. I cannot tell from the paper whether this actually happens; the number of nodes depends on r, which is never reported. That is precisely the issue: the central mechanism is unverifiable from the manuscript.\n\nOn top of that, the surrogate g is never fixed (several examples are listed, none chosen), the threshold defining the \"highest values\" in the sample mean is unspecified, no learning-rate/λ/τ values are given, no standard deviations or significance tests appear despite five seeds, only four of eight GLUE tasks are used, and the code link is a broken GitHub URL with a space in it. The novelty overlap with PiSSA is acknowledged only by citation, not discussed.\n\nWho is this for? People working on PEFT who care about structured parameter selection. It is not ready as a research preprint. I would want a serious referee to look at it, because the underlying idea has legs and the flaws are mostly missing details rather than a demonstrated fatal error, but as it stands the empirical and statistical claims are unsupported. My recommendation: send it to peer review if the venue can demand a major revision; do not accept in this form.","headline":"A GGM-based PEFT selection method with a plausible core and enough missing specification to make the central result unverifiable; deserves a demanding major revision, not rejection without review.","tokens_in":8853,"tokens_out":3753,"would_cite":false,"duration_ms":39927,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Gaussian Graphical Model can decide which SVD components stay trainable, matching LoRA on GLUE with 0.084M parameters.","keywords":["Gaussian Graphical Models","Parameter-Efficient Fine-Tuning","ℓ2,g regularization","SVD principal components","block coordinate descent","GLUE benchmark","RoBERTa","structural sparsity"],"falsifier":"Re-run Ours-2 on the same four GLUE tasks with the same 0.084M budget, but draw the trainable SVD components uniformly at random instead of from the GGM solve, and compare the average score. If random selection matches or exceeds 85.1, the interaction model is not what carries the result.","tokens_in":7853,"feed_emoji":"🧠","tokens_out":8320,"duration_ms":81766,"temperature":0.7,"pith_summary":"The paper proposes a parameter-efficient fine-tuning method that lets a Gaussian Graphical Model decide which principal components of each layer's weights stay trainable. It claims this is the first application of Gaussian Graphical Models to parameter-efficient fine-tuning, and reports that fine-tuning RoBERTa-base this way reaches an average score of 85.1 on four GLUE tasks with only 0.084M trainable parameters, beating LoRA's 84.9 while using less than a third of LoRA's trainable budget. The model's nodes are the SVD principal components of each layer plus its bias; their values are importance scores derived from how much each weight moves the loss. A precision matrix is then estimated under a non-convex $\\ell_{2,g}$ penalty that selects whole nodes rather than individual edges, and a block coordinate descent algorithm solves the resulting problem. If the central claim holds, modeling parameter interactions before freezing weights is a viable alternative to fixed-rank or importance-score-only allocation of the fine-tuning budget.","feed_headline":"Graphical model finds 0.084M trainable parameters that beat LoRA on GLUE","feed_subtitle":"RoBERTa-base tuned through GGM-selected SVD components scores 85.1 on four GLUE tasks, above LoRA's 84.9.","key_machinery":"The load-bearing object is the $\\ell_{2,g}$-regularized Gaussian Graphical Model precision matrix $\\Omega$. Each row of $\\Omega$ encodes how one trainable node interacts with the other nodes, and the regularization $\\|\\bar{\\Omega}\\|_{2,g}$ applies a non-convex concave surrogate $g$ to the $\\ell_2$ norms of the rows of a restricted submatrix, so the penalty drives entire nodes out of the active set rather than merely thinning individual edges. The optimization is solved by block coordinate descent: one step updates $\\Omega$ by gradient descent on a concave $\\log\\det$ objective with a quadratic proximal penalty, and the other step updates an auxiliary variable $\\Delta$ using the non-convex proximal map $T_g(y,\\tau/2\\lambda)$, computed by the GAI fixed-point iteration. The selection rule that comes out of the solve — keep the nodes in $I$ plus the nodes with large row norm, freeze the rest — is what gives the parameter-efficiency result.","core_discovery":"The paper's central claim is that parameter interactions, not just individual importance scores, should decide which weights are trainable, and that a Gaussian Graphical Model is the right way to capture those interactions. For each layer, the method takes the top $r$ SVD principal components $(A_i,B_i)$ plus the bias as graph nodes, and assigns each node a value by averaging the importance score $s^{(k)}$ over its entries. A sample covariance matrix of these node values is formed from early training steps, and the method solves maximize over positive-semidefinite $\\Omega$ of $\\log\\det\\Omega - \\langle\\hat{\\Sigma},\\Omega\\rangle - \\tau\\|\\bar{\\Omega}\\|_{2,g}$, where the last term penalizes, for each node outside the set of 'important nodes' $I$, a concave surrogate $g$ of the row norm $\\|\\bar{\\Omega}_i\\|_2$. After solving, it continues training the important nodes and the nodes whose $\\|\\Omega_i\\|_2$ is large, freezing everything else. On SST-2, MRPC, CoLA, and STS-B, the full version ('Ours-2') reports 0.084M trainable parameters and an average score of 85.1, compared with 84.9 for LoRA and 85.0 for full fine-tuning.","pith_inferences":["The paper leaves the stability of its early-training covariance estimate untested; a practical check would be to refit the GGM after 10%, 25%, and 50% of training and compare the selected node sets.","A random-node control is the sharpest missing experiment: with the same 0.084M budget, random SVD components should be kept trainable to isolate whether the GGM's interaction structure, rather than the low-rank SVD parameterization, drives the 85.1 average.","The node definition is not BERT-specific, so the same GGM selection should apply to vision or multimodal transformers whose weight matrices also show low-rank spectra; the paper only demonstrates RoBERTa-base.","The method estimates the precision matrix once from early samples; a periodic update of $\\Omega$ during training would test whether parameter interactions drift, at the cost of extra computation."],"forward_implications":["If the GGM selection is correct, a fine-tuning budget of 0.084M parameters can match or beat LoRA's 0.3M on the four tested GLUE tasks, so interaction structure is a usable signal for allocating rank across layers.","The method offers a new PEFT baseline: any future low-rank adapter can be compared against a budget allocated by precision-matrix row norms instead of fixed ranks or local importance scores.","Fewer trainable parameters means proportionally smaller optimizer state and stored gradients during fine-tuning, which is the practical memory motivation for PEFT that the paper cites.","The gap between Ours-1 and Ours-2 in the ablation suggests that adding the 'important nodes' set $I$ contributes to the result, so the sample-mean node values carry information beyond the covariance structure."],"supporting_citations":[{"why":"Supplies the Gaussian Graphical Model / graphical lasso formulation that the paper reworks with node-level structural sparsity.","marker":"[9]"},{"why":"Provides the importance-score definition $s^{(k)}$ used to assign values to the SVD-component nodes.","marker":"[26]"},{"why":"Defines AdaLoRA, the adaptive-rank baseline whose local importance scoring the paper argues misses global parameter dependencies.","marker":"[27]"},{"why":"Defines LoRA, the fixed-rank baseline that the method is compared against and whose rank allocation it aims to improve.","marker":"[16]"},{"why":"Defines the GLUE benchmark and the four datasets used in the main experiment table.","marker":"[24]"},{"why":"Defines RoBERTa-Base, the model all fine-tuning experiments are run on.","marker":"[18]"},{"why":"Supplies the GAI fixed-point algorithm used to solve the nonconvex proximal step in the BCD updates.","marker":"[28]"},{"why":"Provides PiSSA, a principal-component adaptation baseline included in the comparison.","marker":"[21]"}],"fun_headline_variants":["First GGM-based PEFT: 0.084M params, beats LoRA on GLUE","Graphical model tunes 0.084M params, edges LoRA on GLUE","0.084M trainable params: GGM PEFT beats LoRA, hits 85.1","GGM-PEFT: 0.084M params for 85.1 on GLUE, beats LoRA","Graphical model PEFT cuts params to 0.084M, tops LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that importance scores from the first few training steps give a sample covariance matrix that accurately represents parameter interactions for the entire fine-tuning run; the paper never reports how many samples or steps were used, nor a stability check.","fun_headline_variants_meta":{"raw":{"variants":["First GGM-based PEFT: 0.084M params, beats LoRA on GLUE","Graphical model tunes 0.084M params, edges LoRA on GLUE","0.084M trainable params: GGM PEFT beats LoRA, hits 85.1","GGM-PEFT: 0.084M params for 85.1 on GLUE, beats LoRA","Graphical model PEFT cuts params to 0.084M, tops LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3816,"prompt_tokens":971,"completion_tokens":2845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2721}},"tokens_in":587,"tokens_out":2845,"duration_ms":19282,"temperature":1.0,"reasoning_tokens":2721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:20.554753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Ours-2 on the same four GLUE tasks with the same 0.084M budget, but draw the trainable SVD components uniformly at random instead of from the GGM solve, and compare the average score. If random selection matches or exceeds 85.1, the interaction model is not what carries the result.","supporting_citations":[{"cited_title":"Sparse inverse covariance estimation with the graphical lasso","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian Graphical Model / graphical lasso formulation that the paper reworks with node-level structural sparsity."},{"cited_title":"Platon: Pruning large transformer models with upper con- fidence bound of weight importance","cited_arxiv_id":null,"evidence_quote":"Provides the importance-score definition $s^{(k)}$ used to assign values to the SVD-component nodes."},{"cited_title":"Roberta: a robustly optimized bert pretraining approach, 2019","cited_arxiv_id":null,"evidence_quote":"Defines RoBERTa-Base, the model all fine-tuning experiments are run on."},{"cited_title":"Structured sparsity optimization with non-convex surrogates of ℓ2,0- norm: A unified algorithmic framework","cited_arxiv_id":null,"evidence_quote":"Supplies the GAI fixed-point algorithm used to solve the nonconvex proximal step in the BCD updates."}],"review_version":1}