pith. sign in

arxiv: 2605.23102 · v1 · pith:R455UFM2new · submitted 2026-05-21 · 📊 stat.ML · cs.LG· stat.ME

LLM Sparsity Prior for Robust Feature Selection

Caleb Skinner , Yihan Guo , Meng Li This is my paper

Pith reviewed 2026-05-25 04:55 UTC · model grok-4.3

classification 📊 stat.ML cs.LGstat.ME
keywords feature selectionspike and slabLLM-informed priorsvariable selectionrobust Bayesian inferenceacute kidney injury predictionhigh-dimensional data
0
0 comments X

The pith

The LLM Sparsity Prior adds hierarchical hyperpriors to spike-and-slab models so they can automatically downweight inaccurate LLM-generated feature weights.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models can supply domain knowledge as prior probabilities for which variables matter in high-dimensional selection problems, but methods that use those probabilities directly lose accuracy when the LLM outputs are off. The paper introduces the LLM Sparsity Prior, which folds LLM weights into the inclusion probabilities of spike-and-slab and spike-and-slab lasso models through two hyperparameters that control overall sparsity level and how tightly the model concentrates on the LLM suggestions. Hierarchical hyperpriors are then placed on those two hyperparameters, letting the data decide how much to trust the LLM information on any given problem. This construction is tested on a private medical dataset for predicting acute kidney injury, where it improves accuracy, surfaces clinically relevant variables missed by baselines, and stays stable across prompt variations and small sample sizes.

Core claim

The LLM Sparsity Prior integrates LLM-generated weights into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models via two interpretable hyperparameters governing global sparsity and weight concentration. Hierarchical hyperpriors on these parameters allow the model to dynamically discount uninformative or misleading weights, improving robustness without sacrificing gains when weights are accurate.

What carries the argument

Two hyperparameters (global sparsity and weight concentration) placed on the LLM-informed inclusion probabilities, together with hierarchical hyperpriors that adaptively control their values.

If this is right

  • LSP maintains prediction accuracy across varying LLM weight quality caused by prompt changes.
  • It recovers clinically relevant features in the acute kidney injury dataset that standard methods miss.
  • Gains are largest in low-data regimes where external prior information is most valuable.
  • The same construction applies to both ordinary spike-and-slab and spike-and-slab lasso formulations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hyperprior structure could be attached to other Bayesian models that ingest external priors, such as Gaussian processes or survival models.
  • In medical applications the approach may reduce the labeled data needed to reach usable performance by letting the model learn when to trust LLM knowledge.
  • If the discounting mechanism works, analogous adaptive weighting could stabilize LLM use in other statistical tasks that currently require careful prompt engineering.

Load-bearing premise

The hierarchical hyperpriors on global sparsity and weight concentration are sufficient to dynamically discount uninformative or misleading LLM-generated weights without manual tuning or post-hoc adjustments.

What would settle it

Run the method on data where LLM weights are replaced by random or deliberately reversed values and check whether prediction accuracy falls below that of a standard spike-and-slab model with no LLM input; if it does, the robustness claim does not hold.

Figures

Figures reproduced from arXiv: 2605.23102 by Caleb Skinner, Meng Li, Yihan Guo.

Figure 1
Figure 1. Figure 1: For both n = 100 and n = 250, LSP is robust, outperforming the respective baseline at all weights and dramatically improving with quality weights. Conversely, LLM-Lasso underperforms the baseline with lower-quality weights. All three LLM-informed methods improve substantially as the weight quality increases. For example, in the high-dimensional regime, the Spike-and-Slab yields an ℓ1 error of 29.78, and th… view at source ↗
Figure 2
Figure 2. Figure 2: MSE over five prompting strategies. Dotted line is the associated baseline. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Structure Recovery of LSP Methods across selected [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
read the original abstract

Large language models (LLMs) offer a scalable mechanism to elicit domain-informed prior information for high-dimensional variable selection. However, existing methods such as LLM-Lasso are sensitive to weight quality, with performance degrading substantially when LLM-generated weights are inaccurate. To address this challenge, we first introduce a framework for quantifying the quality of LLM-generated weights, enabling rigorous evaluation of LLM-informed methods across varying weight regimes. We then propose the LLM Sparsity Prior (LSP), which integrates LLM-generated weights into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models via two interpretable hyperparameters governing global sparsity and weight concentration. Hierarchical hyperpriors on these parameters allow the model to dynamically discount uninformative or misleading weights, improving robustness without sacrificing gains when weights are accurate. Finally, we develop principled prompt engineering strategies and validate the method on a private medical dataset studying Acute Kidney Injury. LSP improves prediction accuracy and identifies clinically relevant features missed by the baselines, with robustness to prompt variation and particular effectiveness in low-data regimes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes the LLM Sparsity Prior (LSP) to incorporate LLM-generated weights into the inclusion probabilities of spike-and-slab and spike-and-slab lasso models via two hyperparameters (global sparsity and weight concentration) equipped with hierarchical hyperpriors; these hyperpriors are claimed to dynamically discount inaccurate LLM weights. The authors also introduce a framework for quantifying LLM weight quality, develop prompt engineering strategies, and report improved prediction accuracy and clinically relevant feature identification on a private medical dataset for Acute Kidney Injury, with robustness to prompt variation and gains in low-data regimes.

Significance. If the hierarchical mechanism reliably marginalizes over poor LLM weights, the work would offer a practical advance in Bayesian variable selection by reducing sensitivity to LLM prior quality without manual tuning. The quality-quantification framework could also serve as a reusable tool for evaluating LLM-informed methods.

major comments (2)
  1. [§3.2] §3.2 (hierarchical model definition): The construction places hyperpriors on the two new free parameters (global sparsity and weight concentration), but no derivation or targeted simulation shows that the posterior over these parameters down-weights systematically biased (as opposed to merely noisy) LLM inclusion probabilities when the likelihood is weak; this is the load-bearing step for the low-data robustness claim.
  2. [Experimental results section] Experimental results section: The reported accuracy improvements and feature-selection gains on the AKI dataset lack ablations that isolate the hierarchical hyperpriors (e.g., LSP with fixed versus hierarchical hyperparameters), so it is unclear whether the robustness is attributable to the proposed mechanism or to other modeling choices.
minor comments (2)
  1. The abstract states numerical improvements without error bars or significance tests; the main text should include these for all reported metrics.
  2. Notation for the two hyperparameters should be introduced once with explicit symbols and kept consistent across equations and text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the two major comments point by point below and will revise the manuscript to include the requested analyses.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (hierarchical model definition): The construction places hyperpriors on the two new free parameters (global sparsity and weight concentration), but no derivation or targeted simulation shows that the posterior over these parameters down-weights systematically biased (as opposed to merely noisy) LLM inclusion probabilities when the likelihood is weak; this is the load-bearing step for the low-data robustness claim.

    Authors: We acknowledge that the manuscript does not contain a dedicated derivation or simulation isolating the posterior behavior specifically for systematically biased LLM weights under weak likelihood. The hierarchical hyperpriors are constructed to permit the data to modulate the effective weight concentration and global sparsity, but we agree that explicit demonstration of down-weighting in biased cases would strengthen the low-data robustness claim. We will add a targeted simulation study to §3.2 in the revision. revision: yes

  2. Referee: [Experimental results section] Experimental results section: The reported accuracy improvements and feature-selection gains on the AKI dataset lack ablations that isolate the hierarchical hyperpriors (e.g., LSP with fixed versus hierarchical hyperparameters), so it is unclear whether the robustness is attributable to the proposed mechanism or to other modeling choices.

    Authors: We agree that the current experiments do not isolate the contribution of the hierarchical hyperpriors. We will add an ablation comparing the full hierarchical LSP against versions with fixed hyperparameters (set to LLM-informed values without hyperpriors) in the experimental results section of the revision. revision: yes

Circularity Check

0 steps flagged

No circularity: LSP hyperparameters and hyperpriors are independent modeling choices, not reductions of fitted quantities.

full rationale

The paper introduces two new hyperparameters (global sparsity and weight concentration) with hierarchical hyperpriors placed on them to allow dynamic discounting of LLM weights. This construction is presented as an extension to spike-and-slab models rather than any quantity being defined in terms of itself or a fitted parameter being relabeled as a prediction. No self-citation chains, uniqueness theorems from prior author work, or ansatzes smuggled via citation appear in the abstract or description. The inclusion probabilities are not shown to reduce by construction to quantities already present in the data or LLM weights; the hyperpriors are an additional layer whose behavior is claimed to be learned from the likelihood. The derivation chain is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The model rests on standard spike-and-slab assumptions plus two new scalar hyperparameters whose hyperpriors are asserted to handle weight quality automatically; no invented physical entities.

free parameters (2)
  • global sparsity hyperparameter
    Controls the baseline prior inclusion probability across all features; its value is governed by a hyperprior rather than fixed by hand.
  • weight concentration hyperparameter
    Controls how sharply the prior trusts the LLM-provided weights; again governed by a hyperprior.
axioms (1)
  • domain assumption Spike-and-slab prior structure remains valid when inclusion probabilities are modulated by external LLM weights.
    Invoked when the authors state that LLM weights are integrated into the prior inclusion probabilities of Spike-and-Slab and Spike-and-Slab Lasso models.

pith-pipeline@v0.9.0 · 5709 in / 1375 out tokens · 20179 ms · 2026-05-25T04:55:03.987197+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages

  1. [1]

    and George, Edward I

    Chipman, Hugh A. and George, Edward I. and McCulloch, Robert E. , journal =. 2010 , doi =

    work page 2010

  2. [2]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Ab Initio Nonparametric Variable Selection for Scalable Symbolic Regression with Large p , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    work page 2025

  3. [3]

    arXiv preprint arXiv:2509.07121 , year =

    Posterior Summarization for Variable Selection in Bayesian Tree Ensembles , author =. arXiv preprint arXiv:2509.07121 , year =

    work page arXiv

  4. [4]

    Biometrics , volume =

    Bayesian covariate-dependent graph learning with a dual group spike-and-slab prior , author =. Biometrics , volume =

    work page

  5. [5]

    Rockova, Veronika and George, Edward , journal=. The

    work page

  6. [6]

    Tadesse, Mahlet and Vannucci, Marina , year=

    work page

  7. [7]

    Journal of the American Statistical Association , volume =

    Polson, Nicholas and Scott, James and Windle, Jesse , year =. Journal of the American Statistical Association , volume =

    work page

  8. [8]

    Zhang, Erica and Goto, Ryunosuke and Sagan, Naomi and Mutter, Jurik and Phillips, Nick and Alizadeh, Ash and Lee, Kangwook and Blanchet, Jose and Pilanci, Mert and Tibshirani, Robert , year =

    work page

  9. [9]

    George, E. I. and McCulloch, R. E. , year =. Variable selection via. Journal of the American Statistical Association , volume=

    work page

  10. [10]

    Journal of the American Statistical Association , volume =

    Mitchell, Toby and Beauchamp, John , year =. Journal of the American Statistical Association , volume =

    work page

  11. [11]

    The Annals of Statistics , volume =

    Rockova, Veronika , year =. The Annals of Statistics , volume =

    work page

  12. [12]

    1995 , journal =

    Bayesian graphical models for discrete data , author =. 1995 , journal =

    work page 1995

  13. [13]

    2015 , journal =

    Slope-adaptive variable selection via convex optimization , author =. 2015 , journal =

    work page 2015

  14. [14]

    False discoveries occur early on the

    Su, Weijie and Bogdan, Malgorzata and Candes, Emmanuel , year =. False discoveries occur early on the. The Annals of Statistics , volume =

    work page

  15. [15]

    2025 , archivePrefix=

    Large language models for statistical inference: Context augmentation with applications to the two-sample problem and regression , author =. 2025 , archivePrefix=. 2506.23862 , howpublished =

    work page arXiv 2025

  16. [16]

    Advances in Neural Information Processing Systems , editor =

    Language models are few-shot learners , author=. Advances in Neural Information Processing Systems , editor =

    work page

  17. [17]

    2019 , howpublished =

    Language models are unsupervised multitask learners , author =. 2019 , howpublished =

    work page 2019

  18. [18]

    Zico , booktitle =

    Manikandan, Hariharan and Jiang, Yiding and Kolter, J. Zico , booktitle =. Language models are weak learners , year=

    work page

  19. [19]

    2023 , booktitle =

    Chain-of-thought prompting elicits reasoning in large language models , author=. 2023 , booktitle =

    work page 2023

  20. [20]

    Advances in Neural Information Processing Systems , editor=

    Solving quantitative reasoning problems with language models , author=. Advances in Neural Information Processing Systems , editor=. 2022 , pages =

    work page 2022

  21. [21]

    Challenging

    Suzgun, Mirac and Scales, Nathan and Sch. Challenging. Findings of the Association for Computational Linguistics , year =. doi:10.18653/v1/2023.findings-acl.824 , pages =

    work page doi:10.18653/v1/2023.findings-acl.824 2023

  22. [22]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , publisher =

    Language models as knowledge bases? , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , publisher =. 2019 , pages =

    work page 2019

  23. [23]

    Nature Methods , year =

    Transfer learning enables predictions in network biology , author =. Nature Methods , year =

    work page

  24. [24]

    Cui, Haoyang and Wang, Chen and Maan, Haroon and Wang, Bin , journal =. sc. 2024 , number=

    work page 2024

  25. [25]

    2022 , pages =

    Dinh, Tuan and Zeng, Yuchen and Zhang, Ruisu and Lin, Ziqian and Gira, Michael and Rajput, Shashank and Sohn, Jy-yong and Papailiopoulos, Dimitris and Lee, Kangwook , booktitle=. 2022 , pages =

    work page 2022

  26. [26]

    Kristy Choi and Chris Cundy and Sanjari Srivastava and Stefano Ermon , year=

    work page

  27. [27]

    Theory of

    Bruno de Finetti , publisher =. Theory of

    work page

  28. [28]

    The Annals of Statistics , volume =

    The formal definition of reference priors , author =. The Annals of Statistics , volume =

    work page

  29. [29]

    Proceedings of the Royal Society of London

    An invariant form for the prior probability in estimation problems , author =. Proceedings of the Royal Society of London. Series A , volume =

    work page

  30. [30]

    Jeong and Zachary C

    Daniel P. Jeong and Zachary C. Lipton and Pradeep Ravikumar , year=

    work page

  31. [31]

    2402.18609 , archivePrefix=

    Yang, Tianze and Yang, Tianyi and Lyu, Fuyuan and Liu, Shaoshan and Liu, Xue , year=. 2402.18609 , archivePrefix=

    work page arXiv

  32. [32]

    1945 , journal =

    Individual comparisons by ranking methods , author =. 1945 , journal =

    work page 1945

  33. [33]

    and Polson, Nicholas G

    Carvalho, Carlos M. and Polson, Nicholas G. and Scott, James G. , year =. The. Biometrika , volume =

    work page

  34. [34]

    , year =

    Barbieri, Maria Maddalena and Berger, James O. , year =. Optimal predictive model selection , journal =

    work page

  35. [35]

    and Madigan, David and Raftery, Adrian E

    Hoeting, Jennifer A. and Madigan, David and Raftery, Adrian E. and Volinsky, Chris T. , year =. Statistical Science , volume =

    work page

  36. [36]

    Ryan, C. T. and Zeng, Z. and Chatterjee, S. and Wall, M. J. and Moon, M. R. and Coselli, J. S. and Rosengart, T. K. and Li, M. and Ghanta, R. K. , year =. Machine learning for dynamic and early prediction of. The Journal of Thoracic and Cardiovascular Surgery , volume =

    work page

  37. [37]

    2025 , author =

    Introducing. 2025 , author =

    work page 2025

  38. [38]

    2015 , author =

    A meta-analysis of the association of estimated. 2015 , author =

    work page 2015

  39. [39]

    2024 , author =

    Urine output is an early and strong predictor of. 2024 , author =

    work page 2024

  40. [40]

    2024 , author =

    Blood transfusion reactions and risk of. 2024 , author =

    work page 2024

  41. [41]

    2020 , author =

    Decrease in. 2020 , author =

    work page 2020

  42. [42]

    Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers) , year =

    Yuchi, Fengting and Du, Li and Eisner, Jason , editor =. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers) , year =

    work page

  43. [43]

    and Radcliffe, Evan and Rajagopal, Guru R

    Atil, Berk and Aykent, Sarp and Chittams, Alexa and Fu, Lisheng and Passonneau, Rebecca J. and Radcliffe, Evan and Rajagopal, Guru R. and Sloan, Adam and Tudrej, Tomasz and Ture, Ferhan and Wu, Zhe and Xu, Lixinyu and Baldwin, Breck , year =. Non-determinism of deterministic. Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems , editor =

    work page

  44. [44]

    , booktitle=

    Bai, Ray and Rockova, Veronika and George, Edward I. , booktitle=

    work page

  45. [45]

    Linero, Antonio and Yang, Yun , journal=

    work page

  46. [46]

    Sunil , journal=

    Ishwaran, Hemant and Rao, J. Sunil , journal=

    work page

  47. [47]

    Proceedings of the 38th International Conference on Machine Learning , volume =

    Marginal contribution feature importance - an axiomatic approach for explaining data , author=. Proceedings of the 38th International Conference on Machine Learning , volume =. 2021 , editor =

    work page 2021

  48. [48]

    Heuss, Maria and de Rijke, Maarten and Anand, Avishek , year =. Ranking. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information , pages =

    work page