REVIEW 3 major objections 6 minor 27 references
Stochastic Encodings for Active Feature Acquisition
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that latent-space gradient scores of a stochastically encoded predictor outperform RL and CMI-based active feature acquisition baselines on the evaluated tasks.
desk verdict Solid empirical AFA paper with a real gap between the claimed non-greedy mechanism and what the acquisition objective actually computes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the acquisition score of Eqs. (2)--(3): a class-probability-weighted expectation, over samples from the stochastic latent encoder, of the normalized squared Euclidean norm of the predictor's class-gradient in the latent block belonging to feature $i$. It is made tractable by factorizing the encoder into independent per-feature normal distributions (each feature gets $l$ latent components), by regularizing the latent space with a variational information-bottleneck KL term so gradients reflect label-relevant structure, and by taking many latent samples during scoring so unobserved feature values are implicitly considered.
What would settle it
Run SEFA on the paper's indicator problem with $d=4$ and record the first acquisition: if the indicator feature is not acquired first in a large fraction of instances, the stochastic expectation is not capturing long-term value as claimed. A complementary check is a synthetic task where a feature has a large latent gradient norm but no effect on prediction accuracy under any unobserved combination; SEFA should not systematically select it.
Extended reading notes
Core claim
SEFA trains a per-feature stochastic encoder $p_\theta(z|x_S)$ with a variational information-bottleneck term, then scores each candidate feature $i$ by $R(x_O,i)=\sum_{c} p_{\theta,\phi}(Y=c|x_O)\,\mathbb{E}_{z\sim p_\theta(z|x_O)}\, r(c,z,i)$, where $r(c,z,i)$ is the squared Euclidean norm of $\nabla_z p_\phi(Y=c|z)$ restricted to feature $i$'s latent block, normalized over all blocks. Because the encoders are stochastic, the expectation samples many possible realizations of unobserved features; because the latent blocks are per-feature, gradients can be attributed to individual features; because the sum is weighted by current class probabilities, acquisitions focus on disambiguating likely classes rather than merely lowering entropy. The paper's thesis is that this score is a faithful, tractable proxy for the long-term predictive value of measuring a feature, and that this is why SEFA outperforms both RL and CMI-based baselines on the evaluated tasks.
Load-bearing premise
The whole method rests on the assumption that the normalized Euclidean norm of the predictor's latent gradient for a feature, averaged over stochastic samples and weighted by current class probabilities, ranks features by how much measuring them will improve future predictions.
Editorial extensions
If this is right
- If the gradient-norm score is a faithful proxy, then one can build competitive AFA systems without RL training loops or generative CMI estimators, avoiding their known instabilities.
- The paper's Proposition 4.2 implies that expectation over unobserved feature values — which SEFA approximates stochastically — is sufficient to make greedy acquisition optimal on the indicator problem, so the non-greedy behaviour is not incidental.
- The ablation results imply each design choice (stochastic encoder, multiple latent samples, latent-space scoring, and probability weighting in multi-class settings) contributes to the measured gains, so simplified variants of SEFA should not match its full performance.
- On multi-class tasks, probability weighting is the component that shifts acquisitions toward distinguishing likely classes; on binary tasks the paper proves the weighting cancels out, so any advantage there comes from the stochastic latent scoring itself.
Reading between the lines
- An untested consequence is that SEFA's score could be read as a gradient-based proxy for expected loss reduction; a direct comparison against one-step lookahead on expected 0-1 loss would clarify when the proxy over- or under-orders features.
- Because the encoder factorizes per feature, correlations between features are only exploited through the predictor's gradients, so SEFA's margin over baselines on datasets with strongly correlated features is a natural stress test the paper does not run.
- The paper's limitation that SEFA is classification-only suggests a discretized auxiliary classification head could extend the same acquisition score to regression; this is the authors' own suggested direction and can be tested without changing the scoring rule.
- The gradient-norm heuristic may favour features with large but redundant latent gradients; measuring how often SEFA's first acquisition on the indicator task is the indicator itself would isolate the stochastic expectation's contribution from the heuristic's bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SEFA, a latent-variable method for active feature acquisition. The model trains per-feature stochastic encoders and a predictor with a negative log-likelihood loss plus an information-bottleneck regularization, then acquires features by scoring normalized latent-gradient norms of the class probability, averaged over latent samples and weighted by current class probabilities. The authors provide a theoretical critique of greedy conditional mutual information (CMI) maximization, including an indicator-feature example where CMI is provably suboptimal, and report extensive experiments on synthetic, tabular, image, and medical datasets claiming consistent improvements over RL-based and CMI-based baselines.
Significance. If the empirical claims hold, SEFA is a practically useful contribution: it avoids RL training instabilities and generative-model CMI estimation, and it ships with reproducible code, five-seed experiments, standard errors, and detailed ablations. The CMI critique in Section 4 is clear and the indicator example is instructive. However, the paper's central conceptual claim---that SEFA is non-greedy because it reasons over possible unobserved realizations---is not supported by the acquisition objective as written. The theoretical results concern a different, intractable objective, and the link between that theory and Eq. (2) is missing. The empirical wins are therefore best interpreted as evidence for a strong heuristic, not as validation of the stated mechanism.
major comments (3)
- [Section 5.2, Eqs. (2)-(3)] The acquisition score averages r(c,z,i) over z ~ p_theta(z|x_O). For an unobserved feature i, z_i is drawn from the encoder conditioned on the imputed value and mask 0, not from a distribution over plausible true values of x_i and not from p(x_U|x_O). The training loss in Eq. (1) does not calibrate this imputation-conditional encoder to the post-measurement conditional distribution, so the claim that 'multiple possible latent realizations (including those associated with different unobserved feature values)' are considered is not implemented by the objective. The stochasticity is encoder noise under a fixed imputation, not a posterior over unobserved features. This undermines the 'non-greedy by design' claim and the Monte Carlo tree search analogy. The authors should either provide a formal derivation of what Eq. (2) optimizes or explicitly reposition SEFA as a heuristic whose non-greedy behavior is an empirical hypothesis rather than a proven property.
- [Section 4, Propositions 4.1 and 4.2] The paper motivates SEFA with the proof that greedy CMI fails on the indicator problem and that maximizing E_{p(x_U|x_O)} I(X_i; Y | x_O, x_U) is optimal. However, the text states that this objective is not used because it is intractable, and SEFA's Eq. (2) does not implement this or any other integral over p(x_U|x_O). The theoretical results therefore do not provide support for SEFA's acquisition mechanism; they only establish that some non-myopic objectives exist. The gap between the theoretical motivation and the actual acquisition objective should be stated explicitly, and the paper should not imply that Propositions 4.1-4.2 validate the heuristic in Eq. (2).
- [Section 5.3 and Table 3] The claim that calculating the acquisition objective in latent space is superior because latent gradients are 'more meaningful and comparable' is not cleanly tested by the 'Feature Space Calculation' ablation. That ablation uses a separately trained VAE and MLP, which introduces a different imputation model and different predictor capacity, so the comparison conflates the representation space with the choice of generative model. An ablation that computes the same gradient-norm objective on features transformed to a comparable continuous scale (e.g., copula-transformed inputs) without a separate generative model would better isolate the effect of the latent space. Without this, the latent-space benefit remains plausible but not isolated.
minor comments (6)
- [Abstract and Section 5.2] The phrase 'reasoning about the features across many possible unobserved realizations in a stochastic latent space' overstates what Eq. (2) computes; consider rewording to describe sampling from the current encoder distribution.
- [Section 6.2, Table 2] On Cube the margin over DIME and Opportunistic RL is 0.003, and on MiniBooNE the margin over Fixed MLP is 0.003; the text should quantify these small margins rather than only emphasizing larger differences.
- [Appendix K] Hyperparameter configurations are selected by validation acquisition-curve area with three repeats, but main results use five seeds; please report whether the selected configurations were stable across the repeats.
- [Eq. (1)] The notation x_{S∩S'} is unclear because S is not defined as the full feature set; please clarify how the subsampling process interacts with the observed set during training.
- [Figure 3] The zoomed-in insets are too small to read; enlarge them or tabulate early-acquisition differences.
- [Section 6.3] The gene-cancer association citations are post hoc and should be described as illustrative rather than as validation of acquisition optimality.
Circularity Check
No significant circularity: SEFA's acquisition objective is an explicitly hand-crafted heuristic, the optimality proposition is disclaimed as not used, and the empirical claims are measured against external benchmarks.
full rationale
The main claimed results are empirical: SEFA outperforms baselines on synthetic and real datasets (Tables 1-2). The acquisition score R(x_O,i) in Eq. (2)-(3) is defined directly as a probability-weighted expected normalized latent-gradient norm; it is not fitted to any target acquisition outcome, so there is no fitted-input-called-prediction step. The paper explicitly states 'Note we do not use this as our acquisition objective' for the theoretically justified objective E_{p(x_U|x_O)} I(X_i;Y|x_O,x_U) in Prop. 4.2, so the theoretical optimality result is not claimed to be implemented by Eq. (2). The information-bottleneck term is a standard variational upper bound, not a derivation of the acquisition objective. Training uses NLL plus DKL regularization and hyperparameters tuned on validation area under the acquisition curve, which is standard and independent of test performance. The strongest conceptual caveat is that Eq. (2) averages gradients over p_theta(z|x_O), which for missing features is an imputation-conditioned stochastic encoding rather than a posterior over true unobserved feature values; the paper's 'many possible unobserved realizations' language is an analogy, not a theorem. That is an unsupported proxy assumption, but it does not make the result equivalent to its inputs by construction. Self-citations (e.g., Yoon et al. 2019 for the synthetic setup) are not load-bearing: the synthetic tasks are externally defined and SEFA is compared against non-author baselines. No circularity pattern is exhibited.
Assumptions & free parameters
free parameters (5)
- beta =
0.0001 to 0.005 depending on dataset
- latent_components_per_feature =
4 to 8 (Table 16)
- number_of_acquisition_samples =
200
- number_of_training_samples =
100
- encoder_predictor_widths_and_depth =
varies by dataset
assumptions (5)
- ad hoc to paper Gradient norm of class probability with respect to latent components is a valid importance measure for feature acquisition
- domain assumption Sampling from the encoder posterior p(z|x_O) approximates the distribution of unobserved feature values needed for non-myopic decisions
- domain assumption Feature-wise independent encoders with a shared predictor can capture inter-feature dependencies
- domain assumption The variational information bottleneck with N(0,1) prior removes label-irrelevant noise from the latent space
- standard math Standard probability calculus and information theory identities
Cite this review
Pith. "Pith review of Stochastic Encodings for Active Feature Acquisition." pith.science (2026). https://pith.science/paper/S57Q435A
@misc{pith2026250801957,
author = {Pith},
title = {Pith review of: Stochastic Encodings for Active Feature Acquisition},
year = {2026},
howpublished = {\url{https://pith.science/paper/S57Q435A}},
note = {Machine review of arXiv:2508.01957}
}
read the original abstract
Active Feature Acquisition is an instance-wise, sequential decision making problem. The aim is to dynamically select which feature to measure based on current observations, independently for each test instance. Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic acquisitions. To address these shortcomings, we introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a stochastic latent space. Extensive evaluation on a large range of synthetic and real datasets demonstrates that our approach reliably outperforms a diverse set of baselines.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[1]
The train set is size 60,000, and the validation and test sets are both size 10,000
This is so all the models have stronger purely predictive performance. The train set is size 60,000, and the validation and test sets are both size 10,000. AUROC is used as the evaluation metric. Cube. The Cube dataset is a synthetic dataset that is regularly used to evaluate Active Feature Acquisition methods (Rückstieß et al., 2013; Shim et al., 2018; Z...
work page 2013
-
[3]
is selected after ST6GAL1 (feature 18). This is because it has been linked to: bladder cancer, breast cancer, gastric carcinoma, liver cancer, lung adenocarcinoma, lung squamous cell carcinoma, ovarian cancer, cervical squamous cell carcinoma, head-neck squamous cell carcinoma, pancreatic adenocarcinoma, and kidney renal clear cell carcinoma (Deng et al.,...
work page 2021
-
[5]
Note this is not an exhaustive list
(Penna-Martinez et al., 2014). Note this is not an exhaustive list. 20 Stochastic Encodings for Active Feature Acquisition 1 2 3 4 5 6 C7orf51 DEF6 DNASE1L3 EFS FOXE1 GPR81 GRIA2 GSDMC HOXA9 KAAG1 KLF5 LOC283392 LTBR LYPLAL1 PON3 POU3F3 SERPINB1 ST6GAL1 TMEM106A ZNF583 ZNF790 Breast 1 2 3 4 5 6 C7orf51 DEF6 DNASE1L3 EFS FOXE1 GPR81 GRIA2 GSDMC HOXA9 KAAG1...
work page 2014
-
[7]
Dynamic Feature Acquisition with Arbitrary Conditional Flows
Li, Y . and Oliva, J. B. Dynamic Feature Acquisition with Arbitrary Conditional Flows. arXiv preprint arXiv:2006.07701,
work page Pith review arXiv 2006
-
[9]
Rangrej, S. B. and Clark, J. J. A Probabilistic Hard Attention Model for Sequentially Observed Scenes. arXiv preprint arXiv:2111.07534,
-
[13]
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learn- ing Algorithms
Xiao, H., Rasul, K., and V ollgraf, R. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learn- ing Algorithms. arXiv preprint arXiv:1708.07747,
-
[16]
(Zhang et al., 2021), KAAG1 (feature
work page 2021
-
[17]
(Shenoy et al., 2024); colon cancer acquiring HOXA9 (Cui et al.,
work page 2024
Show all 27 references
-
[19]
However, feature 1 consistently has a lower entropy, so it would be selected before feature 2 by CMI
x1 x2 p(Y = 1|x) p(Y = 2|x) p(Y = 3|x) H(Y |x) H(Y ) − H(Y |x) 1 Missing 0.03346 0.48327 0.48327 0.81652 0.28210 2 Missing 0.48327 0.03346 0.48327 0.81652 0.28210 3 Missing 0.48327 0.48327 0.03346 0.81652 0.28210 Missing 1 0.69912 0 .15044 0 .15044 0.82016 0.27845 Missing 2 0....
2019
-
[22]
a deep learning method for feature selection. After flattening the images to vectors, the features found by STG were: • MNIST: [153, 154, 210, 211, 243, 269, 271, 295, 327, 348, 350, 375, 405, 409, 427, 430, 461, 514, 543, 655] • Fashion MNIST: [10, 38, 121, 146, 202, 246, 248...
2012
-
[25]
The information network is used to score each feature
uses two separate networks, one for prediction and one for predicting the CMI of features with the label. The information network is used to score each feature. The information network limits the output 26 Stochastic Encodings for Active Feature Acquisition to a minimum of zer...
2000
-
[26]
The encoder is a Partial V AE
is an advanced generative modeling method for AFA. The encoder is a Partial V AE. We encode the label in the same way as a categorical feature. We do not include a separate predictor, instead we follow the original paper to make predictions: features are encoded to a latent di...
2017
-
[27]
However, the pattern is as expected, the models that train a policy network by simulating acquisition are slower to train than the generative models and SEFA
These results should be treated carefully, since this depends on the number of epochs, different methods converge at different rates. However, the pattern is as expected, the models that train a policy network by simulating acquisition are slower to train than the generative m...
2015
-
[191]
The evaluation metric is accuracy. TCGA. The Cancer Genome Atlas (TCGA) consists of genetic data for over 11,000 cancer patients (Weinstein et al., 2013). The data was accessed at https://www.cancer.gov/ccg/research/genome-sequencing/tcga. We construct the classification task ...
2013
-
[792]
The evaluation metric is accuracy. I. Model Details and Implementations All models were implemented using PyTorch (Paszke et al., 2017); code is available at https://github.com/ a-norcliffe/SEFA. I.1. General Model Details Here we provide details that tend to be shared across ...
2017
-
[2010]
DOI: https://doi.org/10.24432/C5QC87. Roe, B. P., Yang, H.-J., Zhu, J., Liu, Y ., Stancu, I., and McGregor, G. Boosted decision trees as an alternative to artificial neural networks for particle identification. Nuclear Instruments and Methods in Physics Research Section A: Acc...
-
[2011]
The labels are median house prices in California districts expressed in 100,000 dollars
https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_ california_housing.html. The labels are median house prices in California districts expressed in 100,000 dollars. There are 8 continuous features that can be found at the above URL. To convert this to a ...
2005
-
[2012]
DNASE1L3 as a Prognostic Biomarker Associated with Immune Cell Infiltration in Cancer
Deng, Z., Xiao, M., Du, D., Luo, N., Liu, D., Liu, T., Lian, D., and Peng, J. DNASE1L3 as a Prognostic Biomarker Associated with Immune Cell Infiltration in Cancer. On- coTargets and Therapy, pp. 2003–2017,
2003
-
[2013]
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347,
-
[2014]
Feature-Budgeted Random Forest
Nan, F., Wang, J., and Saligrama, V . Feature-Budgeted Random Forest. In International Conference on Machine Learning, pp. 1983–1991. PMLR,
1983
-
[2016]
M., and Chen, T
He, W., Mao, X., Ma, C., Huang, Y ., Hernàndez-Lobato, J. M., and Chen, T. BSODA: a Bipartite Scalable Frame- work for Online Disease Diagnosis. In Proceedings of the ACM Web Conference 2022, pp. 2511–2521,
2022
-
[2018]
Opportunistic Learning: Budgeted Cost- Sensitive Learning from Data Streams
Kachuee, M., Goldstein, O., Kärkkäinen, K., and Sar- rafzadehm, M. Opportunistic Learning: Budgeted Cost- Sensitive Learning from Data Streams. In International Conference on Learning Representations, 2019a. Kachuee, M., Karkkainen, K., Goldstein, O., Zamanzadeh, D., and Sarra...
1902 arXiv
-
[2019]
Context-aware feature query to improve the prediction performance
Kachuee, M., Hosseini, A., Moatamed, B., Darabi, S., and Sarrafzadeh, M. Context-aware feature query to improve the prediction performance. In 2017 IEEE Global Confer- ence on Signal and Information Processing (GlobalSIP), pp. 838–842. IEEE,
2017
-
[2020]
J., Zhang, C., and Tschiatschek, S
Yin, H., Li, Y ., Pan, S. J., Zhang, C., and Tschiatschek, S. Reinforcement Learning with Efficient Active Feature Acquisition. arXiv preprint arXiv:2011.00825,
2011 arXiv
-
[2021]
Datum-Wise Classification: A Sequential Approach to 10 Stochastic Encodings for Active Feature Acquisition Sparsity
Dulac-Arnold, G., Denoyer, L., Preux, P., and Gallinari, P. Datum-Wise Classification: A Sequential Approach to 10 Stochastic Encodings for Active Feature Acquisition Sparsity. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athen...
2011
-
[2023]
Active Infor- mation Acquisition
He, H., Mineiro, P., and Karampatziakis, N. Active Infor- mation Acquisition. arXiv preprint arXiv:1602.02181,
-
[2024]
Deep Reinforcement Learning and the Deadly Triad
Van Hasselt, H., Doron, Y ., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep Reinforcement Learning and the Deadly Triad. arXiv preprint arXiv:1812.02648,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.