{"id":"8c4e9468-9632-4f67-bae4-8213e1fc2004","arxiv_id":"2611.16792","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Uncertainty-based active learning reduces labeling requirements by 13-30% compared to random sampling for descriptor-based models, but the best strategy varies with model class and dataset.","lead":"This paper systematically compares random, uncertainty-based, and diversity-based sampling strategies for building training datasets for machine learning models that predict molecular and materials properties. The study finds that uncertainty sampling generally reduces the number of expensive labels needed to reach a target accuracy, but the benefit depends strongly on the model, dataset, and property.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Uncertainty-sampling advantage for descriptor-based ensemble is modest and may not generalize to the most practically relevant model class (SchNet).","rationale":"The reader's weakest_assumption correctly identifies the single-run SchNet experiments as a weakness. However, the reader frames this as a statistical-variance issue that could be fixed by adding error bars. I see a deeper structural issue: the central claim (13–30% label savings from US) is demonstrated only for the descriptor-based ensemble, which is the less accurate model class. The paper's own results show that SchNet is superior at every budget, so the practical recommendation for users of structure-aware models is unsupported. The reader's verdict (CONDITIONAL) is appropriate, but the condition should be stronger: the paper should either (a) provide evidence that the US advantage holds for SchNet, or (b) explicitly restrict the central claim to descriptor-based models. The agreement is 'partial' because the reader focuses on statistical reliability of SchNet results, whereas I focus on the mismatch between the claim and the model class for which it is demonstrated.","tokens_in":13620,"tokens_out":643,"duration_ms":6060,"concrete_test":"Run the SchNet active learning experiments on QM9 and Matbench MP with at least 3 random seeds (as was done for the ensemble). Compute N10% for each seed and report the mean and standard deviation. If the relative ordering of strategies (US vs RS vs DS) is consistent across seeds and the 13–30% savings are observed for SchNet, the concern is resolved. If the ordering is inconsistent or the savings are smaller, the paper's central claim should be restricted to descriptor-based models.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that uncertainty-based sampling reduces labels by 13–30% relative to random sampling—rests entirely on the descriptor-based ensemble experiments (Table III). For SchNet, which is the more accurate model on both large datasets (RMSE 0.090 vs 0.339 eV on QM9; 0.070 vs 0.268 eV/atom on Matbench MP), no strategy reaches the 10% threshold within the budget, so N10% cannot be computed. The paper acknowledges this (Section III.E, Limitations) and notes that SchNet was run only once per configuration. However, the practical recommendation for users who can afford structure-aware models (SchNet or similar) is therefore unsupported: the 13–30% savings are demonstrated only for a model class that is substantially less accurate at every budget tested. The paper's own Figure 6 shows that the ensemble is never competitive with SchNet in absolute error, so a practitioner choosing SchNet would have no evidence from this study about which sampling strategy to use. The claim that 'uncertainty-based sampling generally outperforms alternative strategies' is thus true only for the weaker model class, and the paper does not establish that this finding transfers to the model class that matters most for accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents a systematic benchmark of active learning strategies (random, uncertainty, diversity sampling) for molecular and materials property prediction using descriptor-based ensemble models and SchNet across four datasets. The central claim is that uncertainty-based sampling reduces the number of labeled examples required to reach within 10% of full-data error by 13-30% relative to random sampling for descriptor-based ensemble models. The paper also finds that SchNet outperforms the ensemble at all budgets but its learning curves lack error bars due to single-run training.","tokens_in":13992,"tokens_out":2112,"duration_ms":14486,"significance":"If the results hold, the paper provides practical guidelines for data acquisition in computational materials science, particularly for descriptor-based models where uncertainty sampling offers modest but consistent savings. The benchmark is thorough in terms of datasets and model classes, and the code is provided. However, the main practical finding is limited to the weaker model class (ensemble), and the most practically relevant model (SchNet) yields no actionable N10% values. The paper's strength is its systematic experimental design with stratified splits, multiple seeds for ensemble, and hyperparameter tuning via cross-validation.","major_comments":[{"comment":"The central quantitative claim (13-30% reduction in labels via uncertainty sampling) is derived entirely from the descriptor-based ensemble experiments (Table III). For SchNet, which is the more accurate model on both large datasets (RMSE 0.090 vs 0.339 eV on QM9; 0.070 vs 0.268 eV/atom on Matbench MP), no strategy reaches the 10% threshold within the budget, so N10% cannot be computed. The paper acknowledges this (Section III.E, Limitations) and notes that SchNet was run only once per configuration. However, the practical recommendation for users who can afford structure-aware models (SchNet or similar) is therefore unsupported: the 13-30% savings are demonstrated only for a model class that is substantially less accurate at every budget tested. The paper's own Figure 6 shows that the ensemble is never competitive with SchNet in absolute error, so a practitioner choosing SchNet would have no evidence from this study about which sampling strategy to use. The claim that 'uncertainty-based sampling generally outperforms alternative strategies' is thus true only for the weaker model class, and the paper does not establish that this finding transfers to the model class that matters most for accuracy. This is a load-bearing limitation because the abstract and conclusions present the finding as general without sufficiently qualifying the model class.","section":"Table III, Section III.E","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The SchNet experiments were performed with only a single random seed per configuration. The authors acknowledge this limitation, but the consequence is that the SchNet learning curves lack error bars, and the relative ranking of sampling strategies for SchNet (e.g., US vs DS vs RS) may be affected by random initialization variance that is not quantified. Given that the differences between strategies for SchNet are small (e.g., final RMSE 0.114 vs 0.119 vs 0.126 eV on QM9), the ranking could plausibly change with additional runs. This undermines the reliability of any SchNet-specific conclusions, including the observation that DS outperforms RS on Matbench MP (Fig. 4b). The paper should either provide multiple runs or explicitly state that no reliable conclusions can be drawn for SchNet regarding strategy ranking.","section":"Section III.E, Limitations","authors_acknowledge":"Consequently, we are unable to report standard devia- tions for the SchNet learning curves, which hinders a rig- orous assessment of the model’s statistical variance across different random weight initializations.","authors_acknowledge_p":0.93},{"comment":"The uncertainty sampling (US) strategy uses query-by-committee with a committee of three different models (RF, KRR, XGB) for the ensemble, but for SchNet the committee is formed by bootstrap resampling of the same model. This asymmetry means that the US signal is qualitatively different between the two model classes: for the ensemble, disagreement reflects model class diversity; for SchNet, it reflects training data stochasticity. The paper does not discuss how this difference might affect the comparison of US effectiveness across model classes. Since the central claim about US superiority is based on the ensemble experiments, this is not a fatal flaw, but it should be acknowledged as a confound when interpreting the SchNet results.","section":"Section II.C, Equation (1)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states that 'uncertainty-based sampling generally outperforms alternative strategies' without specifying that this conclusion is drawn from descriptor-based ensemble models. Consider adding a qualifier such as 'for descriptor-based models' to avoid overgeneralization.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 2b (SchNet on QM9) shows that no strategy reaches the 10% threshold, but the y-axis range is truncated near the threshold. Consider extending the y-axis to show the full range of errors or adding a horizontal line at the threshold for clarity.","section":"Section III.B.1, Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The footnote states that N10% is rounded to the nearest 1000 labels (100 for small datasets) and that percentages are computed from unrounded crossings. This is fine, but the table would benefit from including the unrounded values or confidence intervals (e.g., from the five seeds) for the ensemble experiments to convey uncertainty.","section":"Table III","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of diversity sampling (TypiClust) uses k-means clustering in descriptor space for classical models and in learned latent space for SchNet. The number of clusters is set to |L(t)| + b, which grows with iterations. This choice is not justified; a brief rationale would help.","section":"Section II.C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 6 compares ensemble vs SchNet using the 'best strategy per dataset and model.' For Matbench MP, the ensemble uses US and SchNet uses DS. This is an asymmetric comparison; consider also showing the ensemble with DS or SchNet with US to illustrate the range of outcomes.","section":"Section III.D, Figure 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The descriptor-based ensemble uses Morgan fingerprints and MACCS keys for QM9, but the scalar properties (μ, α, ZPVE, etc.) are also included. It is unclear whether these scalar properties are used as features or only for stratification. Please clarify.","section":"Section II.B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The hyperparameters for RF, KRR, and XGB are listed for each dataset. It would be helpful to note whether these were selected via cross-validation on the full training pool or on the initial labeled set L(0).","section":"Appendix A, Table IV","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Pourbaix stability dataset is described as having 6,870 samples, but the training pool is 5,496 (80% of 6,870). The paper states that 'structural relaxations were not performed at the HSE06 level for curating this dataset.' This is a minor point, but it would be useful to clarify whether the structures are experimental or from a different DFT level.","section":"Section II.A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The redundancy analysis states that 'even for RS, only 15-25% of the pool is sufficient on QM9 and Matbench MP datasets.' This is consistent with prior work (Li et al.). The paper could strengthen this point by directly comparing the N10% fractions to those reported in Li et al. for similar models.","section":"Section III.C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conclusion states that 'uncertainty sampling was the most label-efficient strategy for the descriptor-based models on all four datasets, reducing the number of labels required by roughly 13-30% relative to random sampling.' This is accurate, but the 13-30% range should be explicitly tied to the values in Table III (e.g., QM9: 25% to 17% = 32% reduction; QM9small: 46% to 40% = 13% reduction; Matbench MP: 15% to 13% = 13% reduction; Pourbaix: 77% to 54% = 30% reduction). The paper currently states '13-30%' without showing the calculation.","section":"Section IV (Conclusions)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'N10%' as a metric for data efficiency. It is defined as the minimum number of labels such that RMSE ≤ 1.1 × RMSE_full. This metric is reasonable, but it is a relative measure that depends on the model's own full-data performance. The paper correctly notes that N10% values are not comparable across model classes (Section III.D). However, the abstract and conclusions sometimes imply that N10% reductions are universally beneficial without emphasizing that they are model-specific.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper is a systematic benchmark of three sampling strategies (random, uncertainty, diversity) across two model classes (descriptor-based ensemble, SchNet) and four datasets. The main finding—that uncertainty sampling reduces labels by 13–30% relative to random for the descriptor-based ensemble—is cleanly demonstrated with five random seeds and is a useful quantitative result for practitioners using composition-based models. The inclusion of the Pourbaix stability dataset (HSE06 labels) is a nice addition; it's the kind of small, expensive dataset where active learning matters most, and the savings there are the largest (22% of the pool).\n\nThe paper is well-executed and the authors are transparent about limitations. The SchNet experiments were run only once per configuration, so the learning curves lack error bars. The authors state this explicitly in Section III.E. The stress-test concern—that the central claim does not transfer to SchNet, which is the more accurate model on both large datasets—is correct but the paper does not overclaim. The authors note that N10% could not be computed for SchNet and that the best strategy for SchNet changed with the dataset. The practical recommendation for users of structure-aware models is therefore unsupported by this study, but the paper does not pretend otherwise.\n\nThe soft spot is real: the 13–30% savings are demonstrated only for the descriptor-based ensemble, which is substantially less accurate than SchNet at every budget. A practitioner choosing SchNet gets no evidence from this study about which sampling strategy to use. This is a limitation of scope, not a flaw in execution. The paper would be stronger if it included error bars for SchNet or used a cheaper structure-aware model that allows multiple runs, but the authors acknowledge the constraint.\n\nWho this is for: practitioners using descriptor-based models (composition-based features) who want quantitative guidance on sampling strategies. The paper delivers that. For users of graph neural networks, the results are suggestive at best.\n\nRecommendation: Accept with minor revision. The single-run SchNet limitation should be stated more prominently in the abstract or conclusions, not just in a dedicated limitations section. The authors should also clarify that the 13–30% savings apply to the ensemble, not to SchNet. The paper deserves a serious referee; it is a solid empirical contribution that will be useful to the community.","headline":"Useful benchmark with a clear limitation: the central claim about uncertainty sampling rests on the weaker model class, and the paper is honest about it.","tokens_in":14484,"tokens_out":565,"duration_ms":120086},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Uncertainty sampling cuts labeling needs by 13–30% for molecular and materials property prediction.","keywords":["active learning","materials informatics","molecular property prediction","uncertainty sampling","query by committee","data efficiency","SchNet","descriptor-based models"],"falsifier":"If a replication using the same datasets and models but with multiple random seeds per SchNet configuration found that random sampling consistently matches or outperforms uncertainty sampling for SchNet on both QM9 and Matbench MP, the paper's claim that uncertainty sampling is generally most effective for ensembles but dataset-dependent for SchNet would need to be revised for the deep-learning case.","tokens_in":15107,"feed_emoji":"","tokens_out":1474,"duration_ms":11075,"temperature":0.7,"pith_summary":"This paper systematically compares strategies for selecting training data when building machine-learning models that predict properties of molecules and materials. The central finding is that uncertainty-based sampling—choosing the next examples to label based on how much the model's predictions disagree—consistently reduces the number of labeled examples required to reach a target accuracy by 13–30% relative to random sampling, across four datasets and several model types. However, the paper also shows that the effectiveness of any sampling strategy depends strongly on the interplay between the model architecture, the representation of the input data, and the specific prediction task. The authors derive practical guidelines for choosing a data acquisition strategy under a given labeling budget, emphasizing that no single strategy is universally optimal.","feed_headline":"Uncertainty sampling cuts labeling needs by 13–30%","feed_subtitle":"Systematic study across four datasets shows query-by-committee is most label-efficient for descriptor-based property models.","key_machinery":"The central experimental apparatus is a pool-based active learning framework where models are iteratively retrained on growing labeled sets. Three acquisition strategies are compared: random sampling (baseline), uncertainty sampling using query-by-committee (variance of predictions from three different regressors for the ensemble, or from bootstrap-resampled SchNet models), and diversity sampling using TypiClust (k-means clustering followed by typicality scoring). Performance is measured by the N10% metric—the smallest training set size at which test RMSE falls within 10% of the model's full-data RMSE—and by absolute RMSE at each budget.","core_discovery":"The paper establishes that for descriptor-based ensemble models (random forest, kernel ridge regression, gradient boosting), uncertainty sampling via query-by-committee is the most label-efficient strategy across all four datasets tested (QM9, QM9small, Matbench MP, Pourbaix stability). It reduces the number of labeled examples needed to reach within 10% of the full-data error by roughly 13–30% compared to random sampling, with the largest gains on the smallest and most expensive dataset (Pourbaix stability). Diversity sampling (TypiClust) gives no consistent advantage over random sampling. For the deep-learning model SchNet, the relative ranking of strategies changes with the dataset, and no strategy reaches the 10% threshold within the labeling budget. The paper also quantifies dataset redundancy: under random sampling, only 15–25% of the pool is needed for the two large datasets (QM9, Matbench MP), whereas the smaller datasets require 46–77%.","pith_inferences":["The paper's finding that diversity sampling underperforms uncertainty sampling for descriptor-based models suggests that coverage of input space without feedback from model errors is a weak signal for label efficiency. A natural extension would be to test hybrid strategies that combine uncertainty and diversity criteria.","Because the SchNet results lack error bars (single seed per configuration), the observed differences between strategies for SchNet may not be statistically significant. A replication with multiple seeds would clarify whether the dataset-dependent ranking is robust.","The N10% metric, while useful, conflates two distinct quantities: the model's absolute accuracy and the data efficiency. The paper's cross-model comparisons using absolute errors partially address this, but a normalized efficiency metric (e.g., relative reduction in labels to reach a fixed absolute error) could sharpen comparisons across model classes.","The Pourbaix stability dataset, being small and expensive, represents the regime where active learning is most valuable. The 22% reduction in labels achieved by uncertainty sampling suggests that similar gains may be attainable for other high-fidelity materials properties computed with hybrid functionals or correlated wavefunction methods."],"forward_implications":["If the findings hold generally, researchers with limited labeling budgets (e.g., expensive DFT calculations) should prioritize uncertainty-based acquisition over random or diversity sampling when using descriptor-based ensemble models.","The high redundancy observed in large datasets (QM9, Matbench MP) suggests that many published models could be trained on far fewer examples without substantial accuracy loss, potentially reducing computational and data-generation costs.","For deep-learning models like SchNet, the choice of sampling strategy matters less than the model's intrinsic accuracy; the paper implies that investing in better representations or architectures may yield larger gains than optimizing the sampling strategy.","The paper's guidelines—use uncertainty sampling for ensembles, consider dataset size and redundancy when planning acquisition, and evaluate the model's starting accuracy before deploying active learning—provide a practical decision framework for practitioners."],"supporting_citations":[{"why":"Establishes the baseline finding that large materials datasets are highly redundant, which the present work confirms and extends.","marker":"[1]"},{"why":"Provides the SchNet architecture used as the deep-learning model throughout the study.","marker":"[7]"},{"why":"Supplies the QM9 dataset of molecular properties used for the HOMO-LUMO gap prediction task.","marker":"[8]"},{"why":"Supply the Matbench MP dataset of inorganic crystal formation energies used as one of the four benchmark datasets.","marker":"[9, 10]"},{"why":"Supplies the Pourbaix stability dataset with HSE06-level labels, which is the smallest and most expensive dataset in the study.","marker":"[12]"},{"why":"Introduces the query-by-committee uncertainty sampling method that is the central active learning strategy evaluated.","marker":"[24]"},{"why":"Provides the TypiClust diversity sampling algorithm used as the alternative acquisition strategy.","marker":"[25]"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The SchNet models were trained with only one random seed per configuration, so the learning curves for SchNet lack error bars and the reported rankings of sampling strategies for SchNet may be affected by unquantified random initialization variance.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8.9e-05,"raw_usage":{"cost":8.8634e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":8.8634e-05,"upstream_inference_prompt_cost":8.827e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":972,"prompt_tokens":970,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":970,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":1491,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T12:44:21.398096+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a replication using the same datasets and models but with multiple random seeds per SchNet configuration found that random sampling consistently matches or outperforms uncertainty sampling for SchNet on both QM9 and Matbench MP, the paper's claim that uncertainty sampling is generally most effective for ensembles but dataset-dependent for SchNet would need to be revised for the deep-learning case.","supporting_citations":[{"cited_title":"et al., Nat. Commun. 14, 7283","cited_arxiv_id":null,"evidence_quote":"Establishes the baseline finding that large materials datasets are highly redundant, which the present work confirms and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SchNet architecture used as the deep-learning model throughout the study."},{"cited_title":"Sci. Data1, 140022","cited_arxiv_id":null,"evidence_quote":"Supplies the QM9 dataset of molecular properties used for the HOMO-LUMO gap prediction task."},{"cited_title":"npj Comput. Mater. 11, 150","cited_arxiv_id":null,"evidence_quote":"Supplies the Pourbaix stability dataset with HSE06-level labels, which is the smallest and most expensive dataset in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the query-by-committee uncertainty sampling method that is the central active learning strategy evaluated."},{"cited_title":"inProc. ICML","cited_arxiv_id":null,"evidence_quote":"Provides the TypiClust diversity sampling algorithm used as the alternative acquisition strategy."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}