{"id":"aff4285f-8ea3-4670-9add-53c5ec903609","arxiv_id":"1908.02398","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding 871 polymer Tg values to the Polymer Genome dataset lowers the in-sample RMSE from 57 K to 27 K, but no cross-validation shows whether true predictive accuracy improved.","lead":"Researchers collected 871 new experimental glass transition temperatures for polymers and retrained the Polymer Genome machine learning model on the combined 1,321 polymers. The paper reports the prediction error dropped from 57 K to 27 K, but this number is computed on the training data, not a held-out test set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 27 K RMSE for PG-1 is an in-sample training error, not an out-of-sample prediction, so it cannot be compared with PG-0's 57 K error on unseen polymers.","rationale":"The reader's weakest assumption is exactly the load-bearing vulnerability I identified: PG-1's reported RMSE is evaluated on its own training data. The paper does include one legitimate out-of-sample check, PG-0 predicting the 871 new polymers, and that result (>50 K RMSE) is honest evidence that PG-0 lacks transferability. But the improvement claim rests on the 27 K figure for PG-1 on the full 1,321-polymer set, which is a training-fit metric. Because the abstract and Results present 27 K as predictive accuracy and compare it with a genuinely held-out 57 K, the main conclusion is not supported as written. The collected 871 Tg values may still be a useful dataset, and the flaw is reparable by adding cross-validation or a fixed holdout split. However, no such validation appears in the manuscript, so the current central claim cannot be accepted. This supports the reader's REJECT verdict; I see no additional concern that would move the verdict in another direction.","tokens_in":4488,"tokens_out":4278,"duration_ms":44634,"concrete_test":"Retrain PG-1 with identical fingerprints and kernel hyperparameters under 5-fold cross-validation on the 1,321-polymer dataset, and report the pooled held-out RMSE and R2. If the cross-validated RMSE is not below roughly 30 K, or is not clearly better than PG-0's out-of-sample RMSE on the 871 new polymers, the claimed improvement from 57 K to 27 K is an artifact of training-set evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that retraining on the full 1,321-polymer dataset lowers prediction RMSE from 57 K to 27 K. However, Figure 2(b) reports the PG-1 RMSE computed on the same full dataset used to train PG-1; no held-out subset or cross-validation is described anywhere in Results or Methods. By contrast, the 57 K value for PG-0 is an out-of-sample error on the 871 newly collected polymers, which PG-0 had not seen. Comparing an in-sample fit to an out-of-sample test conflates memorization with generalization. For GPR with an RBF plus white-noise kernel, training-set RMSE can be driven near the measurement noise level even when held-out error is much larger, and the reported shrinkage of GPR uncertainties on training points is expected by construction. The sentence 'The RMSE of prediction for the extended dataset, when compared to the earlier one, decreased to 27 K from 57 K' therefore does not establish improved predictive capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on the collection of 871 additional experimental glass transition temperature (Tg) values for polymers, augmenting an earlier dataset of 450 polymers to form a combined set of 1,321 polymers. The authors evaluate the previous Polymer Genome (PG-0) model on the new data, finding a high out-of-domain RMSE of 57 K, then retrain a Gaussian process regression model (PG-1) on the full 1,321 data points and report that the RMSE 'of prediction' drops to 27 K. The paper argues that this improvement demonstrates the value of data expansion and yields a more versatile predictive model. The main contributions are the new data collection and the qualitative demonstration that PG-0 transfers poorly to out-of-domain polymers.","tokens_in":4827,"tokens_out":3599,"duration_ms":37756,"significance":"The new dataset of 871 experimentally measured Tg values is a valuable community resource, and the demonstration that PG-0's predictions on these new polymers carry an RMSE of 57 K is a useful, falsifiable negative result about the transferability of models trained on a narrow chemical space. If the reported improvement to 27 K RMSE were a genuine out-of-sample prediction error, the result would be significant for polymer informatics, showing that data expansion can substantially improve predictive accuracy. However, the central quantitative claim is undermined by the evaluation protocol: the 27 K figure is computed on the same data used for training and therefore cannot be interpreted as prediction error. The paper's strengths — careful data collection, use of an established fingerprinting scheme, and consistent application of GPR — are real, but they are attached to a headline result that does not support the stated conclusion of improved predictive capability. The manuscript is more of a dataset and benchmark report than a validated methodological advance.","major_comments":[{"comment":"The abstract and Results state that 'The RMSE of prediction for the extended dataset ... decreased to 27 K from 57 K.' This comparison is invalid because PG-1 was trained on the full set of 1,321 polymers and then evaluated on that same full set, whereas the 57 K value for PG-0 was an out-of-sample error computed on the 871 newly collected polymers not present in PG-0's training set. The 27 K figure is therefore a training-set RMSE, not a prediction error. For Gaussian process regression with a radial basis function plus white-noise kernel, the training-set RMSE can be driven toward the noise level even when held-out error is much larger, so this number carries no evidence of generalization. To support the claim of improved predictive capability, the authors must evaluate PG-1 on data not used for training — for example, treat the 871 new polymers as a held-out test set, or perform k-fold cross-validation over the full 1,321 — and report the resulting out-of-sample RMSE alongside the 57 K baseline.","section":"Results, Fig. 2(b)"},{"comment":"No validation protocol is described. The Methods section explains the fingerprinting approach and the GPR kernel but does not state how the kernel hyperparameters were optimized, whether any portion of the 1,321 polymers was held out for testing, or whether cross-validation was used. In the absence of this information, the reported R2=0.92 and RMSE=27 K in Figure 2(b) cannot be interpreted as predictive performance. The manuscript should explicitly describe the train/test split or cross-validation procedure, the hyperparameter optimization scheme, and the metrics computed on held-out data.","section":"Methods"},{"comment":"The claim that the retrained model 'is hence more versatile' is unsupported by the reported experiments. Versatility is a statement about out-of-sample and out-of-domain generalization, yet the quantitative evidence in the manuscript is an in-sample fit on the training data. Without held-out evaluation, the discussion overstates the conclusion that adding diverse data has led to 'a significantly better predictive capability of PG.'","section":"Discussion"}],"minor_comments":[{"comment":"References 7 and 8 do not support the statement about inverse design algorithms: reference 7 is a blog post on variational autoencoders and reference 8 concerns a vendor-managed inventory optimization problem; neither is a representative citation for polymer inverse design. Please cite relevant literature on generative models for materials discovery.","section":"References"},{"comment":"The axes of the parity plots in Figure 2 are not labeled, and the parity line is not identified. Units (K) should be given on both axes, and the dashed line should be clearly described as the ideal parity line.","section":"Figure 2"},{"comment":"The statement that 'the uncertainties calculated by GPR shown by the yellow error bars have also decreased significantly' is made without quantitative support; the authors should report the average or distribution of predicted uncertainties for the new polymers under PG-0 versus PG-1.","section":"Results"},{"comment":"The manuscript says 'the RMSE ... is greater than 50 K' for PG-0 on the new 871 polymers, but the figure caption reports RMSE=57 K. Please make the value consistent or state whether 57 K is the exact value and 'greater than 50 K' is a rounding description.","section":"Results"}],"recommendation":"reject","confidential_remarks":"The core quantitative result is an in-sample training error presented as a prediction error, which is a fundamental methodological flaw for a paper whose central claim is about predictive capability. The dataset itself is valuable and the transferability failure of PG-0 is interesting, but the manuscript in its current form does not support the headline conclusion. If the authors can supply a proper cross-validation analysis and reframe the contribution as a dataset and benchmark study, a resubmission might be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline number in this paper—27 K RMSE for the retrained PG-1 model—is an in-sample training error. PG-1 was trained on all 1,321 polymers and then evaluated on that same set. The 57 K figure for PG-0, by contrast, is an out-of-sample error on the 871 newly collected polymers. Comparing the two conflates memorization with generalization. The stress-test note is right, and the paper's central claim falls.\n\nWhat is actually new: the collection of 871 experimental Tg values from public sources. That is a genuine addition to the field, and the Polymer Genome platform has a track record of making models available. If the dataset is released in a usable form, it is a useful resource for benchmarking future polymer ML models. The paper also makes a reasonable qualitative observation that PG-0, trained on only 450 polymers, predicts poorly for many out-of-domain polymers and that GPR uncertainties are higher for those points. That part is plausible and consistent with the model.\n\nThe soft spot is not minor: the evaluation of PG-1 is missing any holdout set or cross-validation. A GPR with RBF and white-noise kernels can drive training RMSE down to measurement noise levels, so the 27 K number tells us almost nothing about predictive accuracy on unseen polymers. The paper does not report a train/test split, validation, or any independent test. In fact, comparing Figure 2(a) and 2(b) is apples-to-oranges for exactly that reason. The authors might well have a decent model—the methodology is standard—but this manuscript does not support the claim.\n\nThere is also a small citation oddity: reference 7 is a towardsdatascience blog post, and reference 8 is an unrelated inventory management paper. These look like filler and should be fixed if the paper goes further.\n\nWho is this for? A reader interested in polymer data resources, not in ML methodology. The paper is a brief data report. It deserves a serious referee because the dataset expansion is useful, but the referee should require a proper out-of-sample evaluation (cross-validation or a held-out test) and a clear statement of the training/evaluation split. As is, the core quantitative claim is not established. I would not cite the 27 K number, but I would encourage the authors to release the data and rerun with a sound evaluation.\n\nRecommendation: send to peer review only if the authors fix the evaluation. Otherwise, it is a dataset announcement, not a prediction improvement paper.","headline":"The 27 K RMSE for PG-1 is an in-sample training error, not an out-of-sample prediction, so the paper's headline improvement over PG-0's 57 K is not established.","tokens_in":5198,"tokens_out":2445,"would_cite":false,"duration_ms":23149,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding 871 polymers to a polymer informatics model's training data cuts glass-transition temperature prediction error from 57 K to 27 K.","keywords":["machine learning","glass transition temperature","Gaussian process regression","polymer fingerprinting","property prediction","materials data","transferability"],"falsifier":"Retrain the model on a random 80% of the 1,321 polymers, predict the remaining 20%, and compute the RMSE; if the held-out RMSE is much larger than 27 K, for instance 50 K or more, then the reported improvement overstates what the model will do on unseen polymers.","tokens_in":4297,"feed_emoji":"🧪","tokens_out":8315,"duration_ms":82711,"temperature":0.7,"pith_summary":"This paper seeks to show that expanding a machine-learning model's training data from 450 to 1,321 polymers substantially improves its predictions of the glass transition temperature ($T_g$), the temperature above which a polymer turns from glass-like to rubber-like. The authors collected 871 new experimental $T_g$ values, showed that the old model failed on them (root-mean-square error above 50 K), then retrained the same Gaussian-process model on the combined data and report the error dropping from 57 K to 27 K. If this holds, the model becomes accurate enough to be a practical screening tool across a much more chemically diverse polymer space, and the same data-collection-and-retraining recipe could improve other property predictions.","feed_headline":"Feeding 871 new polymers cuts glass-transition error from 57 to 27 K","feed_subtitle":"Retraining on 1,321 polymers brings predictions in line with experimental measurement scatter.","key_machinery":"The load-bearing mechanism is the hierarchical polymer fingerprint: each repeat unit is encoded as a vector of 128 atomic-fragment counts, 39 quantitative structure-property relationship descriptors (such as estimated surface area and fraction of rotatable bonds), and 22 morphological features (such as aromatic-ring topology and side-chain lengths). This fingerprint is mapped to $T_g$ by Gaussian process regression with a sum kernel of radial-basis-function and white-noise terms, so the prediction for a new polymer is a weighted average of similar training polymers and comes with an uncertainty estimate. Retraining after data expansion changes the kernel's similarity landscape, giving accurate predictions in regions where uncertainty was previously large.","core_discovery":"The central claim is that the earlier model, trained on 450 $T_g$ values, generalizes poorly to polymers outside its training domain: its root-mean-square error on the 871 newly collected polymers exceeds 50 K. Augmenting the training set with those 871 polymers and retraining the same Gaussian-process regression pipeline yields a model whose RMSE on the full 1,321-polymer dataset is 27 K, down from 57 K, with $R^2$ rising from 0.71 to 0.92 and visibly smaller uncertainty bands. The authors attribute the improvement to the new data filling chemical spaces the old model had not seen, especially the high-$T_g$ region of roughly 500–700 K, and present the retrained model as more versatile and transferable.","pith_inferences":["Editorial inference: because the paper evaluates the retrained model on the same 1,321 polymers used to train it, the 27 K figure is a training-fit error, not a measured generalization error; a held-out test would likely show a larger RMSE.","Editorial inference: the pattern of large errors and high uncertainty for chemically distant polymers indicates the model is an interpolation tool, so predictions for chemistry outside the expanded space remain untrustworthy.","Editorial inference: the success of adding data in sparse regions suggests active learning—selecting the next polymers to measure by their current prediction uncertainty—could achieve similar gains with fewer experiments.","Editorial inference: if other property models in the platform had similarly sparse training regions, comparable data expansion would likely yield similar accuracy improvements."],"forward_implications":["The model is now accurate to roughly the scatter of experimental $T_g$ measurements, making it useful for screening candidate polymers before synthesis.","The same data-infusion workflow can be applied to the platform's other property models, such as band gap, dielectric constant, and tensile strength, provided comparable experimental data are collected.","The domain of applicability explicitly expands beyond the original 450 polymers; predictions for high-$T_g$ polymers in the 500–700 K range are now backed by training data.","Future data additions should prioritize diverse chemistries, since poor predictions in this study coincided with high Gaussian-process uncertainty.","A retrained model with calibrated uncertainties enables inverse design, such as searching for polymers whose predicted $T_g$ falls in a target window."],"supporting_citations":[{"why":"Supplies the hierarchical fingerprinting scheme and Gaussian-process regression workflow that the new model is built on.","marker":"(13)"},{"why":"One of the data sources for the 871 newly collected experimental glass-transition values.","marker":"(9)"},{"why":"Additional compilation of experimental polymer property data used to extend the $T_g$ dataset.","marker":"(10)"},{"why":"Online polymer property repository from which experimental $T_g$ values were collected.","marker":"(11)"},{"why":"Provides the SMILES line notation used to encode each polymer repeat unit for fingerprinting.","marker":"(12)"},{"why":"Foundational description of Gaussian processes for regression, the algorithm behind the predictive model.","marker":"(5)"},{"why":"Source of the quantitative structure-property relationship descriptors used as the second fingerprint level.","marker":"(14)"}],"fun_headline_variants":["871 new polymers halve Tg prediction error","Polymer Tg error drops to 27 K with more data","Retrained ML model hits 27 K on glass transition","Adding 871 polymers cuts Tg error to 27 K","New data improves polymer Tg predictions to 27 K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 27 K error is measured on the same 1,321 polymers used to train the new model, with no held-out test set, so the improvement assumes that the training fit reflects how well the model will predict truly new polymers.","fun_headline_variants_meta":{"raw":{"variants":["871 new polymers halve Tg prediction error","Polymer Tg error drops to 27 K with more data","Retrained ML model hits 27 K on glass transition","Adding 871 polymers cuts Tg error to 27 K","New data improves polymer Tg predictions to 27 K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1840,"prompt_tokens":919,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":843}},"tokens_in":535,"tokens_out":921,"duration_ms":10317,"temperature":1.0,"reasoning_tokens":843,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:44:55.173112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the model on a random 80% of the 1,321 polymers, predict the remaining 20%, and compute the RMSE; if the held-out RMSE is much larger than 27 K, for instance 50 K or more, then the reported improvement overstates what the model will do on unseen polymers.","supporting_citations":[],"review_version":1}