Assessing and Improving Machine Learning Model Predictions of Polymer Glass Transition Temperatures

Chiho Kim; Manav Ramprasad

REVIEW 3 major objections 4 minor 5 references

Assessing and Improving Machine Learning Model Predictions of Polymer Glass Transition Temperatures

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding 871 polymers to a polymer informatics model's training data cuts glass-transition temperature prediction error from 57 K to 27 K.

desk verdict The 27 K RMSE for PG-1 is an in-sample training error, not an out-of-sample prediction, so the paper's headline improvement over PG-0's 57 K is not established. read the letter →

arxiv 1908.02398 v1 pith:5C6CH4LS submitted 2019-08-06 physics.comp-ph cond-mat.mtrl-sci

Manav Ramprasad , Chiho Kim This is my paper · ORCID

classification physics.comp-phcond-mat.mtrl-sci

keywords machinelearningglasstransitiontemperatureGaussianprocessregressionpolymerfingerprintingpropertypredictionmaterialsdatatransferability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to show that expanding a machine-learning model's training data from 450 to 1,321 polymers substantially improves its predictions of the glass transition temperature ($T_g$), the temperature above which a polymer turns from glass-like to rubber-like. The authors collected 871 new experimental $T_g$ values, showed that the old model failed on them (root-mean-square error above 50 K), then retrained the same Gaussian-process model on the combined data and report the error dropping from 57 K to 27 K. If this holds, the model becomes accurate enough to be a practical screening tool across a much more chemically diverse polymer space, and the same data-collection-and-retraining recipe could improve other property predictions.

What carries the argument

The load-bearing mechanism is the hierarchical polymer fingerprint: each repeat unit is encoded as a vector of 128 atomic-fragment counts, 39 quantitative structure-property relationship descriptors (such as estimated surface area and fraction of rotatable bonds), and 22 morphological features (such as aromatic-ring topology and side-chain lengths). This fingerprint is mapped to $T_g$ by Gaussian process regression with a sum kernel of radial-basis-function and white-noise terms, so the prediction for a new polymer is a weighted average of similar training polymers and comes with an uncertainty estimate. Retraining after data expansion changes the kernel's similarity landscape, giving accurate predictions in regions where uncertainty was previously large.

What would settle it

Retrain the model on a random 80% of the 1,321 polymers, predict the remaining 20%, and compute the RMSE; if the held-out RMSE is much larger than 27 K, for instance 50 K or more, then the reported improvement overstates what the model will do on unseen polymers.

Watch

Extended reading notes

Core claim

The central claim is that the earlier model, trained on 450 $T_g$ values, generalizes poorly to polymers outside its training domain: its root-mean-square error on the 871 newly collected polymers exceeds 50 K. Augmenting the training set with those 871 polymers and retraining the same Gaussian-process regression pipeline yields a model whose RMSE on the full 1,321-polymer dataset is 27 K, down from 57 K, with $R^2$ rising from 0.71 to 0.92 and visibly smaller uncertainty bands. The authors attribute the improvement to the new data filling chemical spaces the old model had not seen, especially the high-$T_g$ region of roughly 500–700 K, and present the retrained model as more versatile and transferable.

Load-bearing premise

The 27 K error is measured on the same 1,321 polymers used to train the new model, with no held-out test set, so the improvement assumes that the training fit reflects how well the model will predict truly new polymers.

Editorial extensions

If this is right

The model is now accurate to roughly the scatter of experimental $T_g$ measurements, making it useful for screening candidate polymers before synthesis.
The same data-infusion workflow can be applied to the platform's other property models, such as band gap, dielectric constant, and tensile strength, provided comparable experimental data are collected.
The domain of applicability explicitly expands beyond the original 450 polymers; predictions for high-$T_g$ polymers in the 500–700 K range are now backed by training data.
Future data additions should prioritize diverse chemistries, since poor predictions in this study coincided with high Gaussian-process uncertainty.
A retrained model with calibrated uncertainties enables inverse design, such as searching for polymers whose predicted $T_g$ falls in a target window.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the paper evaluates the retrained model on the same 1,321 polymers used to train it, the 27 K figure is a training-fit error, not a measured generalization error; a held-out test would likely show a larger RMSE.
Editorial inference: the pattern of large errors and high uncertainty for chemically distant polymers indicates the model is an interpolation tool, so predictions for chemistry outside the expanded space remain untrustworthy.
Editorial inference: the success of adding data in sparse regions suggests active learning—selecting the next polymers to measure by their current prediction uncertainty—could achieve similar gains with fewer experiments.
Editorial inference: if other property models in the platform had similarly sparse training regions, comparable data expansion would likely yield similar accuracy improvements.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The headline number in this paper—27 K RMSE for the retrained PG-1 model—is an in-sample training error. PG-1 was trained on all 1,321 polymers and then evaluated on that same set. The 57 K figure for PG-0, by contrast, is an out-of-sample error on the 871 newly collected polymers. Comparing the two conflates memorization with generalization. The stress-test note is right, and the paper's central claim falls.

What is actually new: the collection of 871 experimental Tg values from public sources. That is a genuine addition to the field, and the Polymer Genome platform has a track record of making models available. If the dataset is released in a usable form, it is a useful resource for benchmarking future polymer ML models. The paper also makes a reasonable qualitative observation that PG-0, trained on only 450 polymers, predicts poorly for many out-of-domain polymers and that GPR uncertainties are higher for those points. That part is plausible and consistent with the model.

The soft spot is not minor: the evaluation of PG-1 is missing any holdout set or cross-validation. A GPR with RBF and white-noise kernels can drive training RMSE down to measurement noise levels, so the 27 K number tells us almost nothing about predictive accuracy on unseen polymers. The paper does not report a train/test split, validation, or any independent test. In fact, comparing Figure 2(a) and 2(b) is apples-to-oranges for exactly that reason. The authors might well have a decent model—the methodology is standard—but this manuscript does not support the claim.

There is also a small citation oddity: reference 7 is a towardsdatascience blog post, and reference 8 is an unrelated inventory management paper. These look like filler and should be fixed if the paper goes further.

Who is this for? A reader interested in polymer data resources, not in ML methodology. The paper is a brief data report. It deserves a serious referee because the dataset expansion is useful, but the referee should require a proper out-of-sample evaluation (cross-validation or a held-out test) and a clear statement of the training/evaluation split. As is, the core quantitative claim is not established. I would not cite the 27 K number, but I would encourage the authors to release the data and rerun with a sound evaluation.

Recommendation: send to peer review only if the authors fix the evaluation. Otherwise, it is a dataset announcement, not a prediction improvement paper.

Referee Report

3 major / 4 minor

Summary. The paper reports on the collection of 871 additional experimental glass transition temperature (Tg) values for polymers, augmenting an earlier dataset of 450 polymers to form a combined set of 1,321 polymers. The authors evaluate the previous Polymer Genome (PG-0) model on the new data, finding a high out-of-domain RMSE of 57 K, then retrain a Gaussian process regression model (PG-1) on the full 1,321 data points and report that the RMSE 'of prediction' drops to 27 K. The paper argues that this improvement demonstrates the value of data expansion and yields a more versatile predictive model. The main contributions are the new data collection and the qualitative demonstration that PG-0 transfers poorly to out-of-domain polymers.

Significance. The new dataset of 871 experimentally measured Tg values is a valuable community resource, and the demonstration that PG-0's predictions on these new polymers carry an RMSE of 57 K is a useful, falsifiable negative result about the transferability of models trained on a narrow chemical space. If the reported improvement to 27 K RMSE were a genuine out-of-sample prediction error, the result would be significant for polymer informatics, showing that data expansion can substantially improve predictive accuracy. However, the central quantitative claim is undermined by the evaluation protocol: the 27 K figure is computed on the same data used for training and therefore cannot be interpreted as prediction error. The paper's strengths — careful data collection, use of an established fingerprinting scheme, and consistent application of GPR — are real, but they are attached to a headline result that does not support the stated conclusion of improved predictive capability. The manuscript is more of a dataset and benchmark report than a validated methodological advance.

major comments (3)

[Results, Fig. 2(b)] The abstract and Results state that 'The RMSE of prediction for the extended dataset ... decreased to 27 K from 57 K.' This comparison is invalid because PG-1 was trained on the full set of 1,321 polymers and then evaluated on that same full set, whereas the 57 K value for PG-0 was an out-of-sample error computed on the 871 newly collected polymers not present in PG-0's training set. The 27 K figure is therefore a training-set RMSE, not a prediction error. For Gaussian process regression with a radial basis function plus white-noise kernel, the training-set RMSE can be driven toward the noise level even when held-out error is much larger, so this number carries no evidence of generalization. To support the claim of improved predictive capability, the authors must evaluate PG-1 on data not used for training — for example, treat the 871 new polymers as a held-out test set, or perform k-fold cross-validation over the full 1,321 — and report the resulting out-of-sample RMSE alongside the 57 K baseline.
[Methods] No validation protocol is described. The Methods section explains the fingerprinting approach and the GPR kernel but does not state how the kernel hyperparameters were optimized, whether any portion of the 1,321 polymers was held out for testing, or whether cross-validation was used. In the absence of this information, the reported R2=0.92 and RMSE=27 K in Figure 2(b) cannot be interpreted as predictive performance. The manuscript should explicitly describe the train/test split or cross-validation procedure, the hyperparameter optimization scheme, and the metrics computed on held-out data.
[Discussion] The claim that the retrained model 'is hence more versatile' is unsupported by the reported experiments. Versatility is a statement about out-of-sample and out-of-domain generalization, yet the quantitative evidence in the manuscript is an in-sample fit on the training data. Without held-out evaluation, the discussion overstates the conclusion that adding diverse data has led to 'a significantly better predictive capability of PG.'

minor comments (4)

[References] References 7 and 8 do not support the statement about inverse design algorithms: reference 7 is a blog post on variational autoencoders and reference 8 concerns a vendor-managed inventory optimization problem; neither is a representative citation for polymer inverse design. Please cite relevant literature on generative models for materials discovery.
[Figure 2] The axes of the parity plots in Figure 2 are not labeled, and the parity line is not identified. Units (K) should be given on both axes, and the dashed line should be clearly described as the ideal parity line.
[Results] The statement that 'the uncertainties calculated by GPR shown by the yellow error bars have also decreased significantly' is made without quantitative support; the authors should report the average or distribution of predicted uncertainties for the new polymers under PG-0 versus PG-1.
[Results] The manuscript says 'the RMSE ... is greater than 50 K' for PG-0 on the new 871 polymers, but the figure caption reports RMSE=57 K. Please make the value consistent or state whether 57 K is the exact value and 'greater than 50 K' is a rounding description.

Circularity Check

1 steps flagged · score 6.0 of 10

PG-1's 27 K RMSE is an in-sample training error on the same 1,321 polymers used to fit the model, not an independent prediction, so the claimed improvement over PG-0's 57 K out-of-sample error is not established.

fitted input called prediction [Abstract and Results, Figure 2(b) caption]
"Using the full dataset of 1321 polymers, a new ML model for Tg was built following past work. The RMSE of prediction for the extended dataset, when compared to the earlier one, decreased to 27 K from 57 K."

The paper trains PG-1 on the full 1,321-polymer dataset and then reports the 27 K RMSE on that same dataset, calling it a prediction. For a GPR model with an RBF plus white-noise kernel, the posterior mean on training inputs is an interpolation of the training targets, so the training-set RMSE is a measure of fit rather than predictive accuracy. In contrast, the 57 K RMSE for PG-0 is evaluated on the 871 newly collected polymers that were not used to train PG-0. Therefore the headline comparison is between an in-sample training error (27 K) and an out-of-sample test error (57 K), and the improvement is partially forced by construction: the model was fit to the very points on which the error is reported. No held-out subset or cross-validation split is described in Results or Methods.

full rationale

The central circularity is the evaluation protocol for PG-1. The paper uses all 1,321 polymers both to train the new model and to compute the reported 27 K RMSE, then presents this in-sample error as a predictive improvement over PG-0's out-of-sample 57 K error on unseen polymers. For a Gaussian process regressor with an RBF plus white-noise kernel, predictions at training points interpolate the training targets, so a low training RMSE is expected by construction and does not demonstrate better generalization. The fingerprinting and GPR methodology is inherited from the authors' prior work, but that is a normal methodological starting point rather than a self-citation used to force the result; no uniqueness theorem or circular definition is invoked. If the authors had evaluated PG-1 on a held-out portion of the 1,321 polymers, the reported improvement would be a genuine out-of-sample result. As reported, the 27 K RMSE is a fitted-input quantity relabeled as a prediction, so the paper deserves a partial-circularity score rather than a clean bill.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central result relies on data quality, fingerprint adequacy, and the implicit assumption that training-set error measures prediction quality. The last is the most fragile, as the paper's headline RMSE is computed on the training set.

free parameters (1)

GPR kernel hyperparameters (RBF length scales and white-noise variance) = not reported
The GPR model with RBF and white-noise kernels requires hyperparameters that are fit to the data; the paper does not list their values or how they were optimized, but the prediction error depends on them.

assumptions (4)

domain assumption Experimental Tg values from the cited handbooks and database are accurate and self-consistent.
The model is trained and evaluated on these literature values; any measurement error or inconsistency propagates into the reported RMSE. This is standard for materials informatics but is an unverified input.
domain assumption The hierarchical polymer fingerprint (128 atomic fragments, 39 QSPR descriptors, 22 morphological features) captures the physical features that determine Tg.
The paper uses this fingerprint from prior work (ref 13) without new validation; if the fingerprint misses important chemical features, the model's predictions are limited regardless of data size.
domain assumption Gaussian process regression with a sum of RBF and white-noise kernels is an appropriate modeling choice for Tg.
The model architecture is taken from previous Polymer Genome work; the paper provides no comparison with other algorithms.
domain assumption The 1,321 polymers are representative of the polymer chemical space of interest.
The paper claims the new data adds diversity, but there is no formal analysis of coverage or bias in collection.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessing and Improving Machine Learning Model Predictions of Polymer Glass Transition Temperatures." pith.science (2026). https://pith.science/paper/5C6CH4LS

@misc{pith2026190802398,
  author       = {Pith},
  title        = {Pith review of: Assessing and Improving Machine Learning Model Predictions of Polymer Glass Transition Temperatures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5C6CH4LS}},
  note         = {Machine review of arXiv:1908.02398}
}

read the original abstract

The success of the Materials Genome Initiative has led to opportunities for data-driven approaches for materials discovery. The recent development of Polymer Genome (PG), which is a machine learning (ML) based data-driven informatics platform for polymer property prediction, has significantly increased the efficiency of polymer design. Nevertheless, continuous expansion of the "training data" is necessary to improve the robustness, versatility and accuracy of the ML predictions. In order to test the performance and transferability of the predictive models presently available in PG (which were previously trained on a dataset of 450 polymers), we have carefully collected additional experimental glass transition temperature (Tg) data for 871 polymers from multiple data sources. The Tg values predicted by the present PG models for the polymers in the newly collected dataset were compared directly with the experimental Tg to estimate the accuracy of the present model. Using the full dataset of 1321 polymers, a new ML model for Tg was built following past work. The RMSE of prediction for the extended dataset, when compared to the earlier one, decreased to 27 K from 57 K. To further improve the performance of the Tg prediction model, we are continuing to accumulate new data and exploring new ML approaches.

Figures

Figures reproduced from arXiv: 1908.02398 by the authors.

**Figure 1.** The key elements of machine learning in materials science. a. Schematic view of an example data set. b. Statement of the problem “What is the Tg of new polymer?” c. Creation of a prediction model via the fingerprinting and learning steps. The efficacy of this philosophy has been recently demonstrated as part of the “Polymer Genome” (PG) Project (6). In order to improve upon the predictive capabilities of the ML mode… view at source ↗

**Figure 3.** (a) Distribution of the Tg values for all the polymers considered in this work. (b) Sample polymer dataset with SMILES representation and experimental Tg values. In order to capture the key features that may control the Tg, we utilized the hierarchical polymer fingerprinting scheme (13). The fingerprint building process consists of three hierarchical levels of features. The first one is at the atomic scale wherein t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 5 canonical work pages

[1]

Polystyrene X11, X12, X13,…, X1M 373

work page
[2]

Poly(propylene) X21, X22, X23,…, X2M 264

work page
[3]

Polyethyleneoxide206 … … Fingerprinting, learning and prediction Polymer FingerprintTg(K)

work page
[4]

Properties of Polymers: Their Correlation with Chemical Structure

Polyethyleneoxide X31, X32, X33,…, X3M 206 … … FingerprintingLearningNew case Instant property predictionML modelf(Xi1, Xi2, Xi3, …, XiM) = Tg(i)PolymerTg(K) New polymer? (a) (b) (c) Results Although efficient, ML models are accurate and reliable only within the domain of the dataset the model was trained on. Predictions made for cases that fall outside t...

work page 1997
[7]

Intuitively Understanding Variational Autoencoders

“Intuitively Understanding Variational Autoencoders” (2018, Feb 4) https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf 8. Sadeghi, J., Sadeghi, S., and Niaki, Seyed Taghi A. "Optimizing a hybrid vendor-managed inventory and transportation problem with fuzzy demand: An improved particle swarm optimization algorith...

work page doi:10.1016/j.ins.2014.02.075 2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Polystyrene X11, X12, X13,…, X1M 373

work page

[2] [2]

Poly(propylene) X21, X22, X23,…, X2M 264

work page

[3] [3]

Polyethyleneoxide206 … … Fingerprinting, learning and prediction Polymer FingerprintTg(K)

work page

[4] [4]

Properties of Polymers: Their Correlation with Chemical Structure

Polyethyleneoxide X31, X32, X33,…, X3M 206 … … FingerprintingLearningNew case Instant property predictionML modelf(Xi1, Xi2, Xi3, …, XiM) = Tg(i)PolymerTg(K) New polymer? (a) (b) (c) Results Although efficient, ML models are accurate and reliable only within the domain of the dataset the model was trained on. Predictions made for cases that fall outside t...

work page 1997

[5] [7]

Intuitively Understanding Variational Autoencoders

“Intuitively Understanding Variational Autoencoders” (2018, Feb 4) https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf 8. Sadeghi, J., Sadeghi, S., and Niaki, Seyed Taghi A. "Optimizing a hybrid vendor-managed inventory and transportation problem with fuzzy demand: An improved particle swarm optimization algorith...

work page doi:10.1016/j.ins.2014.02.075 2018