REVIEW 5 major objections 6 minor 8 references
Optimization of Functional Materials Design with Optimal Initial Data in Surrogate-Based Active Learning
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Adequate initial data is decisive for convergence in surrogate-based active learning, with recommended starting sizes from 25 points for 40-bit systems to 3,000 for 160-bit systems.
desk verdict Qualitative message is solid; the specific recommended initial-data sizes rest on a post hoc tuned convergence threshold and lack repeated-run statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the factorization-machine surrogate: a supervised model whose trained parameters ($w_0$, $w_i$, $v_{i,f}$) plug directly into a quadratic unconstrained binary optimization (QUBO) matrix, so each active-learning cycle proposes a candidate structure by solving a QUBO. To detect when optimization actually converges, the paper uses averaged piecewise linear regression: five piecewise regressions with 20 pieces each and shifted ranges, averaged to fit the non-smooth FOM-versus-cycle distribution, with convergence declared when the regression gradient first drops below a threshold. That regression threshold is the tool that converts scattered FOM data into the recommended initial-data sizes.
What would settle it
Run the 120-bit transparent radiative cooling optimization starting with 500 initial data (below the recommended 1,000) and track both the regression-gradient crossing and the actual minimum FOM per cycle; if the minimum FOM keeps decreasing substantially for hundreds of cycles after the gradient crosses -3, the threshold is not tracking real convergence.
Extended reading notes
Core claim
The central claim is that convergence of surrogate-based active learning depends strongly on the number of initial data points, and that a threshold-based analysis of FOM curves can identify how many are needed. Using a factorization-machine surrogate mapped to a QUBO and solved by simulated annealing, the authors optimize multilayered transparent radiative cooling windows over seven design-space sizes. They fit averaged piecewise linear regressions to FOM-versus-cycle scatter and define convergence onset as the first cycle where the regression gradient crosses -3 (or -2 for the 160-bit system). Under this criterion, 40- and 60-bit systems converge within 500 iterations even with 25 initial data, while 80-, 100-, 120-, 140-, and 160-bit systems respectively demand 100, 200, 1,000, 2,000, and 3,000 initial data for the same guarantee. The authors conclude that adequate initial data is essential for reliable, cost-effective optimization of large functional-material design spaces.
Load-bearing premise
The whole recommendation stands on the assumption that the first time the averaged regression slope crosses -3 (or -2 for the 160-bit problem) marks the point where real optimization improvement begins, rather than an artifact of the fitting method.
Editorial extensions
If this is right
- Practitioners should stop using a fixed initial data size such as 25 or 50 for all design spaces; the paper's results imply the starting dataset must grow with the binary dimension of the design space.
- For the TRC window problem, systems up to 60 bits converge within 500 cycles with only 25 initial data, so small design spaces can keep the cheap start.
- For large systems (120 bits and above), the required initial data jumps into the thousands, suggesting that earlier studies using small fixed initial data may have systematically underestimated achievable FOMs or underreported convergence failures.
- A transparent radiative cooling window optimized under this scheme saves up to about 34% of cooling energy relative to conventional glass windows, with larger savings in tropical climates.
Reading between the lines
- The recommended sizes likely transfer only to FM/QUBO surrogate pipelines with similar FOM landscapes; other surrogates such as deep networks or Gaussian processes may need a different scaling rule, and this is a testable hypothesis the paper does not address.
- The quoted initial-data sizes imply an empirical scaling law: required initial data grows much faster than linearly with bit count, from 25 points at 40 bits to 3,000 at 160 bits; if that trend holds, 200-bit systems would need thousands more before optimization can start.
- The -3/-2 gradient threshold could be replaced by a statistical control chart on rolling FOM improvements, which would give an online stopping rule rather than a post-hoc regression analysis; such a rule would make the recommendations actionable during optimization.
- Because the authors used simulated annealing instead of a real quantum annealer, the exact initial-data sizes could shift when the QUBO solver's solution quality changes; this is an untested extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how the number of initial data points affects convergence in surrogate-based active learning with factorization-machine surrogates and QUBO solvers (here implemented with simulated annealing) for designing multilayer transparent radiative cooling windows. It fits FOM-versus-optimization-cycle trajectories with an averaged piecewise linear regression, defines the initiation point of convergence as the first cycle at which the regression gradient crosses -3 (or -2 for the 160-bit system), and recommends initial data sizes of 25, 100, 200, 1,000, 2,000, and 3,000 for 40-, 80-, 100-, 120-, 140-, and 160-bit systems, respectively, to achieve convergence within 500 iterations. The paper also demonstrates a 60-bit optimized TRC window and estimates cooling-energy savings of up to about 34% relative to conventional glass windows.
Significance. If the quantitative recommendations held, the work would provide practical guidance for choosing initial data sizes in FM-based active learning and would be a useful reference for quantum-annealing-assisted optimization workflows. The qualitative finding that larger design spaces require more initial data, and that too little initial data makes early active-learning acquisitions resemble random sampling, is clearly supported by the displayed trajectories and by Figure 4. The paper also ships a concrete application to TRC windows with energy simulations, which is a strength. However, the central quantitative claim, the specific optimal initial-data sizes, rests on a convergence detector whose threshold is tuned post hoc for one system and for which no uncertainty quantification or validation against direct optimization progress is provided; this is the main weakness and limits the significance of the reported numbers.
major comments (5)
- [§3.1, §4.2] The convergence criterion is not independently validated. Section 3.1 states that convergence starts when the gradient of the regression line first reaches -3, but no independent measure of optimization progress is used to justify this specific value. Section 4.2 then changes the threshold to -2 for the 160-bit system because the -3 threshold gives an initiation point of 909, which is called 'overly underestimated.' Since the recommended initial-data sizes are defined as the smallest counts whose initiation point falls within 500 iterations, the recommendations are directly determined by this threshold choice. A different threshold (for example -1 or -5) would shift the crossing and likely change the recommended sizes. The authors should show that the regression-gradient crossing tracks actual optimization progress rather than smoothing artifacts, and they should report sensitivity of the recommended sizes to the threshold value.
- [§4.2, Fig. 4] The reported initiation points and recommended initial-data sizes appear to come from single optimization trajectories. Both the random initial dataset and the simulated annealer are stochastic, so the point values in Figure 4 may not be reproducible. The paper provides no error bars, repeated-seed statistics, or confidence intervals for any initiation point. Because the quantitative recommendations are the central claim, the authors should present distributions over multiple runs or otherwise demonstrate that the reported counts are stable.
- [§3.1] The averaged piecewise linear regression procedure is under-specified. The paper states that five piecewise regressions with 'different ranges,' each with 20 pieces, are averaged, but it does not define how the ranges are chosen, how the breakpoints are optimized, how the averaging is performed, or why 20 pieces and five regressions are selected. Since the gradient profile is the sole basis for the initiation-point estimate, this missing specification prevents reproduction and makes it impossible to assess robustness of the gradient values to the regression configuration.
- [§4.2] The 500-iteration horizon is load-bearing but not justified. The recommended initial-data sizes are the smallest tested values whose initiation point falls within 500 iterations, yet the paper does not explain why 500 cycles is the relevant horizon. A change in the horizon would change which initial-data counts qualify as satisfactory, and the trade-off between the cost of additional initial data and the cost of additional optimization cycles is not quantified. The authors should report the sensitivity of their recommendations to the horizon choice, for example 250, 500, 1,000, and 2,000 iterations.
- [§4.3] The paper concedes that the reported FOM may not be a global minimum and that 'achieving a global optimal structure may require additional optimization cycles.' This is a fair limitation, but it underscores that 'convergence' as used here means stabilization of the algorithm's own trajectory under a heuristic criterion, not verified convergence to the optimum. The quantitative guidance should be framed in this more limited way and should be validated against at least one case with a known or independently computed optimum before being presented as optimal initial data sizes.
minor comments (6)
- [§1] The reference 'Kiati et. al.' should be 'Kitai et al.'
- [§4.1] The phrase 'we adapt the averaged piecewise linear regression technique' should read 'we adopt' unless the authors mean that the technique is adapted, in which case the adaptation should be described.
- [§2.2, Eqs. (2)-(3)] The notation νy is used both for the QUBO objective and for the optimized output value, which is confusing; please distinguish the objective function from the optimized state νx.
- [§3.1] The comparison of polynomial regression degrees is anecdotal, based on a single 120-bit case; a systematic comparison with a quantitative fit-quality metric would better justify the choice of averaged piecewise linear regression.
- [General] No data or code availability statement is provided. Since the quantitative results depend on a custom regression procedure, making the code and data available would materially improve reproducibility.
- [Fig. 4 caption] The caption refers to a 'predefined threshold (-3)' and then notes that the 160-bit system uses -2; this is internally inconsistent with Section 4.2, where -2 is introduced as a post hoc adjustment.
Circularity Check
Recommended initial-data sizes are read off from the paper's own gradient-threshold convergence detector, whose threshold is adjusted post hoc for the 160-bit system; the quantitative recommendations are therefore fitted to the criterion rather than independently measured.
-
fitted input called prediction
[Section 4.2 (Optimal Number of Initial Data) and Figure 5 note; threshold adjustment introduced in same section.]
"Hence, the initiation point for convergence is 909 if the threshold is −3, which is overly underestimated (Figure 5D). Adjusting the threshold from −3.0 to −2.0 yields a more accurate determination of the initiation point, which is aligned with the observed trends in smaller systems (i.e., 40 to 140-bit systems, Figures 4 and 6). Therefore, it is the more proper strategy to determine the initiation point with smaller absolute threshold values for a large system."
The recommended initial-data size for the 160-bit system (3,000) is justified by claiming that with 3,000 initial data 'FOM converges well,' but the quantitative support is the initiation point from the regression-gradient criterion. At the originally preset threshold, that criterion gives an initiation point of 909, which does not satisfy the 'satisfactory convergence within 500 iterations' condition used for the smaller systems. The threshold is then lowered to −2 precisely so that the initiation point aligns with the desired conclusion. Thus the quantitative recommendation for the largest system is not derived from an independent convergence measure; the free threshold parameter is tuned post hoc to make the detector fire in the intended place.
-
self definitional
[Section 3.1 (Determining Convergence) and Section 4.2 (Optimal Number of Initial Data).]
"We decide that convergence starts when the gradient of -3 in the regression line is first observed. ... For example, 80, 100, 120, and 140-bit systems respectively need 100, 200, 1,000, and 2,000 initial data to ensure satisfactory convergence within 500 iterations."
The central recommended initial-data sizes are defined as the smallest tested counts for which the paper's own threshold-based detector fires before 500 cycles. 'Convergence' is operationalized entirely by the first crossing of a chosen gradient value in an averaged piecewise linear regression, with no external benchmark such as distance to a known optimum, target FOM, or out-of-sample validation. The threshold, the regression form, the number of pieces, the averaging scheme, and the 500-cycle horizon are all internal modeling choices. Consequently, the specific quantitative recommendations (25, 100, 200, 1,000, 2,000, 3,000) reduce by construction to the chosen operational criterion: changing the threshold or horizon changes which initial-data sizes qualify.
full rationale
The paper's qualitative finding that larger design spaces need more initial data is supported by the FOM trajectories and Figure 4, and the energy-saving demonstration for the optimized TRC window is an independent application. However, the paper's central quantitative contribution is the set of recommended initial-data sizes, which are read off from a heuristic convergence criterion. That criterion is a gradient threshold on an averaged piecewise linear regression, and for the 160-bit system the threshold is explicitly changed from −3 to −2 because the original threshold gives an initiation point that contradicts the desired narrative. This is a fitted input renamed as a prediction: the recommended initial-data size for the largest system depends on a parameter that was adjusted to make the result align with observed trends. The paper also defines convergence internally via the 500-iteration horizon and the regression-gradient crossing, so the recommended sizes are partly self-definitional: they are the smallest counts that satisfy the paper's own detector, not values validated against an external convergence benchmark. Self-citations to prior Kim et al. work establish the FM-QUBO framework and are not load-bearing for this central claim, and the paper itself concedes that the optimized FOM may not be a global minimum. Because the qualitative trend has independent content while the quantitative recommendations reduce to the chosen thresholds and horizon, a partial circularity score of 6 is appropriate.
Assumptions & free parameters
free parameters (4)
- convergence gradient threshold =
-3 (default), -2 (160-bit system)
- convergence horizon =
500 iterations
- piecewise regression configuration =
20 pieces, 5 averaged regressions
- FM latent space size =
k = 4
assumptions (4)
- domain assumption Simulated annealing gives results similar to quantum computing for QUBOs of sizes 40 to 160 bits.
- domain assumption FM can be translated to QUBO without information loss.
- ad hoc to paper The gradient-threshold criterion identifies where convergence begins.
- domain assumption The FOM and transfer-matrix optical simulation correctly capture TRC window performance.
Cite this review
Pith. "Pith review of Optimization of Functional Materials Design with Optimal Initial Data in Surrogate-Based Active Learning." pith.science (2026). https://pith.science/paper/CNKURN3J
@misc{pith2026250603329,
author = {Pith},
title = {Pith review of: Optimization of Functional Materials Design with Optimal Initial Data in Surrogate-Based Active Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/CNKURN3J}},
note = {Machine review of arXiv:2506.03329}
}
abstract
The optimization of functional materials is important to enhance their properties, but their complex geometries pose great challenges to optimization. Data-driven algorithms efficiently navigate such complex design spaces by learning relationships between material structures and performance metrics to discover high-performance functional materials. Surrogate-based active learning, continually improving its surrogate model by iteratively including high-quality data points, has emerged as a cost-effective data-driven approach. Furthermore, it can be coupled with quantum computing to enhance optimization processes, especially when paired with a special form of surrogate model ($i.e.$, quadratic unconstrained binary optimization), formulated by factorization machine. However, current practices often overlook the variability in design space sizes when determining the initial data size for optimization. In this work, we investigate the optimal initial data sizes required for efficient convergence across various design space sizes. By employing averaged piecewise linear regression, we identify initiation points where convergence begins, highlighting the crucial role of employing adequate initial data in achieving efficient optimization. These results contribute to the efficient optimization of functional materials by ensuring faster convergence and reducing computational costs in surrogate-based active learning.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Introduction 29 The optimal design of functional materials has become increasingly essential for enhan-30 cing their properties Liu et al., 2020; Molesky et al., 2018; Zunger, 2018. However,31 1 arXiv:2506.03329v1 [cs.CE] 3 Jun 2025 their inherently complex geometrical features significantly expand the design spaces, pos-32 ing challenges to optimization ...
work page Pith review arXiv 2020
-
[2]
Theactivelearningalgorithm99 comprises three key components S
Background 96 2.1 Surrogate-Based Active Learning97 Figure 1A illustrates a workflow of the surrogate-based active learning algorithm designed98 tooptimizefunctionalmaterialsthroughiterativeprocesses. Theactivelearningalgorithm99 comprises three key components S. Kim, Jung et al., 2024; S. Kim, Park et al., 2024; S.100 Kim et al., 2022, 2023; Kitai et al....
work page 2024
-
[3]
Method 183 3.1 Determining Convergence184 FOM tends to decrease as the optimization cycle progresses when the active learning185 algorithm works well because this case (optimization of TRC windows) is designed for186 a minimization optimization problem (Figure 1B,C,D). To quantitatively analyze the187 decreasing trend of the FOM with respect to optimizati...
work page 2019
-
[4]
Experiments 221 4.1 FOM Convergence Analysis222 We analyze FOM convergence patterns after optimization with different initial data sizes223 for various design space sizes. FOM convergence can be achieved with only a few optim-224 ization cycles when starting with 25 initial data for a small design space, such as a 40-bit225 system (Figure 1B). However, co...
work page 2019
-
[5]
Conclusion 321 In this work, we studied finding optimal numbers of initial data according to design space322 sizes to achieve reliable and efficient convergence in surrogate-based active learning. We323 adopted averaged piecewise linear regression to fit data by effectively modeling complex324 data distributions, and then we determined the initiation poin...
-
[6]
Acknowledgments 337 This research used resources of the Oak Ridge Leadership Computing Facility at the338 Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S.339 Department of Energy under Contract No. DE-AC05-00OR22725. This material is based340 upon work supported by the U.S. Department of Energy, Office of Science, Nat...
arXiv 2020
-
[450]
O., Mishra, S., Ray, D., & Chandrashekar, P
411 Lye, K. O., Mishra, S., Ray, D., & Chandrashekar, P. (2021). Iterative surrogate model412 optimization (ismo): An active learning algorithm for pde constrained optimiz-413 ation with deep neural networks.Computer Methods in Applied Mechanics and414 Engineering, 374, 113575. 415 Ma, W., Liu, Z., Kudyshev, Z. A., Boltasseva, A., Cai, W., & Liu, Y. (2021...
work page 2021
-
[2023]
142 Our goal in this study is to observe convergence patterns of FOMs, which are mainly143 determined by FM. Therefore, to mitigate costs associated with quantum computing, we144 utilize SA (D-Wave sampler) as a QUBO solver for given surrogates (i.e., QUBOs), which145 may yield similar optimization results to those with quantum computing, especially in146...
work page 2019
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.