{"id":"ae022323-f04a-4664-97d3-2018462d6d42","arxiv_id":"2506.03329","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For factorization machine based active learning, larger design spaces require substantially larger initial datasets to converge quickly, with recommended sizes spanning 25 points for a 40-bit space to 3,000 for a 160-bit space.","lead":"This paper shows that surrogate-based active learning for material design needs many more starting data points as the design space grows, from 25 initial points for small 40-bit problems to 3,000 for large 160-bit problems. The finding offers a practical starting-point guideline for researchers using factorization machine surrogates and quantum-annealing-based optimization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recommended initial-data sizes rest on a post hoc adjusted regression-gradient threshold; without a validated convergence criterion or repeated-run statistics, the quantitative claim is not supported.","rationale":"The reader's weakest assumption identifies the same load-bearing premise: the gradient-threshold criterion for convergence is ad hoc and post hoc adjusted, and the 500-iteration horizon is a second fragile choice. My analysis agrees with this, so the reader's CONDITIONAL verdict is appropriate. I marked 'partial' rather than 'agree' because I place the emphasis more heavily on the threshold adjustment itself: the 160-bit threshold change from -3 to -2 is explicitly post hoc within the paper, and this directly invalidates the quantitative precision of the recommended initial-data sizes. The paper does provide some independent support for the qualitative finding: the FOM trajectories in Figures 1 and 5 show visually that insufficient initial data delays progress, and the energy-saving calculation in Section 3.2 is a concrete downstream application. Yet none of this establishes that the specific numbers (25, 100, 200, 1,000, 2,000, 3,000) are optimal. The test I propose would settle whether the reported numbers survive an independent convergence definition. Since the concern is about precision rather than fundamental correctness of the qualitative trend, keeping the verdict at CONDITIONAL (rather than REJECT) is appropriate, and UNCHANGED reflects that my read does not change the existing verdict.","tokens_in":10672,"tokens_out":2489,"duration_ms":30622,"concrete_test":"For each (design-space size, initial-data count) cell reported in Figure 4, run 10 independent active-learning trajectories with different random seeds. Define convergence independently of the regression threshold: record the cycle at which the running minimum FOM improves by less than 1% over the following 200 cycles. Determine the smallest initial-data count that achieves this alternative convergence definition within 500 cycles for each size. If the resulting recommended counts differ from the paper's list (e.g., by more than one bin), then the reported sizes are threshold artifacts rather than robust properties of the optimization. Alternatively, recompute Figure 4 with gradient thresholds of -1, -2, -4, and -5 and report how the recommended initial-data sizes shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the recommended initial-data sizes (25, 100, 200, 1,000, 2,000, 3,000 for 40, 80, 100, 120, 140, 160-bit systems), derived from the first cycle where the gradient of an averaged piecewise linear regression crosses -3 (Section 3.1). For the 160-bit system, the threshold is changed to -2 because the -3 threshold gives an initiation point of 909, which is described as 'overly underestimated' and the adjustment is justified only by alignment with observed trends (Section 4.2). This makes the threshold a free parameter tuned post hoc to produce the desired narrative. Since the recommended initial-data size is defined as the smallest count whose initiation point falls within 500 iterations, any shift in the threshold changes which count qualifies. A threshold of -1 would mark convergence earlier and could make smaller initial datasets appear sufficient; a threshold of -5 would delay the crossing and could demand larger datasets. The number of pieces in the piecewise regression and the averaging procedure also affect gradient values. No error bars or repeated-seed statistics are reported for any initiation point, so the specific quantitative recommendations are not robust. The qualitative direction—larger design spaces require more initial data—is supported by the trajectories and Figure 4, and the paper itself notes the global optimum may not be reached (Section 4.3), so the concern is not that the qualitative finding is false, but that the headline numbers are an artifact of an arbitrary and post hoc adjusted convergence criterion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the number of initial data points affects convergence in surrogate-based active learning with factorization-machine surrogates and QUBO solvers (here implemented with simulated annealing) for designing multilayer transparent radiative cooling windows. It fits FOM-versus-optimization-cycle trajectories with an averaged piecewise linear regression, defines the initiation point of convergence as the first cycle at which the regression gradient crosses -3 (or -2 for the 160-bit system), and recommends initial data sizes of 25, 100, 200, 1,000, 2,000, and 3,000 for 40-, 80-, 100-, 120-, 140-, and 160-bit systems, respectively, to achieve convergence within 500 iterations. The paper also demonstrates a 60-bit optimized TRC window and estimates cooling-energy savings of up to about 34% relative to conventional glass windows.","tokens_in":10976,"tokens_out":5054,"duration_ms":58440,"significance":"If the quantitative recommendations held, the work would provide practical guidance for choosing initial data sizes in FM-based active learning and would be a useful reference for quantum-annealing-assisted optimization workflows. The qualitative finding that larger design spaces require more initial data, and that too little initial data makes early active-learning acquisitions resemble random sampling, is clearly supported by the displayed trajectories and by Figure 4. The paper also ships a concrete application to TRC windows with energy simulations, which is a strength. However, the central quantitative claim, the specific optimal initial-data sizes, rests on a convergence detector whose threshold is tuned post hoc for one system and for which no uncertainty quantification or validation against direct optimization progress is provided; this is the main weakness and limits the significance of the reported numbers.","major_comments":[{"comment":"The convergence criterion is not independently validated. Section 3.1 states that convergence starts when the gradient of the regression line first reaches -3, but no independent measure of optimization progress is used to justify this specific value. Section 4.2 then changes the threshold to -2 for the 160-bit system because the -3 threshold gives an initiation point of 909, which is called 'overly underestimated.' Since the recommended initial-data sizes are defined as the smallest counts whose initiation point falls within 500 iterations, the recommendations are directly determined by this threshold choice. A different threshold (for example -1 or -5) would shift the crossing and likely change the recommended sizes. The authors should show that the regression-gradient crossing tracks actual optimization progress rather than smoothing artifacts, and they should report sensitivity of the recommended sizes to the threshold value.","section":"§3.1, §4.2"},{"comment":"The reported initiation points and recommended initial-data sizes appear to come from single optimization trajectories. Both the random initial dataset and the simulated annealer are stochastic, so the point values in Figure 4 may not be reproducible. The paper provides no error bars, repeated-seed statistics, or confidence intervals for any initiation point. Because the quantitative recommendations are the central claim, the authors should present distributions over multiple runs or otherwise demonstrate that the reported counts are stable.","section":"§4.2, Fig. 4"},{"comment":"The averaged piecewise linear regression procedure is under-specified. The paper states that five piecewise regressions with 'different ranges,' each with 20 pieces, are averaged, but it does not define how the ranges are chosen, how the breakpoints are optimized, how the averaging is performed, or why 20 pieces and five regressions are selected. Since the gradient profile is the sole basis for the initiation-point estimate, this missing specification prevents reproduction and makes it impossible to assess robustness of the gradient values to the regression configuration.","section":"§3.1"},{"comment":"The 500-iteration horizon is load-bearing but not justified. The recommended initial-data sizes are the smallest tested values whose initiation point falls within 500 iterations, yet the paper does not explain why 500 cycles is the relevant horizon. A change in the horizon would change which initial-data counts qualify as satisfactory, and the trade-off between the cost of additional initial data and the cost of additional optimization cycles is not quantified. The authors should report the sensitivity of their recommendations to the horizon choice, for example 250, 500, 1,000, and 2,000 iterations.","section":"§4.2"},{"comment":"The paper concedes that the reported FOM may not be a global minimum and that 'achieving a global optimal structure may require additional optimization cycles.' This is a fair limitation, but it underscores that 'convergence' as used here means stabilization of the algorithm's own trajectory under a heuristic criterion, not verified convergence to the optimum. The quantitative guidance should be framed in this more limited way and should be validated against at least one case with a known or independently computed optimum before being presented as optimal initial data sizes.","section":"§4.3"}],"minor_comments":[{"comment":"The reference 'Kiati et. al.' should be 'Kitai et al.'","section":"§1"},{"comment":"The phrase 'we adapt the averaged piecewise linear regression technique' should read 'we adopt' unless the authors mean that the technique is adapted, in which case the adaptation should be described.","section":"§4.1"},{"comment":"The notation νy is used both for the QUBO objective and for the optimized output value, which is confusing; please distinguish the objective function from the optimized state νx.","section":"§2.2, Eqs. (2)-(3)"},{"comment":"The comparison of polynomial regression degrees is anecdotal, based on a single 120-bit case; a systematic comparison with a quantitative fit-quality metric would better justify the choice of averaged piecewise linear regression.","section":"§3.1"},{"comment":"No data or code availability statement is provided. Since the quantitative results depend on a custom regression procedure, making the code and data available would materially improve reproducibility.","section":"General"},{"comment":"The caption refers to a 'predefined threshold (-3)' and then notes that the 160-bit system uses -2; this is internally inconsistent with Section 4.2, where -2 is introduced as a post hoc adjustment.","section":"Fig. 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's core quantitative message is narrower than the title suggests: it is a heuristic recommendation for one material class, one surrogate family, and one solver. The qualitative trend is useful and the application is concrete, so I would not recommend rejection. However, the specific initial-data numbers are not supported without a validated convergence criterion, repeated-run statistics, and sensitivity analysis. If the authors cannot provide such evidence, the paper should be reframed as a qualitative study rather than a source of optimal initial-data sizes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the qualitative finding—larger design spaces need more initial data in FM-based active learning—is probably right and clearly demonstrated. The quantitative recommendations (25, 100, 200, 1000, 2000, 3000 for 40–160 bits) are not supported as stated, because they depend on a convergence criterion whose threshold is tuned per system and never validated against actual optimization progress.\n\nWhat's actually new and good: the paper systematically maps initial-data size vs. design-space size for a specific workflow (FM+QUBO surrogate), where prior work just used fixed 25 or 50. The averaged piecewise linear regression is a reasonable heuristic for noisy FOM-vs-cycle data and beats the polynomial alternatives they tried. They also use a physically meaningful case study (TRC windows) and even run EnergyPlus to show the designed window saves energy. The authors are honest that the 160-bit threshold was adjusted post hoc (Section 4.2) and that the global optimum may not be reached (Section 4.3). Those admissions are to their credit.\n\nThe soft spots are concentrated in the headline claim. The set of 'optimal initial data sizes' is basically the smallest counts for which their gradient-threshold detector fires within 500 iterations. The threshold -3, relaxed to -2 for the 160-bit system, is arbitrary; a different threshold would change the recommended sizes. No repeated seeds or error bars are reported, so I can't tell whether the differences between, say, 100 and 200 initial data are real. The 500-iteration horizon is also unexplained. The circularity is real: the result defines 'optimal' as 'what makes our heuristic fire.' That does not make the qualitative trend wrong, but it means the paper should be read as a practical rule of thumb, not a measured optimum.\n\nAnother minor point: the energy-saving numbers come from a single optimized structure, so they illustrate rather than validate the method.\n\nWho it's for: practitioners using FM-based active learning (especially with quantum annealers) who need a starting point for initial-data size. It is not a fundamental contribution, but it fills a practical gap.\n\nBottom line: I would send it to peer review, but with a clear request to reframe the quantitative recommendations as heuristics, to justify or eliminate the threshold adjustment, and to add repeated-run statistics. With those revisions it could be a useful reference for the community.","headline":"Qualitative message is solid; the specific recommended initial-data sizes rest on a post hoc tuned convergence threshold and lack repeated-run statistics.","tokens_in":11458,"tokens_out":2320,"would_cite":false,"duration_ms":24782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adequate initial data is decisive for convergence in surrogate-based active learning, with recommended starting sizes from 25 points for 40-bit systems to 3,000 for 160-bit systems.","keywords":["active learning","surrogate model","factorization machine","QUBO","initial data size","convergence analysis","transparent radiative cooling","materials optimization"],"falsifier":"Run the 120-bit transparent radiative cooling optimization starting with 500 initial data (below the recommended 1,000) and track both the regression-gradient crossing and the actual minimum FOM per cycle; if the minimum FOM keeps decreasing substantially for hundreds of cycles after the gradient crosses -3, the threshold is not tracking real convergence.","tokens_in":10490,"feed_emoji":"❄️","tokens_out":4981,"duration_ms":48294,"temperature":0.7,"pith_summary":"Surrogate-based active learning for functional materials usually launches with a fixed number of initial data points, often 25 or 50, regardless of how large the design space is. This paper argues that the right amount of starting data is decisive: too little, and the surrogate's early queries behave like random sampling, delaying or derailing convergence, especially for large systems. Working on transparent radiative cooling windows encoded as 40- to 160-bit binary design spaces, it recommends initial data sizes of 25 (40-bit), 100 (80-bit), 200 (100-bit), 1,000 (120-bit), 2,000 (140-bit), and 3,000 (160-bit) to reach convergence within 500 optimization cycles. The paper's point is that scaling the initial dataset with the search space is a cheap way to avoid wasted computation in the active-learning loop.","feed_headline":"Starter data size decides materials-search convergence","feed_subtitle":"Fixed 25-point starts fail past 60 bits; recommended sizes reach 3,000 for 160-bit designs.","key_machinery":"The load-bearing mechanism is the factorization-machine surrogate: a supervised model whose trained parameters ($w_0$, $w_i$, $v_{i,f}$) plug directly into a quadratic unconstrained binary optimization (QUBO) matrix, so each active-learning cycle proposes a candidate structure by solving a QUBO. To detect when optimization actually converges, the paper uses averaged piecewise linear regression: five piecewise regressions with 20 pieces each and shifted ranges, averaged to fit the non-smooth FOM-versus-cycle distribution, with convergence declared when the regression gradient first drops below a threshold. That regression threshold is the tool that converts scattered FOM data into the recommended initial-data sizes.","core_discovery":"The central claim is that convergence of surrogate-based active learning depends strongly on the number of initial data points, and that a threshold-based analysis of FOM curves can identify how many are needed. Using a factorization-machine surrogate mapped to a QUBO and solved by simulated annealing, the authors optimize multilayered transparent radiative cooling windows over seven design-space sizes. They fit averaged piecewise linear regressions to FOM-versus-cycle scatter and define convergence onset as the first cycle where the regression gradient crosses -3 (or -2 for the 160-bit system). Under this criterion, 40- and 60-bit systems converge within 500 iterations even with 25 initial data, while 80-, 100-, 120-, 140-, and 160-bit systems respectively demand 100, 200, 1,000, 2,000, and 3,000 initial data for the same guarantee. The authors conclude that adequate initial data is essential for reliable, cost-effective optimization of large functional-material design spaces.","pith_inferences":["The recommended sizes likely transfer only to FM/QUBO surrogate pipelines with similar FOM landscapes; other surrogates such as deep networks or Gaussian processes may need a different scaling rule, and this is a testable hypothesis the paper does not address.","The quoted initial-data sizes imply an empirical scaling law: required initial data grows much faster than linearly with bit count, from 25 points at 40 bits to 3,000 at 160 bits; if that trend holds, 200-bit systems would need thousands more before optimization can start.","The -3/-2 gradient threshold could be replaced by a statistical control chart on rolling FOM improvements, which would give an online stopping rule rather than a post-hoc regression analysis; such a rule would make the recommendations actionable during optimization.","Because the authors used simulated annealing instead of a real quantum annealer, the exact initial-data sizes could shift when the QUBO solver's solution quality changes; this is an untested extension."],"forward_implications":["Practitioners should stop using a fixed initial data size such as 25 or 50 for all design spaces; the paper's results imply the starting dataset must grow with the binary dimension of the design space.","For the TRC window problem, systems up to 60 bits converge within 500 cycles with only 25 initial data, so small design spaces can keep the cheap start.","For large systems (120 bits and above), the required initial data jumps into the thousands, suggesting that earlier studies using small fixed initial data may have systematically underestimated achievable FOMs or underreported convergence failures.","A transparent radiative cooling window optimized under this scheme saves up to about 34% of cooling energy relative to conventional glass windows, with larger savings in tropical climates."],"supporting_citations":[{"why":"Establishes the factorization-machine-to-QUBO pipeline for quantum-annealing-assisted active learning, which this paper inherits as its optimization engine.","marker":"Kitai et al., 2020"},{"why":"Demonstrates the transparent radiative cooler design with quantum computing and supplies the FM-based active learning workflow used here.","marker":"S. Kim et al., 2022"},{"why":"Provides the wide-angle spectral filter and TRC window case with fixed initial data (25 or 50), the practice this paper argues against.","marker":"S. Kim, Jung et al., 2024"},{"why":"Supplies the QUBO formulation and quantum annealing-aided optical diode design that underpin the surrogate-solving step.","marker":"S. Kim, Park et al., 2024"},{"why":"Provides the piecewise linear regression method that the averaged piecewise regression extends for convergence detection.","marker":"Jekel & Venter, 2019"},{"why":"Explains why high-order polynomial regression is unsuited for these FOM distributions, motivating the averaged piecewise regression choice.","marker":"Gelman & Imbens, 2019"}],"fun_headline_variants":["Initial data size sets the convergence pace in materials design","From 25 to 3,000 seeds: design space size dictates the need","Why materials search needs more seed data as spaces grow","Optimal initial data: the key to faster materials optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole recommendation stands on the assumption that the first time the averaged regression slope crosses -3 (or -2 for the 160-bit problem) marks the point where real optimization improvement begins, rather than an artifact of the fitting method.","fun_headline_variants_meta":{"raw":{"variants":["Initial data size sets the convergence pace in materials design","From 25 to 3,000 seeds: design space size dictates the need","Why materials search needs more seed data as spaces grow","Optimal initial data: the key to faster materials optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1822,"prompt_tokens":930,"completion_tokens":892,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":822}},"tokens_in":546,"tokens_out":892,"duration_ms":10489,"temperature":1.0,"reasoning_tokens":822,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:04:56.053224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 120-bit transparent radiative cooling optimization starting with 500 initial data (below the recommended 1,000) and track both the regression-gradient crossing and the actual minimum FOM per cycle; if the minimum FOM keeps decreasing substantially for hundreds of cycles after the gradient crosses -3, the threshold is not tracking real convergence.","supporting_citations":[],"review_version":1}