{"id":"63098480-cef3-41b5-b439-05abd3ab8d29","arxiv_id":"2506.19384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A progressive quadtree search with consistency-based sample selection designs electromagnetic structures with 1000 simulations, beating baselines that use up to 7000.","lead":"This paper introduces PQS, a search method that designs electromagnetic structures such as antennas and frequency-selective surfaces using far fewer expensive full-wave simulations. It works by representing layouts as a quadtree, searching coarse patterns first, and using a consistency check on a surrogate predictor to decide when to trust it or explore randomly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison rests on single real-simulation runs; the 10-run robustness study uses a surrogate whose KTau=0.7969 may not preserve method-level ordering, so the claimed margin over baselines is not statistically established.","rationale":"The reader's weakest assumption (quadtree expressiveness at Nmax=32) is a genuine limitation, but it is not the most load-bearing threat to the central claim. PQS only needs to beat the implemented baselines under 1000 evaluations; it does not need to contain the true pixel-level optimum. The ablation (Nmax=16/32/64) supports that 32 is a reasonable operating point on HGA, and the optimized visualizations show blocky but manufacturable designs. So even if a pixel-level optimizer would eventually do better, that would not by itself falsify 'PQS outperforms baselines under limited budget.' By contrast, the single-run real-simulation comparison directly underpins every quantitative superiority statement in the abstract and Section 5.1. The paper's own robustness section substitutes a surrogate with KTau=0.7969 and reports absolute numbers that differ from the real-simulation results, so it cannot serve as a statistical replacement. If repeated real-simulation runs show PQS's margin is within noise, the main claim is not established. This does not accuse the authors of anything; it simply identifies the weakest evidentiary link and a concrete way to test it. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":20011,"tokens_out":10631,"duration_ms":112999,"concrete_test":"Run the full Table 3 protocol 10 times with independent seeds for PQS and for the strongest baseline on each task (e.g., RS and cVAE) under the real full-wave simulator, keeping all budgets and hyperparameters fixed. Report mean±std of Agg Obj. If PQS's mean is not above the baseline mean by more than the pooled standard error on at least one task, the headline outperformance claim fails. As a cheaper ancillary check, hold out 100 designs per method from the real-simulation logs and compare surrogate-predicted vs. real-simulated objectives to verify whether the KTau=0.7969 surrogate preserves method-level ordering; if it does not, Table 4's robustness conclusions are invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that PQS 'markedly surpasses all baselines' (Sec. 5.1, Table 3) is supported only by single real-simulation runs for each method. No seeds, number of runs, or error bars are reported for Table 3; the only multi-run experiment (Table 4) replaces the real simulator with a surrogate trained on 37,354 HGA samples with KTau=0.7969. A rank correlation of 0.80 does not guarantee that method-level ordering under the surrogate matches the real simulator, especially for methods that can exploit surrogate artifacts (e.g., overoptimistic predictor optima). Moreover, Table 4's absolute objective values for PQS (4.34±0.34 dB) differ from the real-simulation value in Table 3 (3.66 dB), so the robustness study does not directly corroborate the main numbers. Without repeated real-simulation runs, the observed margins (e.g., 15.20 vs 7.28 dB on DualFSS) could partly reflect run-to-run noise rather than a systematic advantage. This is load-bearing because the central claim is a comparative empirical claim; if the margin collapses under repetition, the paper's main conclusion is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Progressive Quadtree-based Search (PQS), a method for electromagnetic structure (EMS) design under limited full-wave simulation budgets. PQS represents binary layouts as quadtrees that are progressively refined, and uses a consistency-based sample selection mechanism that measures the Kendall-tau agreement between consecutive surrogate predictions to balance exploitation and exploration. The method is evaluated on two real-world tasks, Dual-layer Frequency Selective Surface (DualFSS) and High-gain Antenna (HGA), where PQS uses 1000 simulations and is compared against predictor-based, generative, and random-sampling baselines, some of which use up to 7000 simulations. The paper reports that PQS achieves the best aggregate objective on both tasks and includes an ablation study of the quadtree depth and of the two proposed components.","tokens_in":20275,"tokens_out":4573,"duration_ms":45442,"significance":"If the reported empirical results are reliable, PQS offers a practically meaningful reduction in the number of expensive full-wave simulations required for EMS design, in some cases outperforming strong baselines with 75-85% fewer evaluations. The quadtree representation and the consistency-driven selection mechanism are novel and potentially transferable to other discrete design problems with costly evaluations. The paper also provides ablations investigating the effect of Nmax and the individual contributions of QSS and CSS. However, the statistical evidence supporting the headline comparative claim is currently incomplete: the central comparison rests on single real-simulation runs, and the multi-run robustness study uses a surrogate with only moderate rank correlation.","major_comments":[{"comment":"The headline comparison that PQS 'markedly surpasses all baselines' is supported only by single runs for each method on the real simulators. No number of seeds, repeated runs, or error bars are reported for Table 3. Because the objective is a minimum over several performance criteria and the search itself involves randomness (random leaf selection, Bernoulli splits), the observed margins, such as 15.20 dB vs. 7.28 dB on DualFSS, could partly reflect run-to-run variation. This is load-bearing for the central claim; please provide repeated real-simulation runs (or a clearly justified statement that the simulators and pipelines are deterministic) and report means and variances.","section":"5.1 / Table 3"},{"comment":"The robustness study replaces the real simulator with a surrogate trained on 37,354 HGA samples with a Kendall-tau of 0.7969, and the paper asserts this 'is generally sufficient to capture relative performance trends.' A rank correlation of 0.80 does not guarantee that the method-level ordering under the surrogate matches the real simulator, especially for predictor-based methods that may exploit surrogate artifacts. Moreover, the absolute PQS value in Table 4 (4.34±0.34 dB) differs from the real-simulation value in Table 3 (3.66 dB), so Table 4 does not directly corroborate the main result. Please validate method-level ordering on a hold-out portion of the real simulator, or better, repeat a small number of real-simulation runs.","section":"5.2 / Table 4"},{"comment":"The simulation-count reduction used in the time-saving calculation is swapped between the two tasks. For DualFSS, generative methods use 7000 simulations and PQS uses 1000, so the reduction is 6000, not 3000; for HGA, the reduction is 3000, not 6000. As a result, the computed time savings of 20.27 days and 38.80 days are attributed to the wrong tasks, and the abstract's claim that PQS 'saves 20.27-38.80 days' rests on this arithmetic error. Please correct the calculation and the attribution, and rerun the associated cost-reduction percentages.","section":"Appendix C, Time-saving Calculation"},{"comment":"The quadtree representation caps all designs at Nmax=32 homogeneous rectangular leaf nodes, which implicitly assumes that near-optimal EMS layouts can be well approximated by at most 32 rectangular blocks. The paper provides an ablation on HGA (Nmax ∈ {16,32,64}) but no analogous evidence on DualFSS or a second task, and the generality of this 'blockiness' assumption across EMS problems is untested. Please state this limitation explicitly and, if possible, provide additional evidence that Nmax=32 is a reasonable default for other EMS design tasks.","section":"4.1 / Eq. (6) / Table 5"}],"minor_comments":[{"comment":"The text refers to 'Table 5.1', which should be 'Table 4'.","section":"5.2"},{"comment":"The symbol n is used both for the number of candidate samples in the Kendall-tau computation and for a quadtree node; please disambiguate the notation.","section":"Eq. (9)"},{"comment":"The column header 'xDesign Space Dimension' is garbled; it should read 'Design Space Dimension' or similar.","section":"Table 2"},{"comment":"The ablation results in Table 6 appear to be single runs as well; please clarify whether these are single runs or averages, and if single, add error bars or repeated runs.","section":"5.3 / Table 6"},{"comment":"The claim that PQS 'cuts evaluation costs by 75-85%' should be recomputed after fixing the swapped simulation-count reduction in the supplementary calculation.","section":"Abstract / Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practically important problem and the proposed method is interesting, but the empirical evidence for the central comparative claim is not yet statistically solid. The single-run real-simulation results in Table 3 and the surrogate-based robustness study in Table 4 need to be strengthened or clearly qualified. The swapped time-saving calculation is an easily correctable error, but it currently affects a headline number in the abstract. I recommend major revision rather than rejection because the issues are fixable within the manuscript's scope and the method itself has merit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of arXiv:2506.19384. The core idea is worth noting: instead of optimizing pixel-by-pixel, PQS uses a quadtree to represent a binary layout as a small set of homogeneous blocks, progressively refines promising regions, and uses Kendall's tau between successive predictor versions to decide how many candidates to simulate versus explore. That's a sensible combination for expensive simulation problems, and it is genuinely new in the EMS literature. The reported results show large margins over strong baselines at 1000 simulations, and the ablation studies (Nmax, QSS, CSS) are internally consistent. The paper also honestly points out that EMS lacks the public datasets and pretrained models available in NAS and molecule design.\n\nThe weaknesses are real and, in places, load-bearing. First, the headline comparison in Table 3 is single-run: no seeds, no repeats, no error bars. The robustness study in Table 4 uses a surrogate with KTau=0.797, which does not guarantee that method-level ordering matches the real simulator—and the absolute values differ from Table 3 (4.34 vs 3.66 on HGA), so it does not directly corroborate the main numbers. Second, the initial dataset for PQS is described as 'derived from a progressive design strategy,' while baselines start from random samples. That could be a substantial unfair advantage, and it is not controlled or explained. Third, the time-saving calculation in the appendix swaps the simulation reductions (writes 3,000 for DualFSS when the actual reduction is 6,000, and 6,000 for HGA when it is 3,000), which flips the claimed days saved. That's a careless but easily fixed error. The constraint that designs must be representable by at most 32 homogeneous rectangles (Nmax=32) is an untested generality assumption; the provided ablation on a single task is supportive but thin.\n\nThe math and the citation pattern look fine; the method is coherent and the reimplementation of baselines appears reasonable, though no code is released. A serious referee could push for repeated real-simulation runs on at least one task, a controlled initialization comparison, and a corrected time analysis. This is a solid engineering contribution that needs validation work, not a desk-reject. I'd send it to review with a request for major revision.","headline":"Useful hierarchical quadtree search idea with a suggestive but under-validated empirical claim; worth reviewing, needs real-simulation repeats and a fair initialization control.","tokens_in":20769,"tokens_out":4666,"would_cite":false,"duration_ms":48179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A progressive quadtree search with consistency-based selection designs electromagnetic structures in 1,000 simulations, beating generative baselines that use 4,000-7,000.","keywords":["electromagnetic structure design","quadtree representation","surrogate-assisted optimization","consistency-based sample selection","limited evaluation budget","frequency selective surface","high-gain antenna","Kendall's tau"],"falsifier":"Run PQS on a binary layout task whose known global optimum is a fine-grained pattern that cannot be assembled from 32 rectangles, and compare its best aggregate objective under 1,000 simulations with a pixel-level search using the same budget; if the pixel-level search consistently wins, the quadtree block budget is the limiting factor.","tokens_in":19846,"feed_emoji":"📡","tokens_out":9954,"duration_ms":87402,"temperature":0.7,"pith_summary":"Electromagnetic structure design requires searching a huge binary layout space, and each candidate layout must be checked with a full-wave simulation that can take minutes to hours. The paper tries to establish that this process can run on a tight budget if the layout is represented as a quadtree, so that search starts with coarse homogeneous blocks and refines only promising regions, and if the limited simulation budget is allocated by the consistency of surrogate predictions rather than by their raw accuracy. Concretely, it reports that with 1,000 simulations the Progressive Quadtree-based Search (PQS) reaches an aggregate objective of 15.20 dB on a dual-layer frequency-selective surface and 3.66 dB on a high-gain antenna, above all tested baselines, including generative methods that used 7,000 and 4,000 simulations. If this holds, EMS design could be done with 75-85 percent fewer simulations than generative pipelines, saving an estimated 20-38 days of product design time per cycle.","feed_headline":"Quadtree search cuts EM design simulations by 75-85 percent","feed_subtitle":"A quadtree layout plus consistency-based sampling reaches FSS and antenna targets with 1,000 simulations.","key_machinery":"The quadtree representation is the dimensionality-reduction device: it recursively partitions the $m\\times n$ binary layout into rectangular leaf blocks, each labeled 0 or 1, so the design space has size $2^{|L|}$ instead of $2^{mn}$, and the layout is reconstructed via Eqn. 5. The progressive tree search grows the tree by randomly resampling a leaf's state or splitting it into four children (Bernoulli with probability 1/2), maintains a Top-K list by surrogate score, and then applies depth-wise importance assignment to refine the most influential subregions of the Top-K designs. The Consistency-based Sample Selection uses Kendall's tau between successive predictor rankings (Eqn. 9) as a reliability signal, with the exploitation/exploration split $R_p=\\tau R$, $R_r=(1-\\tau)R$ (Eqn. 10). Together these mechanisms let an imperfect predictor guide optimization when its rankings are stable and force extra random evaluations when they are not.","core_discovery":"The central claim, stated on the paper's own terms, is that PQS makes EMS optimization dramatically more sample-efficient by changing the representation and the selection rule rather than by building a larger or better predictor. The quadtree representation stores each homogeneous rectangular region as a single bit and reconstructs the $m\\times n$ layout from its leaf nodes (Eqn. 5), cutting the search space from $2^{mn}$ to $2^{|L|}$; progressive tree search expands from $|L|=1$ up to $N_{\\max}=32$ leaves, resampling leaf states or splitting nodes, and then refines the Top-K candidates by depth-wise importance assignment. The Consistency-based Sample Selection computes Kendall's tau $\\tau$ between the predictor rankings from iterations $t-1$ and $t$ (Eqn. 9) and allocates a fraction $\\tau$ of the next batch to the surrogate's top-ranked candidates and $1-\\tau$ to random ones (Eqn. 10). With a budget of 1,000 full-wave simulations on two real tasks, the method reports aggregate objectives of 15.20 dB (DualFSS) and 3.66 dB (HGA), and on the high-gain antenna robustness benchmark it reports the highest mean (4.34 dB) and lowest variance across 10 runs among all compared methods.","pith_inferences":["The same quadtree-plus-consistency loop should transfer to other binary layout optimization problems, such as metasurface patterning, photonic-crystal design, or chip-level shielding, where manufacturability favors large homogeneous regions; the search would need no task-specific change beyond the geometry of the layout.","The Kendall-tau gating rule encodes a general principle for surrogate-based optimization: trust the surrogate's recommendations in proportion to how stable its rankings have been, a rule that could be dropped into Bayesian optimization or evolutionary search with any learned model.","A direct stress test would be to run PQS on a task whose known optimum has fine pixel-level texture; if performance degrades as the optimum becomes less block-representable, the $N_{\\max}=32$ cap is a structural limitation rather than a benign default.","The reported time savings assume simulation cost dominates; in settings where predictor training is the bottleneck, the benefit of the quadtree search would shrink, so the method's advantage is largest in exactly the expensive-simulation regime the paper targets."],"forward_implications":["With 1,000 full-wave simulations, PQS reaches 15.20 dB on DualFSS and 3.66 dB on HGA, higher than every baseline tested, including generative methods that used 7,000 and 4,000 simulations.","The reported budget cut of 75-85 percent relative to generative approaches corresponds to an estimated 20.27-38.80 days of saved simulation time per product design cycle.","Surrogate-assisted methods that rely directly on predictor scores can underperform random sampling under a 1,000-sample budget, so the consistency-gated selection rule is a necessary part of the method, not a minor add-on.","The ablation fixing $N_{\\max}=32$ leaves indicates that moderate block granularity outperforms both 16 and 64 leaves on HGA, implying there is a representational sweet spot for the quadtree.","Across 10 runs on the high-gain antenna task, PQS has the highest mean aggregate objective (4.34 dB) and the lowest variance among all compared methods, which supports the claim that the method is robust to initialization and data variability."],"supporting_citations":[{"why":"Supplies Kendall's tau, the rank-correlation measure that PQS uses to quantify prediction consistency and set the exploitation/exploration split.","marker":"Kendall, 1938"},{"why":"Defines the Surrogate-GA baseline, a surrogate-assisted genetic algorithm run under the same 1,000-simulation budget.","marker":"Zhu et al., 2020"},{"why":"Defines the Surrogate-GW baseline, a surrogate-assisted grey wolf optimizer used as a comparison method.","marker":"Dong & Dong, 2020"},{"why":"Defines the InvGrad baseline, which performs gradient ascent through a learned surrogate and is compared against PQS.","marker":"Trabucco et al., 2022"},{"why":"Defines the cGAN generative baseline whose required 7,000 simulations on DualFSS anchor the cost-reduction comparison.","marker":"An et al., 2021"},{"why":"Defines the cVAE generative baseline used for conditional layout generation in the comparisons.","marker":"Lin et al., 2022"},{"why":"Defines the IDN inverse-design baseline that maps target performance values directly to layouts.","marker":"Ma et al., 2020"},{"why":"Defines the GenCO constrained generative optimization baseline compared against PQS.","marker":"Ferber et al., 2024"},{"why":"Defines the TS-DDEO two-stage surrogate-assisted evolutionary baseline used in the comparison.","marker":"Zheng et al., 2023b"},{"why":"Defines the SAHSO surrogate-assisted hybrid swarm optimization baseline used in the comparison.","marker":"Li et al., 2022"}],"fun_headline_variants":["Quadtree search: 75-85% fewer EM simulations, same quality","1,000 simulations for EM design: quadtree cuts costs 75-85%","EM design with quadtree: progressive search saves 20-38 days","Consistency-based sampling enables EM design with 1,000 sims","Progressive quadtree search: limited-budget EM design wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that near-optimal EMS layouts can be expressed with at most 32 homogeneous rectangular blocks; if the true optimum needs finer pixel-level detail, PQS cannot reach it no matter how well the search or sample selection works.","fun_headline_variants_meta":{"raw":{"variants":["Quadtree search: 75-85% fewer EM simulations, same quality","1,000 simulations for EM design: quadtree cuts costs 75-85%","EM design with quadtree: progressive search saves 20-38 days","Consistency-based sampling enables EM design with 1,000 sims","Progressive quadtree search: limited-budget EM design wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1514,"prompt_tokens":1028,"completion_tokens":486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":644,"tokens_out":486,"duration_ms":5176,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:07:12.066706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PQS on a binary layout task whose known global optimum is a fine-grained pattern that cannot be assembled from 32 rectangles, and compare its best aggregate objective under 1,000 simulations with a pixel-level search using the same budget; if the pixel-level search consistently wins, the quadtree block budget is the limiting factor.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Kendall's tau, the rank-correlation measure that PQS uses to quantify prediction consistency and set the exploitation/exploration split."},{"cited_title":"Multiplexing the aperture of a metasurface: inverse design via deep-learning-forward genetic algorithm","cited_arxiv_id":null,"evidence_quote":"Defines the Surrogate-GA baseline, a surrogate-assisted genetic algorithm run under the same 1,000-simulation budget."},{"cited_title":"and Dong, Z","cited_arxiv_id":null,"evidence_quote":"Defines the Surrogate-GW baseline, a surrogate-assisted grey wolf optimizer used as a comparison method."},{"cited_title":"Design-bench: Benchmarks for data-driven offline model-based optimization","cited_arxiv_id":null,"evidence_quote":"Defines the InvGrad baseline, which performs gradient ascent through a learned surrogate and is compared against PQS."},{"cited_title":"Machine-learning-assisted inverse design of scattering enhanced metasurface","cited_arxiv_id":null,"evidence_quote":"Defines the cVAE generative baseline used for conditional layout generation in the comparisons."},{"cited_title":"M., Zharmagambetov, A., Huang, T., Dilkina, B., and Tian, Y","cited_arxiv_id":null,"evidence_quote":"Defines the GenCO constrained generative optimization baseline compared against PQS."}],"review_version":1}