{"id":"8d32072d-8803-4d19-8981-31538f1a9926","arxiv_id":"2510.23472","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BBOPlace-Bench is a unified benchmark for black-box optimization of chip placement, where evolutionary algorithms under mask-guided and hyperparameter formulations beat analytical and RL baselines on wirelength metrics.","lead":"A new benchmark for black-box optimization (BBO) in chip placement combines three existing search formulations and tests five algorithm families on industrial chip suites. It gives BBO researchers a common playground for a real, expensive, high-dimensional design problem, with early evidence that evolutionary search can beat analytical and reinforcement-learning placers on wirelength.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HPO GP-HPWL advantage may reflect tuning the DREAMPlace evaluation engine, not an independent BBO placement capability; comparison versus default-hyperparameter baselines is uncontrolled.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: DREAMPlace is not an impartial completion engine because the HPO search space is precisely DREAMPlace's configuration space. This is the most consequential issue because the paper's abstract and introduction tout BBO as competitive with analytical and RL methods, and the GP-HPWL tables are the primary evidence. If the HPO advantage is mainly hyperparameter tuning, the headline shrinks to 'tuned DREAMPlace beats default DREAMPlace,' which is already known from AutoDMP. The concern is neither ad hominem nor a consensus disagreement; it is a controlled-comparison issue that undermines a specific empirical claim. The benchmark itself retains value: it provides a modular, code-released platform for testing BBO algorithms under fixed formulations, and the MP-HPWL results suggest MGO/EA are genuinely competitive on macro placement. A conditional acceptance is therefore appropriate, contingent on running the proposed control. No reason to move the verdict to ACCEPT or REJECT; the reader's CONDITIONAL is well calibrated.","tokens_in":30595,"tokens_out":5610,"duration_ms":55287,"concrete_test":"For each of the six ISPD 2005 chips, take the best macro placement found by MGO-Vanilla-EA (from Table III) and complete standard-cell placement with DREAMPlace configured using the best hyperparameters found by HPO-Vanilla-EA for that same chip and seed, instead of DREAMPlace defaults. Compare the resulting GP-HPWL values against the HPO-Vanilla-EA GP-HPWL entries in Table IV. If the gap narrows to within ~10% on most chips, the reported HPO superiority is largely attributable to tuning the downstream evaluation engine, not to an independently superior BBO placement formulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'some BBO configurations are competitive with representative analytical and RL baselines' relies heavily on GP-HPWL results in Tables IV and VI, where HPO-Vanilla-EA and HPO-PSO achieve average ranks of 2–3 versus DREAMPlace's 9–11. But the evaluation protocol in Section III-D gives the non-HPO formulations (SP, MGO) a fixed downstream engine: after BBO places macros, DREAMPlace completes standard-cell placement using default hyperparameters. In contrast, the HPO formulation (Algorithm 3, Table I) searches exactly DREAMPlace's hyperparameter space, and each solution decoding is a full DREAMPlace run with tuned settings. Thus HPO algorithms control the evaluation engine's configuration, while MGO/SP and the DREAMPlace baseline do not. The paper itself acknowledges the HPO advantage is 'attributable to its capability of tuning the hyperparameters of the analytical placer DREAMPlace.' Consequently, the headline result conflates BBO search ability with hyperparameter tuning of the baseline: HPO may simply show that tuned DREAMPlace beats untuned DREAMPlace, not that a BBO placement method is intrinsically competitive with analytical placement. This does not invalidate the benchmark's utility for comparing BBO algorithms within a fixed formulation, but it weakens the cross-method claim that is a central selling point. A simple control—completing MGO/SP macro placements with HPO-discovered hyperparameters—would determine whether the gap is an artifact of this asymmetry.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BBOPlace-Bench, proposed as the first benchmark tailored to black-box optimization (BBO) for chip placement. It integrates three problem formulations — Sequence Pair (SP), Mask-Guided Optimization (MGO), and Hyperparameter Optimization (HPO) over DREAMPlace — and five BBO algorithm families (SA, Vanilla-EA, CMA-ES, PSO, BO), with standardized preprocessing for ISPD 2005 and ICCAD 2015, and three evaluation levels: MP-HPWL, GP-HPWL, and PPA. The experiments compare formulations and algorithms under a shared protocol and also compare against analytical (DREAMPlace) and RL (AlphaChip, MaskPlace, EfficientPlace) baselines. The headline findings are that MGO and HPO dominate SP, EAs tend to outperform SA and BO, and, under GP-HPWL, HPO-based BBO methods achieve better average ranks than the analytical and RL baselines. The authors are careful to frame the experiments as illustrative case studies and to acknowledge limitations, including the reliance on commercial PPA tools.","tokens_in":31030,"tokens_out":2761,"duration_ms":32150,"significance":"The benchmark is potentially useful to the BBO community: it provides a modular, decoupled interface, unified preprocessing of two industrial benchmark suites, three search-space paradigms, a wide selection of out-of-the-box BBO algorithms, and both cheap (MP-HPWL) and expensive (GP-HPWL/PPA) evaluation settings. If the comparison protocol were unbiased, the result that some BBO configurations can compete with analytical and RL placers would be notable and would support the paper's central claim. The paper also contains genuinely valuable internal comparisons — e.g., the scalability failure of SP, the relative performance of BO in high dimensions, and the trade-off between MGO and HPO — that do not depend on the cross-paradigm baseline comparison. The paper ships code, reports standard deviations over five seeds, and uses Wilcoxon rank-sum tests for within-benchmark comparisons, which are good practices that strengthen the empirical claims.","major_comments":[{"comment":"The central claim that BBO configurations are competitive with representative analytical and RL baselines in GP-HPWL is undermined by an asymmetric evaluation protocol. For SP and MGO, macro placement is performed by the BBO method and then DREAMPlace completes standard-cell placement with (presumably default) hyperparameters. For HPO, the search space is exactly DREAMPlace’s hyperparameter space (Algorithm 3, Table I) and each solution decoding is a full DREAMPlace run with tuned hyperparameters. The paper itself acknowledges this in §IV-B(b): the HPO advantage is ‘attributable to its capability of tuning the hyperparameters of the analytical placer DREAMPlace.’ Thus the high average ranks of HPO-Vanilla-EA (2.0) and HPO-PSO (2.67) in Table IV, and 1.88/1.38 in Table VI, may reflect hyperparameter tuning of the baseline engine rather than an independent black-box placement capability. T","section":"§III-B3, §III-D, Tables IV and VI"},{"comment":"The PPA evaluation selects for each algorithm the single best placement across runs by GP-HPWL, reports no variance, and provides no external baselines. Metrics such as routed wirelength differ by well under 1% in several cases (e.g., superblue1: 75.06 vs 74.90 vs 75.23 m for SA/Vanilla-EA/PSO), yet the text states that PSO ‘performs the best in five cases’ and Vanilla-EA in three. These rankings are not accompanied by any significance test or repeatability measure, and best-of-N selection can introduce bias when runs are few. The claims in this subsection should be softened to descriptive observations or supplemented with variance/statistics over runs.","section":"§IV-C-b, Table VII"},{"comment":"The comparison against external baselines is not controlled for hyperparameter tuning. The BBO algorithms are given a defined search budget and, for HPO, the ability to tune DREAMPlace, while the analytical and RL baselines (DREAMPlace, MaskPlace, EfficientPlace, AlphaChip) are taken from the literature or from default configurations. This is acknowledged only implicitly. A fairer statement would be that BBO with access to the downstream placer’s configuration space is competitive with untuned analytical baselines, or the authors should tune the baselines with a comparable budget. This does not invalidate the within-benchmark comparisons, but it is load-bearing for the abstract’s wording that ‘some BBO configurations ... are competitive with representative analytical and reinforcement learning baselines.’","section":"§IV-A, §IV-B, §IV-C"}],"minor_comments":[{"comment":"For ICCAD 2015, macros are defined as the 512 largest modules by area. This is an important design choice, but no sensitivity analysis is given for this threshold; Table VIII varies macro count only for MGO, not for the SP and HPO formulations. The arbitrariness of this choice should be at least flagged in the main text.","section":"§IV-A"},{"comment":"Typo: ‘cooling schedule (refereed to as temperature)’ should be ‘referred to as temperature.’","section":"§II-A"},{"comment":"Footnote 4 contains a doubled URL prefix: ‘https://https://github.com/...’.","section":"§IV-A"},{"comment":"The formatting of the SP rows in Table IX is unclear: the optimization/evaluation times for SP SA and Vanilla-EA are merged in a way that makes the per-iteration time hard to read. Please reformat the table for readability.","section":"Table IX"},{"comment":"The abstract is appropriately hedged (‘some BBO configurations ... are competitive’), but the Conclusion states more strongly that ‘EAs demonstrate better overall performance than SA and BO ... and also achieve state-of-the-art performance compared to the mainstream chip placement methods.’ Given the asymmetric protocol, the conclusion overstates the evidence. Please align the strength of the claims with the controlled part of the experiments.","section":"Abstract / §V"}],"recommendation":"major_revision","confidential_remarks":"The reviewer’s main concern is not that the benchmark is flawed as a BBO testbed, but that the headline cross-method claim rests on an asymmetric evaluation: HPO searches DREAMPlace’s hyperparameter space while the other formulations and the analytical baseline do not. This is fixable within the manuscript’s scope by adding a control experiment or by substantially qualifying the cross-method claims. The PPA section also needs statistical grounding. I recommend major revision rather than rejection because the core benchmark contribution and the within-formulation algorithm comparisons are sound and useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a real contribution: a unified, modular benchmark for BBO on chip placement, with standardized formats across ISPD 2005 and ICCAD 2015, three problem formulations, five BBO algorithms, and an evaluation pipeline. That fills a genuine gap, and the code is available. Second, don't buy the headline that BBO beats analytical/RL placers on GP-HPWL. The HPO formulation is just tuning DREAMPlace's hyperparameters, and the paper openly says the HPO advantage is 'attributable to its capability of tuning the hyperparameters of the analytical placer DREAMPlace.' So HPO-Vanilla-EA at rank 2 vs DREAMPlace at rank 9 is largely tuned-DREAMPlace vs default-DREAMPlace, not an independent BBO method beating an analytical placer. The stress-test concern lands.\n\nWhat's genuinely new and done well: the benchmark artifact, the decoupling of formulation/algorithm/evaluation, and the systematic comparison of SA, EA variants, and BO across the three formulations under a shared budget and metric definitions. The within-formulation comparisons (e.g., MGO vs SP, EA vs BO at high dimension) are internally consistent, report five seeds with error bars, and use Wilcoxon tests. The tables are readable and the code seems complete enough to reproduce.\n\nSoft spots, in order of importance. (1) The cross-paradigm claim is weakened by the DREAMPlace asymmetry; a control where MGO/SP macro placements are completed with HPO-discovered hyperparameters would settle it. (2) PPA results (Table VII) report a single selected run per algorithm with no variance, no baselines like DREAMPlace or MaskPlace, and the metrics don't show a clear winner; the authors call these illustrative, so treat them as exploratory, not evidence of superiority. (3) The 'first benchmark' claim is acceptable but the novelty is the integration, not the ingredients; MGO is WireMask-BBO and HPO is AutoDMP, both from the same group or closely related. Self-citation is fine when the prior work is reproducible, and it is. (4) The number of macros for ICCAD is set to 512 largest modules by area, which is a choice that affects MGO dimension and is worth flagging.\n\nOverall: the benchmark deserves a serious referee and is likely to be a useful resource for the BBO community, though the paper's framing overreaches. Push the authors to add the control experiment and to tone down the cross-paradigm claims. I'd engage.","headline":"Useful benchmark, but the 'BBO beats analytical' headline rests on HPO tuning DREAMPlace itself.","tokens_in":31478,"tokens_out":1911,"would_cite":false,"duration_ms":21775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the first benchmark built specifically for black-box optimization in chip placement and provides evidence that BBO configurations can match or beat analytical and reinforcement-learning placers on wirelength.","keywords":["black-box optimization","chip placement","macro placement","global placement","wirelength","benchmark","evolutionary algorithms","hyperparameter optimization"],"falsifier":"Run the same five algorithms under the same protocol but replace DREAMPlace with a different analytical placer as both the GP-HPWL evaluator and the HPO decoder; if HPO loses its ranking advantage over MGO and the RL/analytical baselines, the paper's central comparative result is an artifact of DREAMPlace's hyperparameter sensitivity rather than a general BBO advantage.","tokens_in":30547,"feed_emoji":"🧩","tokens_out":9872,"duration_ms":76989,"temperature":0.7,"pith_summary":"Black-box optimization (BBO) has been applied to chip placement for decades, but early formulations were too weak to compete with analytical placers. This paper argues that recent BBO formulations change that, and it introduces a benchmark to make the case testable: BBOPlace-Bench, which unifies three problem formulations—sequence pair, mask-guided optimization, and hyperparameter optimization—under one evaluation protocol on industrial chip cases. Under that protocol, the mask-guided and hyperparameter formulations outperform the permutation-based sequence pair, and evolutionary algorithms (Vanilla-EA, PSO) outperform simulated annealing and Bayesian optimization, especially in high-dimensional spaces. The central empirical claim is that the best configurations—notably hyperparameter-optimized runs of the analytical placer DREAMPlace—achieve better wirelength on the ISPD 2005 and ICCAD 2015 suites than representative analytical and reinforcement-learning baselines. A sympathetic reader would take away that BBO is a viable tool for macro and global placement, and that the benchmark provides a shared testbed for further algorithm development.","feed_headline":"Black-box optimization rivals top placers in new benchmark","feed_subtitle":"Shared benchmark: hyperparameter-tuned DREAMPlace and mask-guided EAs beat analytical and RL placers on wirelength.","key_machinery":"The load-bearing machinery is a decoupled framework that splits chip placement into problem formulation (how a genotype becomes physical coordinates), optimization algorithm, and evaluation. The three formulations are the core objects: sequence pair (two permutations decoded by longest common subsequence into macro coordinates), mask-guided optimization (grid coordinates refined by a wire-mask greedy decoder that places macros on the lowest incremental-wirelength grid), and hyperparameter optimization (a solution is a full configuration of the analytical placer DREAMPlace, and running that placer is the decoding step). The framework standardizes two industrial chip suites and three metric ti","core_discovery":"The paper's central discovery is a layered comparative result. On six ISPD 2005 and eight ICCAD 2015 cases, with five seeds and fixed evaluation budgets (10,000 evaluations for macro-placement HPWL, 200 for global-placement HPWL), the sequence-pair formulation consistently ranks last, while mask-guided optimization (MGO) and hyperparameter optimization (HPO) dominate it. Across algorithms, Vanilla-EA and PSO are the strongest optimizers, particularly as the number of macros grows; Bayesian optimization struggles in high-dimensional MGO spaces but is competitive in the low-budget HPO setting. The headline numbers: HPO-Vanilla-EA and HPO-PSO reach average ranks of 2 and 2.67 on ISPD GP-HPWL, b","pith_inferences":["Because the HPO search space is exactly DREAMPlace's hyperparameter space, an independent reading is that the 'BBO beats analytical placer' result is partly 'tuned DREAMPlace beats default DREAMPlace'; a starker test would use a different analytical placer as the HPO decoder and as the GP-HPWL evaluator for all formulations.","The benchmark's decoupling makes it straightforward to reuse as a testbed for meta-BBO, algorithm configuration, or surrogate modeling on a hard real-world objective—none of which the paper itself explores.","A testable extension: freeze the hyperparameters found by HPO and run each macro placement through DREAMPlace with those settings; if the wirelength gap over MGO-EA persists, the gain is genuinely in macro placement, not just in hyperparameter tuning.","Another extension: vary the standard-cell placer in the GP-HPWL evaluation while keeping macro placement fixed, to measure how much of the reported advantage is downstream-engine-specific."],"forward_implications":["If the results hold, BBO can field competitive solutions for both macro placement and global placement on industrial-scale chips, not just toy circuits.","BBOPlace-Bench gives the BBO community a real-world, high-dimensional, and expensive-evaluation testbed where new algorithms can be compared under identical budgets and metrics.","The HPO formulation's success indicates that tuning a strong analytical placer's hyperparameters is itself a powerful placement strategy, worth pursuing alongside direct macro-placement search.","The weak correlation between GP-HPWL and timing metrics (WNS) warns that wirelength gains do not automatically translate into PPA gains; multi-objective BBO is the next target.","At 200 evaluations, the GP-HPWL setting is an expensive-optimization testbed, and the fact that BO did not dominate there suggests a gap between BO's theoretical sample efficiency and practical performance on placement objectives."],"fun_headline_variants":["Black-box optimization rivals top placers on chip benchmarks","BBOPlace-Bench: Black-box optimization competes with analytical placers","Mask-guided GA challenges analytical and RL placers in BBO benchmark","New benchmark shows black-box optimization can rival standard chip placers","Black-box optimizers go toe-to-toe with chip placement baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparisons assume DREAMPlace is an impartial downstream evaluator for every formulation, even though the HPO formulation's entire search space is DREAMPlace's own configuration space; if DREAMPlace favors its own tuned configurations, the headline rankings would shift.","fun_headline_variants_meta":{"raw":{"variants":["Black-box optimization rivals top placers on chip benchmarks","BBOPlace-Bench: Black-box optimization competes with analytical placers","Mask-guided GA challenges analytical and RL placers in BBO benchmark","New benchmark shows black-box optimization can rival standard chip placers","Black-box optimizers go toe-to-toe with chip placement baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001548,"raw_usage":{"total_tokens":6075,"prompt_tokens":841,"completion_tokens":5234,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":5144}},"tokens_in":585,"tokens_out":5234,"duration_ms":36363,"temperature":1.0,"reasoning_tokens":5144,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:49:21.734606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five algorithms under the same protocol but replace DREAMPlace with a different analytical placer as both the GP-HPWL evaluator and the HPO decoder; if HPO loses its ranking advantage over MGO and the RL/analytical baselines, the paper's central comparative result is an artifact of DREAMPlace's hyperparameter sensitivity rather than a general BBO advantage.","supporting_citations":[],"review_version":1}