{"id":"3349b986-23dd-4295-a643-7368a3858d9c","arxiv_id":"2608.04999","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ORACLE combines preference-conditioned multi-objective DDQN with LLM-guided action masking to size analog circuits, reporting high pass rates and large runtime cuts, but the evaluation compares 10 solutions per target against 1 for a key baseline.","lead":"ORACLE is an open-source reinforcement learning system that sizes analog circuits by optimizing several specifications at once, guided by a preference vector and an LLM that prunes bad design moves. A generalist would read it to assess whether vector-valued multi-objective RL plus LLM masking can reduce manual analog design effort, though the reported speedups rely on an asymmetric evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ORACLE's headline pass-rate, FoM, and runtime claims rest on an internally inconsistent and asymmetric evaluation: reported numbers contradict each other and AutoCKT gets one try vs ORACLE's ten.","rationale":"The reader identified the asymmetric comparison and the generalization assumption as the weakest points; this stress-test finds an even more direct problem: the paper's own tables contradict each other on the headline numbers. The 99.9% pass-rate claim is not derivable from the reported per-topology pass rates, and the FoM values in Table I and Table III for the same method and benchmark violate the stated best-of-10 definition. These arithmetic inconsistencies are not a matter of interpretation or consensus—they indicate that the quantitative claims are unreliable without access to the exact protocol and code. The method itself is a plausible engineering contribution: preference-conditioned vector-valued Q-learning with LLM-guided action masking is a reasonable design, and the authors promise an open-source release. The concern is not with the method's soundness but with the evidence for the abstract's quantitative claims. The reader's REJECT verdict is appropriate because the evaluation does not currently support the announced speedups, pass rates, or FoM gains. A fair re-run against AutoCKT with equal attempts, plus verification of the FoM definitions, would settle whether the claims survive. No ad hominem is intended; the issue is with the reported data, not the authors' integrity.","tokens_in":12884,"tokens_out":6419,"duration_ms":74054,"concrete_test":"Reproduce the two-stage OPAMP benchmark from the released repository with a matched protocol: for each of the 1,000 target specifications, run ORACLE (Cosine) and AutoCKT for exactly one episode each, and additionally run AutoCKT for 10 independent restarts per target. Recompute pass-rate, average FoM (verified as the mean over targets of the per-target best-of-10 FoM), and runtime including any training or fine-tuning cost. Also check the internal consistency condition: the mean over targets of the best-of-10 FoM must be at least as large as the mean FoM over all 10 solutions. If the ORACLE pass-rate or FoM advantage over AutoCKT collapses under matched attempts, or if Table I's 1.45 average FoM remains below Table III's 107.38, then the reported gains are artifacts of the asymmetric and inconsistent evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims of the paper—99.9% pass rate, 5.1x-to-318.6x FoM improvement, and 20.4x-to-104.4x runtime speedup—depend entirely on the evaluation in Tables I-VIII. That evaluation is internally inconsistent and asymmetric. First, the abstract claims a 99.9% pass rate over 2,000 target specifications, but the paper's own tables give 100% on the two-stage OPAMP (ORACLE Cosine, Table I) and 98.4% on the three-stage OTA (ORACLE Cos+LLM, Table V); the weighted average is (1000*100% + 1000*98.4%)/2000 = 99.2%, not 99.9%. Second, Table I reports ORACLE (Cosine) average FoM as 1.45 on the two-stage benchmark, while Table III reports the same method's average FoM as 107.38 over all 10,000 generated solutions. Section IV defines the reported FoM as the highest-FoM successful solution per target among the 10 generated. The mean of per-target maxima cannot be lower than the mean over all solutions for the same targets unless the two tables use different definitions or contain an arithmetic error; no explanation is given. Third, the pass-rate and FoM comparisons are structurally asymmetric: AutoCKT produces exactly one solution per target, whereas ORACLE produces ten and is credited with a pass if any one of the ten satisfies all constraints (Section IV). The paper acknowledges this issue for the Bayesian baselines, which are run with 10 solutions, but does not give AutoCKT the same number of attempts. Therefore the headline improvements over AutoCKT—including the 318.6x FoM figure—are not supported by the data as reported. The runtime speedup numbers also omit training time, but the internal contradictions alone are sufficient to invalidate the quantitative claims until the evaluation is corrected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents ORACLE, a multi-objective reinforcement learning framework for analog circuit sizing. The method uses a vector-valued reward and a preference-conditioned double DQN, with two action-selection scalarizations (normalized-weight and cosine-aligned) and an LLM-based action-masking module. The authors report experiments on two topologies with 1,000 target specifications each, claiming a 99.9% pass-rate, 20.4x-104.4x runtime reductions, and 5.1x-318.6x FoM improvements over AutoCKT, MODEBI, and ABCMOBO.","tokens_in":13222,"tokens_out":5909,"duration_ms":65338,"significance":"If the empirical claims held, ORACLE would be a worthwhile contribution to analog design automation: it addresses a real limitation of scalar-reward RL, provides preference-conditioned inference without retraining, and open-sources the framework. The vector-valued Q-learning formulation and the attempt to combine LLM-based design knowledge with RL exploration are timely. The evaluation, however, is currently too inconsistent and asymmetric to support the headline numbers, so the significance of the results is not yet established.","major_comments":[{"comment":"Table I reports ORACLE (Cosine) average FoM as 1.45 on the 1,000 two-stage targets, while Table III reports the same method's average FoM as 107.38 over all 10,000 generated solutions (10 per target). Since Section IV defines the reported FoM for a target as the highest-FoM successful solution among the 10, the per-target maximum cannot be lower than the average over all 10 solutions for the same targets. Unless the two tables use different FoM definitions, one of these numbers is wrong; no explanation is given. This internal inconsistency directly undermines the FoM improvement claims, including the 318.6x figure in the abstract.","section":"Section IV, Tables I and III"},{"comment":"The abstract's 99.9% pass-rate over 2,000 specifications is not supported by the tables: Table I gives 100% for ORACLE (Cosine/Cos+LLM/NW) on the two-stage OPAMP and Table V gives 98.4% for ORACLE (Cos+LLM) on the three-stage OTA, so the weighted average is (1000x100% + 1000x98.4%)/2000 = 99.2%. Table III, which reports pass-rates across all 10,000 generated solutions, gives 99.19%-99.80%. The manuscript should state exactly which pass-rate statistic is being reported and correct the 99.9% claim.","section":"Abstract and Section IV"},{"comment":"The comparison is asymmetric: ORACLE produces 10 solutions per target and is credited with a pass if any of the 10 meets all constraints, while AutoCKT returns exactly one solution. The hypervolume comparison in Tables IV and VIII is therefore between a 10-point front and a single point, which biases the comparison in ORACLE's favor. To support the claimed 5.1x-318.6x FoM improvements, the authors must give AutoCKT (or any single-solution baseline) the same number of attempts, or compare per-solution statistics such as best-of-k with matched k or the distribution of per-solution pass-rates.","section":"Section IV, Tables I, IV, V, VIII"},{"comment":"The runtime comparison is not clearly defined: it is unclear whether the reported runtimes include training, how many runs or seeds were averaged, and whether all methods use the same simulation environment and action space. ORACLE is trained once on 50 target specifications, while the Bayesian baselines and AutoCKT are run per target; if training time is excluded, the comparison is not apples-to-apples. Please specify the measurement protocol and report variance.","section":"Section IV-A, Tables II and VII"},{"comment":"The LLM-guided masking module is a central contribution, but the paper does not provide the prompt template, the numerical thresholds for the gap discretization in Eq. (9), or the exact masking rules. Table I attributes a 91x FoM improvement to the LLM integration, yet the reader cannot reproduce or audit this result without these details. Please include them in an appendix or supplement.","section":"Section III-D and Section IV-A"}],"minor_comments":[{"comment":"The value of the constant epsilon in Eq. (4) is not specified; please provide the actual value used in the experiments.","section":"Section III-B, Eq. (4)"},{"comment":"The Q-network output dimension is stated as |A|x4, while the general formulation uses N objectives; please clarify whether N=4 for all experiments or whether the network output is generalized to arbitrary N.","section":"Section III-C"},{"comment":"The sentence 'ORACLE meets 100% of the target specifications, representing a 1,011% - 6.6% improvement over the SOTA' is confusing because the range mixes relative improvements from different baselines; please rephrase with explicit per-baseline comparisons.","section":"Section IV-A"},{"comment":"The sentence 'ORACLE (NW) achieves 302x - 1.17x better top-20 FoM compared to ORACLE (Cosine) and ORACLE (Cosine + LLM), respectively' is grammatically ambiguous; please rewrite to make clear which baseline corresponds to which factor.","section":"Section IV-A"},{"comment":"The legends and axis labels in Figures 3 and 5 are small and difficult to read; please increase the font size for publication.","section":"Figures 3 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper contains several self-citations by the same group, but that alone is not unusual. The more serious issue is that the empirical section, which is the basis for the main claims, needs to be redone or substantially revised: the internal FoM inconsistency, the unsupported 99.9% pass-rate, and the asymmetric comparison against AutoCKT all affect the paper's central conclusions. I recommend major revision and re-review rather than acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dominic,\n\nQuick take on ORACLE (2608.04999). The method is a real engineering combination—preference-conditioned vector-valued DDQN for analog sizing, plus LLM-guided action masking, tested on a 2,000-case benchmark with code released. The pieces borrow from PD-MORL and RoSE-Opt, but the specific package and the scale of the test set are new. If the numbers were right, this would be a useful step for analog EDA.\n\nThey are not right as reported. The evaluation is asymmetric: ORACLE gets ten attempts per target and is credited if any one passes; AutoCKT gets one. Hypervolume is compared across fronts of size 10 versus size 1. That alone caps the credible claims.\n\nThere are also internal contradictions that don't need a re-run to spot. The abstract says 99.9% pass rate over 2,000 targets; the tables give 100% on the two-stage OPAMP and 98.4% on the three-stage OTA, which averages to 99.2%. More telling: Table I reports ORACLE (Cosine) average FoM as 1.45 on the two-stage benchmark; Table III reports the same method's average over all 10,000 solutions as 107.38. Since Table I is supposed to be the per-target best FoM, its average cannot be below the average over all solutions. The two tables cannot both be right under the definitions given. The runtime numbers exclude training cost, which for a trained-policy method is a large part of real wall-clock time, and there are no error bars or seed variance anywhere.\n\nThe method itself isn't the problem. The generalization claim—training on 50 specs and testing on 1,000 unseen ones—is worth a proper test. The LLM masking is sensible. The open-source artifact is a plus. But the empirical claims as written are not trustworthy; they need a corrected evaluation: one-solution-to-one-solution or equal retries, training time included, and reproducibility artifacts with exact hyperparameters, seeds, and LLM prompts.\n\nWho this is for: people in analog EDA or MO-RL applications will find the framework worth reading, but they should ignore the headline ratios. It deserves a serious referee because the core idea is legitimate and the benchmark has value, but I'd send it back for major revision, not accept it.\n\nBest,\n[You]","headline":"Plausible framework, but the headline pass-rate, FoM, and speedup numbers don't survive contact with the paper's own tables.","tokens_in":13838,"tokens_out":3995,"would_cite":false,"duration_ms":36953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ORACLE claims a single preference-conditioned RL agent can size analog circuits to 99.9% of 2,000 target specs with 20-104x less runtime than prior methods.","keywords":["analog circuit design automation","multi-objective reinforcement learning","preference-conditioned RL","vector-valued Q-learning","LLM-guided action masking","circuit sizing","Pareto front","DDQN"],"falsifier":"Train ORACLE on the stated 50 specs, then test on 1,000 held-out specs drawn from the same ranges but with preference vectors sampled randomly instead of the fixed 10. If pass-rate drops well below 99% or average FoM becomes comparable to single-objective AutoCkt, the preference-conditioning generalization claim fails. Separately, run the cosine variant with the LLM mask disabled: if pass-rate and runtime barely change, the LLM component is not carrying the reported 91x FoM gain.","tokens_in":12523,"feed_emoji":"⚡","tokens_out":6290,"duration_ms":61886,"temperature":0.7,"pith_summary":"This paper argues that analog circuit sizing should be treated as a multi-objective optimization problem and that reinforcement learning can solve it without collapsing trade-offs into a single number. ORACLE keeps the reward as a vector, conditions one Q-network on a designer-supplied preference vector, and uses an LLM to mask actions that are unlikely to help before each simulation. On two op-amp topologies with 2,000 target specifications, the authors report that a single trained model meets 99.9% of targets, cuts runtime by 20.4x to 104.4x versus state-of-the-art baselines, and produces multiple trade-off solutions per target from one training run. If the results hold, designers would no longer retrain for each new specification set and would get a Pareto front instead of a single operating point.","feed_headline":"One RL agent meets 99.9% of circuit specs, 104x faster","feed_subtitle":"ORACLE's preference-conditioned multi-objective policy replaces retraining with a preference vector and trims actions with an LLM.","key_machinery":"The machinery is a preference-conditioned deep double Q-network with vector outputs. The state is (z_t, z*, d_t), the reward is the normalized per-objective deviation of Eq. (4) rather than a scalar combination, and the network predicts Q_theta(s,a,w) in $R^{{|A| x 4}}$. Action selection uses either the cosine score Q_cos = cos(Q, w) * ||Q||_2 (directional alignment times magnitude) or the normalized weighted score Q_NW = w^T Q. Both feed the same DDQN update with target network and replay buffer. The LLM-guided mask of Eq. (10) restricts the action set to A' before simulation, using a local Llama model prompted with the discretized spec gap. The vector reward is the key design choice: it preserves per-objective information through training, so one policy can express many trade-offs.","core_discovery":"ORACLE's central claim is that preference-conditioned, vector-valued Q-learning can replace scalar-reward RL for analog sizing. The agent learns Q-values in $R^{4}$, one component per objective (gain, unity-gain bandwidth, phase margin, bias current), trained on 50 target specifications with 10 fixed preference vectors each. At inference, changing the preference vector re-targets the same network to a different trade-off region, producing 10 candidate designs per target spec without retraining. Two scalarization strategies are compared, cosine alignment and normalized weighted sum, and an LLM-based action mask filters actions before simulation. Across 2,000 test problems on a two-stage OPAMP and a three-stage OTA, the authors report 99.9% specification pass, higher Pareto hypervolume than AutoCkt, MODEBI, and ABCMOBO, and runtime reductions of 20.4x to 104.4x.","pith_inferences":["A natural stress test is to hold out entire topologies rather than just target specs: if the Q-function trained on the two-stage OPAMP can seed learning on the three-stage OTA without full retraining, the method would show genuine transfer of analog-sizing knowledge.","Sampling preference vectors continuously from the simplex, rather than a fixed set of ten, would reveal whether the cosine and NW scalarizations remain well-behaved outside the training distribution of weights.","The FoM of Eq. (12) is not normalized the same way as the reward of Eq. (4), so the reported 5.1x-318.6x FoM ratios partly depend on how close targets are to the boundary of the spec ranges; recomputing FoM on raw specifications would show how much of the gain is metric-driven.","The LLM mask's contribution could be isolated with a 2x2 ablation (mask on/off crossed with cosine/NW scalarization); the current tables vary both at once, so the 91x FoM jump attributed to the LLM may be confounded with the scalarization change."],"forward_implications":["A single ORACLE model replaces one-solution-per-retraining workflows: after training, the same network generates ten trade-off designs per target spec by enumerating ten preference vectors.","On the two benchmark topologies, average optimization runtime drops to 2.4 to 11.6 minutes versus 59.9 to 338 minutes for the compared baselines, enabling design-space exploration at interactive speed.","The recovered Pareto fronts for each target have higher mean hypervolume than AutoCkt, MODEBI, and ABCMOBO, meaning the method returns a spread of non-dominated designs rather than a single compromise.","LLM-guided action masking improves both reliability and speed by removing likely-useless actions before simulation, with the cosine+LLM variant raising average FoM from 1.45 to 132.3 on the two-stage OPAMP benchmark.","Because the implementation is released as open source, the same vector-valued training loop and masking module can be applied to other analog topologies by swapping the simulation environment."],"supporting_citations":[{"why":"the scalar-reward RL baseline and primary runtime/quality comparator; provides the Gym-style environment ORACLE builds on","marker":"[12]"},{"why":"source of the cosine-aligned preference guidance used for action selection","marker":"[14]"},{"why":"supplies the DDQN update, target network, and replay buffer used for vector Q-learning","marker":"[35]"},{"why":"an MO Bayesian optimization baseline generating 10 solutions per target for pass-rate and FoM comparison","marker":"[37]"},{"why":"an asynchronous batch constrained MO Bayesian optimizer used as the second MO baseline","marker":"[38]"},{"why":"prior work on knowledge-infused RL with action masking that motivates the LLM-guided filtering","marker":"[16]"}],"fun_headline_variants":["ORACLE: multi-objective analog design, 104x faster","Preference-conditioned RL meets 99.9% of circuit specs","LLM-guided action filter makes analog RL 104x faster","One trained model covers 2,000 spec cases without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a policy trained on only 50 target specifications with a fixed set of 10 preference vectors per target will generalize to 1,000 unseen target specifications per topology without retraining, and that the normalized reward makes gain, bandwidth, phase margin, and bias current commensurable so preference-weighted scores reflect real design priorities.","fun_headline_variants_meta":{"raw":{"variants":["ORACLE: multi-objective analog design, 104x faster","Preference-conditioned RL meets 99.9% of circuit specs","LLM-guided action filter makes analog RL 104x faster","One trained model covers 2,000 spec cases without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1446,"prompt_tokens":1043,"completion_tokens":403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":659,"tokens_out":403,"duration_ms":4798,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:51:13.980174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ORACLE on the stated 50 specs, then test on 1,000 held-out specs drawn from the same ranges but with preference vectors sampled randomly instead of the fixed 10. If pass-rate drops well below 99% or average FoM becomes comparable to single-objective AutoCkt, the preference-conditioning generalization claim fails. Separately, run the cosine variant with the LLM mask disabled: if pass-rate and runtime barely change, the LLM component is not carrying the reported 91x FoM gain.","supporting_citations":[{"cited_title":"Asynchronous Batch Constrained Multi-Objective Bayesian Optimization for Analog Circuit Sizing,","cited_arxiv_id":null,"evidence_quote":"an asynchronous batch constrained MO Bayesian optimizer used as the second MO baseline"},{"cited_title":"AutoCkt: Deep Reinforcement Learning of Analog Circuit Designs ,","cited_arxiv_id":null,"evidence_quote":"the scalar-reward RL baseline and primary runtime/quality comparator; provides the Gym-style environment ORACLE builds on"},{"cited_title":"PD-MORL: Preference- Driven Multi-Objective Reinforcement Learning Algorithm,","cited_arxiv_id":null,"evidence_quote":"source of the cosine-aligned preference guidance used for action selection"},{"cited_title":"Deep Reinforcement Learning with Double Q-Learning,","cited_arxiv_id":null,"evidence_quote":"supplies the DDQN update, target network, and replay buffer used for vector Q-learning"},{"cited_title":"Automated Circuit Sizing with Multi-Objective Op- timization Based on Differential Evolution and Bayesian Inference,","cited_arxiv_id":null,"evidence_quote":"an MO Bayesian optimization baseline generating 10 solutions per target for pass-rate and FoM comparison"},{"cited_title":"RoSE- Opt: Robust and Efficient Analog Circuit Parameter Optimization With Knowledge-Infused Reinforcement Learning,","cited_arxiv_id":null,"evidence_quote":"prior work on knowledge-infused RL with action masking that motivates the LLM-guided filtering"}],"review_version":1}