{"id":"f432d93b-e295-4426-833d-acbe8e923da9","arxiv_id":"2607.16127","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new Python framework treats cost as a first-class primitive in autonomous discovery, metering queries and decisions against a shared budget and recording declared vs. settled costs in an append-only ledger.","lead":"This paper presents CADAQUES, an open-source Python framework that makes resource cost a core part of autonomous discovery loops, with separate protocols for deciding and answering queries. It argues that tracking how much each query and each planning decision costs—not just iteration counts—should be a standard part of scientific campaigns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-resource vector budget — the core R1–R4 differentiator — is never exercised end-to-end; only wall time is capped, so token/euro/CPU-hour accounting claims rest on untested code paths.","rationale":"We read the paper as an architecture contribution: the central claim is that representing cost as a vector-valued, transaction-level primitive with declared/settled separation enables fair resource-normalized comparison and budget-adaptive drivers. The evaluation demonstrates this on a single wall-time-budgeted Ising task and shows the framework works for that case. The most load-bearing assumption for the central claim is that the multi-resource accounting (tokens, euros, CPU hours) actually behaves as specified; the paper provides unit tests and code but no end-to-end campaign with a non-time cap. This is explicitly conceded in Section 7.2. It is not an internal inconsistency, but it limits the evidence for the paper's differentiating features. The RQ3 fidelity result is secondary, exploratory, and properly hedged. We therefore agree with the reader's weakest assumption and recommend no change to the CONDITIONAL verdict. A concrete multi-resource campaign would turn the condition into a stronger acceptance.","tokens_in":17280,"tokens_out":5074,"duration_ms":43295,"concrete_test":"Run a multi-resource campaign that caps both seconds and tokens (or euros) as binding constraints. For example, use an Oracle whose declared cost is Cost(seconds=0.05, tokens=10) and a Driver whose last_proposal_cost reports tokens (or set a token-consuming mock). Verify (1) termination occurs when the token cap is exhausted even if seconds remain, and vice versa; (2) the ledger records both components for every Oracle and Driver transaction; (3) the BudgetView.fraction_used reflects the maximum over capped components; and (4) a cost overrun in one component does not prevent charging of the other component. This can be done with the provided package and a few dozen lines of test code. If the semantics hold, the architecture's core claim is supported; if not, the R1–R2 claims need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central architectural claim is that CADAQUES treats cost as a first-class primitive through vector-valued budgets, symmetric charging of Oracle and Driver, and declared vs. settled cost separation (R1–R5). However, the evaluation never caps any resource component other than 'seconds'. RQ2 and RQ3 use only wall-time budgets; the token, euro, and CPU-hour components are never used as binding caps. The vector dominance logic, termination when a non-time component is exhausted, and multi-resource ledger aggregation are therefore validated only by unit tests and code inspection, not by end-to-end campaign execution. This matters because the framework's novelty over existing optimization libraries (which also support cost-aware acquisition and multi-fidelity) is precisely the campaign-level multi-resource accounting with common charging of deliberation and queries. If the vector-budget semantics contain an error not surfaced by unit tests — e.g., in how partial ordering interacts with unconditional settlement, or how last_proposal_cost self-reported values merge with runner-measured wall time — the central R1–R5 claims would fail in the very settings the paper motivates (LLM token budgets, monetary costs). The paper's Section 7.2 explicitly concedes 'does not demonstrate ... a genuinely multi-resource campaign'; this is an honest limitation, but it means the strongest differentiating feature is unvalidated. The RQ3 two-phase result, while statistically exploratory (p=0.038, 10 seeds, finite-size bias comparable to errors), is clearly flagged as such and is less load-bearing than the unexercised core architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CADAQUES, an open-source Python framework whose central principle is that cost should be a first-class primitive of the autonomous-discovery loop. It separates the loop into an Oracle (price/evaluate) and a Driver (propose/observe), charges both against a common vector-valued budget with components for seconds, CPU hours, euros, and tokens, distinguishes declared (ex-ante) from settled (ex-post) costs, and records every transaction in an append-only ledger. The evaluation uses a noisy 2D Ising susceptibility task with a known exact critical temperature: three protocol-conformant drivers are compared under a 10 s wall-time budget (RQ2), three fidelity policies are compared under the same budget (RQ3), and metering overhead is microbenchmarked (RQ4). The paper reports that random search is competitive with the two concentrating strategies, that a two-phase low-fidelity/high-fidelity schedule has the smallest worst-case error in the studied setting, and that transaction-level metering adds about 25 µs per query. The code, tests, and reproduction scripts are shipped and archived at Zenodo.","tokens_in":17646,"tokens_out":4838,"duration_ms":43744,"significance":"If the architecture holds, CADAQUES provides a compact and interoperable contract for budget-governed discovery with transparent accounting, which is a useful complement to existing optimization libraries. The manuscript's strengths include a machine-checked implementation with unit tests encoding the central budget/ledger semantics, a carefully hedged empirical study with honest reporting of negative results (e.g., GP-EI and annealing do not beat random search under the common budget), and an explicit limitations section. The paper is also unusually candid about the exploratory nature of its statistical comparisons. However, the paper's identified differentiator over prior work is the vector-valued, multi-resource campaign abstraction, and this is exactly the part that is not exercised end-to-end anywhere in the evaluation. As presented, the empirical support for the framework's main novel claim rests on code inspection and unit tests rather than on any campaign that uses a binding non-wall-time resource cap.","major_comments":[{"comment":"The evaluation never runs a campaign with a binding cap on any resource other than 'seconds'. RQ2 and RQ3 cap only wall time; the token, euro, and CPU-hour components never constrain a campaign. Consequently, the framework's distinctive R1–R5 vector-budget semantics — componentwise dominance, termination on a non-time cap, and multi-resource ledger aggregation — are validated only by unit tests and code inspection, not by end-to-end use. Since Section 6 positions CADAQUES against libraries that already provide cost-aware acquisition and multi-fidelity methods, the paper's novelty rests on precisely these unexercised paths. To substantiate the central claim, the paper should add at least one end-to-end demonstration with a non-time cap (e.g., a synthetic token or euro budget), or explicitly scope the validated claims to wall-time accounting and present vector budgets as a design provision","section":"Section 5; Section 7.2"},{"comment":"The headline RQ3 result (two-phase policy better than fixed high fidelity, one-sided Mann–Whitney p=0.038) is based on ten seeds, one budget scale, and multiple comparisons, which the paper acknowledges. More seriously, the low-fidelity and two-phase arms are scored on different estimators: Section 5.5 states that the finite-size shift for L=16 is comparable to the median errors reported in Table 3. This estimator difference confounds the comparison as a statement about fidelity economics. The conclusion is appropriately hedged in the summary, but the main text should make the confound more prominent than a single sentence in threats to validity, because otherwise the quantitative comparison in Table 3 is easy to over-read.","section":"Section 5.3, Table 3; Section 5.5"}],"minor_comments":[{"comment":"The vertical axis label reads \"|T* Tc|\"; it should be \"|T* − Tc|\".","section":"Figure 4(b)"},{"comment":"The displayed equation for the exact critical temperature is not typeset correctly in the preprint; the expression \"Tc = 2 / ln(1 + sqrt(2))\" should appear as a proper fraction.","section":"Section 4.2"},{"comment":"The termination rule charges the driver's proposal before the affordability of the ensuing query is checked, so a driver can exhaust the budget on proposals that are then rejected. This is a defensible design, but the resulting failure mode (budget exhausted after zero or very few oracle evaluations) is not discussed. A sentence in Section 3.3 or Section 7.2 would help.","section":"Section 3.3"},{"comment":"The sentence \"A one-sided Mann–Whitney U test of the alternative that GP-EI yields smaller errors than Random gives p=0.92\" is potentially confusing because p>0.5 means the data point in the opposite direction; consider restating as \"no evidence for the alternative (p=0.92, in the direction of Random).\"","section":"Section 5.2.3"},{"comment":"The overhead benchmark uses 5,000-query runs with an AnalyticOracle with zero declared cost; it would be useful to state explicitly whether the 4.0 µs bare-loop baseline includes a query evaluation or only the sampling loop, since the comparison drives the claim of three orders of magnitude headroom.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-engineered and unusually honest about its limitations; the code and tests are a genuine strength. My main reservation is that the claimed novelty over existing cost-aware libraries is the multi-resource vector-budget abstraction, and the evaluation does not exercise any non-wall-time cap. This is fixable within the manuscript's scope — a simple synthetic multi-resource campaign or a clear reframing of the validated claims would suffice — but it is load-bearing for the paper's central contribution. I would encourage the editor to request that revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nRead CADAQUES. The short version: the framework is clean, honest, and genuinely useful as a shared accounting layer for autonomous-discovery campaigns, but the paper's core differentiator—the vector-valued, multi-resource budget—is never exercised end-to-end. Only wall time is capped; tokens, euros, and CPU hours appear only as code paths and unit-test targets. The paper explicitly concedes this in Section 7.2, which is good, but the limitation sits at the center of the contribution, not at the periphery.\n\nWhat is new: the integration of R1–R5 into one protocol-level contract—typed resource vectors, declared vs. settled costs per transaction, symmetric charging of driver and oracle, budget-governed termination, and an append-only ledger. The pieces exist in BoTorch, Hyperband, etc., but not as a campaign-level abstraction. The implementation is small (761 lines, one dependency), structural typing is a nice touch, and the unit tests cover the budget and ledger semantics. The evaluation is transparent: the Ising task is a fair stress test, the stale-calibration overrun (652%) is a good demonstration of declared/settled divergence, and the overhead microbenchmark (~25µs/query) is credible.\n\nThe soft spots are real but mostly acknowledged. RQ2 shows no strategy beats random at this budget, and the paper says so—random's median error is actually lowest. RQ3's two-phase result (p=0.038, 10 seeds) is flagged as exploratory, as is the finite-size bias comparable to the errors. The more serious issue is that the vector budget is never exercised with non-time components. The dominance logic, partial ordering, and termination-when-a-token-or-euro-cap-hits are only unit-tested. If any of those code paths has a semantic bug, the R1–R4 claims fail in the very settings the paper motivates (LLM token budgets, facility costs). The paper's own limitations section says 'does not demonstrate ... a genuinely multi-resource campaign,' so the authors know.\n\nMy take: the paper deserves a serious referee, but the referee should push for either a simple synthetic multi-resource campaign (e.g., a fake oracle with token and euro costs) or a narrowing of the claims in the abstract and contributions. As it stands, the headline feature is plausible but unsupported by the evaluation.\n\nI'd bring this to reading group: it's a good example of honest software-engineering claims and a useful caution about scope creep. I'd cite it when discussing resource-normalized benchmarking, but I wouldn't use it yet as evidence that multi-resource accounting works in practice.","headline":"A clean, honest software-architecture paper whose central multi-resource accounting feature is never exercised end-to-end; worth refereeing, but the headline claim needs either a synthetic multi-resource campaign or narrower wording.","tokens_in":18121,"tokens_out":2495,"would_cite":true,"duration_ms":20816,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating cost as a first-class primitive, the paper's CADAQUES framework charges queries and planning against one vector budget, and its two-phase fidelity schedule beats uniform high fidelity on a noisy Ising task.","keywords":["autonomous discovery","self-driving laboratories","cost-aware optimization","scientific software architecture","Bayesian optimization","provenance","LLM agents"],"falsifier":"Run the same Ising task with a binding token or monetary cap in addition to wall time, with a driver whose proposals consume tokens: if the campaign does not terminate at the cap or the ledger fails to record the token settlements as settled costs, the central vector-budget claim is falsified. Alternatively, repeat RQ3 at a much larger budget (e.g., 100 s) and check whether fixed high fidelity overtakes the two-phase schedule — if it does, the reported advantage is scale-dependent, not structural.","tokens_in":17162,"feed_emoji":"⏱️","tokens_out":5893,"duration_ms":52276,"temperature":0.7,"pith_summary":"The paper argues that autonomous-discovery software has been shaped by numerical-optimization conventions that hide the real cost of experiments and decisions, and it proposes an architecture in which cost is a first-class primitive of the discovery loop. Its central move is to split the loop into an Oracle that answers queries and a Driver that proposes them, then charge both against one vector-valued budget with components for wall time, CPU hours, money, and language-model tokens. Each transaction records the cost declared before execution and the cost settled after, in an append-only ledger, so the discrepancy between prediction and reality becomes data. On a deliberately noisy physics task — locating the critical temperature of the two-dimensional Ising model from finite-size susceptibility estimates — the paper reports that a two-phase schedule of cheap low-fidelity exploration followed by expensive high-fidelity refinement yields lower and less variable final errors than using high fidelity throughout, at the same nominal ten-second wall-time budget. A sympathetic reader would care because, if this holds, resource-normalized comparison of search strategies, planner costs included, becomes possible from a single machine-readable record rather than from analyst reconstruction.","feed_headline":"Two-phase fidelity search beats high fidelity at equal budget","feed_subtitle":"A new cost-aware framework meters queries and planner decisions in one ledger, making resource-normalized comparison possible in autonomous","key_machinery":"The load-bearing object is the vector-valued Cost type — a frozen record over (seconds, cpu_hours, euros, tokens) with componentwise dominance — together with the Budget that caps any subset of these components and the append-only Ledger of transactions. The Oracle/Driver protocols are the structural boundary: any object conforming to price/evaluate or propose/observe can participate, and the runner funnels both parties through the same charging path. The ledger's split of declared and settled costs is what turns cost-model error into observable campaign data, and the read-only BudgetView is what lets a Driver condition its behavior on remaining resources without being able to spend them dir","core_discovery":"CADAQUES is presented as a compact open-source Python framework whose design principle is that cost belongs inside the discovery loop. The required integration surface is two structural protocols: an Oracle exposes price(query) and evaluate(query), and a Driver exposes propose(history, budget_view) and observe(result). The runner meters the Driver's deliberation in wall time, charges both the Driver and the Oracle against a common vector-valued Cost (seconds, cpu_hours, euros, tokens) with componentwise caps, and appends every transaction to a ledger with declared and settled costs. The paper's empirical centerpiece is a search for the critical temperature of the 2D Ising model from noisy fi","pith_inferences":["If the vector-budget semantics generalize from wall time to bind tokens, money, and CPU hours, the same ledger could support comparisons across campaign types where resource valuations differ — but that requires the planned user-extensible component set, since the current fixed vocabulary omits GPU hours, energy, and instrument time.","The RQ2 failure mode suggests a natural follow-up: wrap concentrating drivers with noise-robust incumbent handling or repeated re-estimation before scoring, which the paper itself flags as a possible alternative; this should change the observed ordering.","The two-phase schedule is structurally similar to budget-allocation multi-fidelity schedulers used in hyperparameter optimization, so a Driver implementing one of those schedulers inside CADAQUES would turn its budget allocation into explicit, auditable campaign accounting.","Because settled costs can exceed declarations by a large factor (median 652% in the stale-calibration test), practical deployments would likely need uncertainty margins or probabilistic affordability checks to prevent systematic cap overruns, which the paper lists as future work."],"forward_implications":["Strategy comparisons can be made under equal resource budgets rather than equal iteration counts, with resource-normalized best-so-far curves reconstructed from the ledger.","Planner deliberation stops being free: the cost of fitting surrogates or invoking language models is charged to the same budget as oracle queries, so expensive planners can be judged on whether they earn their keep.","Mismatches between declared and settled costs become a recorded property of each transaction, enabling post-hoc cost-model calibration and overrun analysis without separate logging.","Budget-adaptive drivers are expressible through the read-only BudgetView, and multi-fidelity schedules can be represented at the type level via explicit fidelity fields on queries.","At the measured 10 s budget scale, spending less per query during exploration and more during refinement outperforms uniformly high-fidelity queries on a noisy objective, suggesting that fidelity allocation can matter as much as search-algorithm choice."],"fun_headline_variants":["Cost-aware loop: cheap first, refine later, lower error","Two-phase fidelity beats uniform high fidelity at equal cost","Metered discovery: low-fidelity then high, less error and variance","CADAQUES: cost as first-class primitive in discovery loops"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central architectural claim rests on the untested assumption that the full vector-valued multi-resource budget works as specified, since the paper's own limitations section concedes that no genuinely multi-resource campaign was run: RQ2 and RQ3 cap only wall time and use only the seconds component, leaving the heterogeneous-resource accounting paths unvalidated end-to-end.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware loop: cheap first, refine later, lower error","Two-phase fidelity beats uniform high fidelity at equal cost","Metered discovery: low-fidelity then high, less error and variance","CADAQUES: cost as first-class primitive in discovery loops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1393,"prompt_tokens":843,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":587,"tokens_out":550,"duration_ms":6257,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:15:13.511768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Ising task with a binding token or monetary cap in addition to wall time, with a driver whose proposals consume tokens: if the campaign does not terminate at the cap or the ledger fails to record the token settlements as settled costs, the central vector-budget claim is falsified. Alternatively, repeat RQ3 at a much larger budget (e.g., 100 s) and check whether fixed high fidelity overtakes the two-phase schedule — if it does, the reported advantage is scale-dependent, not structural.","supporting_citations":[],"review_version":1}