{"id":"9471ecc7-0ad1-40d6-aea4-be3f8e629dda","arxiv_id":"2608.08189","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Janus co-evolves LLM-generated proxy evaluators with target programs, reaching the same or better final performance with about 59% fewer real evaluations across five design benchmarks.","lead":"This paper introduces Janus, a framework where large language models evolve both the programs being designed and the cheap proxy evaluators that predict their quality, so that only promising candidates are sent to the expensive real evaluator. In five scientific and engineering design tasks, Janus reached the same final performance as a baseline using about 59 percent fewer real evaluations on average.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Janus's region-conditioned portfolios and novelty bonuses rest on an unvalidated cosine-distance similarity between code embeddings, an assumption the paper itself rejects in its Embed-GP baseline.","rationale":"The reader's weakest assumption identifies the same issue I find most worrying. It is not a manufactured objection: the paper's own critique of Embed-GP applies verbatim to the embedding distance used in Eqs. (4)-(6). The concern is load-bearing because the framework's advertised novelty is the region-conditioned, co-evolved evaluator portfolio; if the regions are behaviorally meaningless, the mechanism that is supposed to explain the savings is not doing the work. I considered whether the lack of significance tests and three seeds is more decisive. That is a real limitation, but it affects confidence in the magnitude of the gains, not the identity of the mechanism; the reader already counted it and issued CONDITIONAL. The embedding issue is the more precise scientific weakness because it targets a stated design assumption and has a direct, cheap experimental resolution. I therefore recommend no change to the reader's verdict: conditional acceptance pending a direct test of the embedding assumption and release of implementation details. The proposed test would settle the concern within the existing benchmarks.","tokens_in":16539,"tokens_out":9435,"duration_ms":98400,"concrete_test":"Compute, for each of the five benchmarks, the Spearman correlation between pairwise cosine distances of code embeddings and pairwise behavioral distances on the pool of real-evaluated candidates, where behavioral distance is the L2 distance between standardized real-evaluator score components (or a direct utility difference where components are unavailable). If the correlation is weak (<0.3) in any domain, rerun Janus on that domain with region labels randomly permuted (preserving region count and sizes) and all other components fixed. If AUBC and Calls@99% do not degrade materially, the embedding-based region assignment is not load-bearing; if they do degrade, the concern is confirmed and the paper should validate or replace the similarity assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Janus's central efficiency gains are attributed to evaluator evolution and region-conditioned portfolio selection, yet the region definition and novelty bonus both depend on the claim that cosine distance in code-embedding space tracks behavior under the real evaluator. Eq. (4) assigns candidates to the nearest centroid in embedding space, Eq. (6) adds a novelty term proportional to that same distance, and the online credit updates in Eq. (5) are regional. If programs with similar source code do not behave similarly under the real evaluator, these regions are arbitrary, and portfolio credits and novelty-driven budget allocations are not aligned with the actual search landscape. The paper never specifies the embedding model, reports no correlation between embedding similarity and real-evaluator behavior, and never ablates the similarity measure. In fact, Section IV (Alternative proxy baselines) criticizes Embed-GP on exactly these grounds: 'semantic similarity may not preserve the task-specific numerical or physical behaviors governing performance.' This self-referential tension matters because the ablations show that removing evaluator evolution or online credits hurts performance; if the embedding assumption is false, the reported advantage may stem from other components (e.g., screening many LLM candidates per real evaluation) rather than from the region-conditioned co-evolution that the paper identifies as the mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Janus, a framework that co-evolves LLM-generated target programs and LLM-generated executable proxy evaluators under a limited real-evaluation budget. Proxy evaluators are calibrated on a real-outcome archive, selected by a promotion-aligned meta-objective, organized into region-conditioned portfolios with online credit updates, and used only to prioritize candidates; real validation is required before any program can enter the target population or update the incumbent. Across five program-design benchmarks (battery fast charging, AQM, FDTD demultiplexing, reactor control, and perishable inventory routing), the authors report that Janus achieves larger area under the best-so-far improvement curve over the real-evaluation budget and higher final performance than a matched algorithm-only baseline, reaching 99% of the baseline's final improvement with on average 59.1% fewer real evaluations. The paper also reports improvements in evaluator ranking accuracy from evaluator evolution and a component ablation study.","tokens_in":16713,"tokens_out":6160,"duration_ms":60984,"significance":"If the result holds, Janus is a useful step toward making LLM-driven program discovery applicable when ground-truth evaluation is expensive. The method is principled in its real-anchored design: proxy predictions only rank candidates, promotion requires real validation, and the meta-objective directly targets the downstream search goal rather than generic regression accuracy. The ablations in Table III show that each component contributes something, and the evaluator-ranking gains in Figure 6 are concrete. The paper is also honest about token costs and includes a Direct-LLM replication protocol. The headline claim is not circular, because the central metric is anchored to real-evaluator outcomes and candidates must pass real validation before affecting the population. The main risks are empirical and methodological: an unvalidated similarity assumption that underpins the proposed mechanism, a mismatch between the motivating expensive evaluations and the actual simulated oracles, and insufficient statistical support for the headline numbers.","major_comments":[{"comment":"The proposed mechanism—region-conditioned portfolios and the novelty term η(x)—rests on the assumption that cosine distance between code embeddings tracks behavior under the real evaluator, but the paper never specifies the embedding model, reports no correlation between embedding distance and real-evaluator agreement, and performs no ablation of the similarity measure. This is exactly the assumption the authors reject for Embed-GP when they write that 'semantic similarity may not preserve the task-specific numerical or physical behaviors governing performance.' Because the ablations in Table III show that evaluator evolution, portfolio selection, and online credits are all load-bearing, and because these components operate through the same embedding-defined regions, the reported gains cannot yet be attributed to the claimed mechanism. Please report the embedding model, provide a correlation analysis between dcos and real-evaluator score differences on held-out candidate pairs, and include an ablation that replaces cosine distance with an alternative (e.g., behavioral features or random region assignments).","section":"§IV-B, Eq. (4), Eq. (6), and Alternative proxy baselines"},{"comment":"The motivating examples are evaluations that take hours to days (Table I), but the five 'real' evaluators used in the experiments are lightweight simulations: 2D FDTD on a 480×320 grid, six 8-second ns-3 runs, 30-cycle PyBaMM at three temperatures, a stiff ODE, and a 36-episode IRP. The abstract's claim that Janus 'extends evaluator-guided LLM discovery... to scientific domains where trustworthy evaluation is scarce and expensive' is therefore not directly demonstrated; the observed 59.1% call reduction is measured under cheap, reproducible oracles. Either add at least one genuinely expensive evaluator (or a realistic cost model) to the benchmark suite, or explicitly restrict the claim to moderate-cost simulated oracles and discuss the transfer of the results to truly expensive settings.","section":"§IV-A and Table II vs. Table I"},{"comment":"Every method-domain condition is run with three random seeds and no significance tests or confidence intervals are reported. The headline mean reduction of 59.1% is computed across five domains with no uncertainty, and Table III reports overlapping standard deviations for several ablations (e.g., AQM rows S and A1). With n=3, the claims that Janus 'achieves' larger AUBC∆ in all five domains and that every ablated component is necessary are not strongly supported. Please provide per-seed results, paired tests or bootstrap intervals for the AUBC∆ and Calls@99% comparisons, and a statement of the variance of the 59.1% estimate across seeds and domains.","section":"§IV-A (Metrics) and §IV-B (Main Results)"},{"comment":"The values of the free parameters λ_R, λ_N, λ_ρ, κ, w_q, w_u, w_n, τ_p, τ_e, b, K, and the archive bounds are not reported in the main text or appendices. These parameters control the balance among exploitation, uncertainty exploration, and novelty, as well as the periods at which evaluators are evolved and refreshed; without them the method is not reproducible and the sensitivity of the headline result to these choices is unknown. Report a full hyperparameter table and a sensitivity analysis for at least the acquisition weights (w_q, w_u, w_n) and the evolution periods (τ_p, τ_e).","section":"§III-A, Algorithm 1, Eq. (3), Eq. (5), Eq. (6)"}],"minor_comments":[{"comment":"The abstract contains a typo: '99/% of the baseline's final improvement' should read '99% of the baseline's final improvement'.","section":"Abstract"},{"comment":"The factor α_{e,r} appears in Eq. (5) before it is defined; move the definition immediately before the equation to improve readability.","section":"§III-C, Eq. (5)"},{"comment":"The shading spans only the minimum and maximum of three seeds, which can suggest more stability than present; plotting the three individual runs or adding standard-error bands would be more informative.","section":"Figure 5"},{"comment":"The 'highest-weight evolved evaluator' is chosen post hoc from the evolved evaluator population; please describe how this selection is made (e.g., by Fmeta on the archive) and clarify that the 40 later candidates are temporally separated from that selection, since this is the basis for the reported NDCG@3 gains.","section":"§IV-C, Figure 6"},{"comment":"The paper never states which embedding model produces z(x); this information should be added to the reproducibility details, as the entire region-conditioning scheme depends on it.","section":"§III-B, Eq. (2)"},{"comment":"The sentence reporting a 'mean knee-location error of 0.500 tested sender-count positions' is confusing; clarify that this is 0.500 grid steps rather than a count of tested positions.","section":"§II-C"}],"recommendation":"major_revision","confidential_remarks":"I see no circularity problem in the main metric: real validation gates the population and the final claim is anchored to real outcomes. The central risk is empirical: the embedding-based region mechanism is unvalidated and is simultaneously the paper's main claimed source of improvement. The benchmark-cost mismatch and the lack of hyperparameter reporting are also serious for reproducibility. I would not accept the paper in its current form, but I believe all these issues can be addressed within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Janus is worth reading. The core combination is new and the design is sensible: LLM-generated executable proxy evaluators co-evolved with target programs, selected by a promotion-aligned objective, used via region-conditioned portfolios with online credits, and kept honest by real-anchored promotion. That last piece matters—candidates must pass real validation before entering the population, so the 59.1% reduction in real evaluations is not circular. The ablations on AQM and FDTD Demux support the component claims, and the ranking-accuracy gains from evaluator evolution are concrete.\n\nThe main soft spot is the embedding assumption. Eq. (4) assigns candidates to regions using cosine distance between code embeddings; Eq. (5) updates evaluator credits per region; Eq. (6) adds a novelty bonus from that same distance. The paper criticizes Embed-GP for assuming semantic similarity tracks behavior, yet never validates that its own embedding distances do. This is load-bearing for the region-conditioned portfolio and novelty term. It is not fatal—regions only gate credit allocation and novelty, not direct fitness prediction—but without a correlation check or an ablation of the embedding choice, the mechanism is under-supported.\n\nThe statistical evidence is also thin: three seeds per domain, no significance tests, no hyperparameter values, and the main LLM version is unnamed. No code or data are released. And the benchmarks are cheap simulations, not the hours-or-days pipelines in Table I; that is a mismatch with the motivation, though not a deal-breaker for testing the algorithm.\n\nIf I were the editor, I would send this to reviewers. The idea is good enough to deserve careful scrutiny. But I would ask for artifacts, more seeds with significance testing, hyperparameter reporting, and a direct validation or ablation of the embedding similarity. The central claim may survive that scrutiny, but right now it is conditional.","headline":"A genuinely new co-evolution framework with plausible headline savings, but the load-bearing embedding assumption and thin statistics make the result conditional.","tokens_in":17318,"tokens_out":3451,"would_cite":false,"duration_ms":34549,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evolving proxy evaluators cut real evaluations by 59.1%","keywords":["LLM-driven program discovery","co-evolution","proxy evaluators","surrogate models","expensive evaluation budgets","distribution shift","program search","scientific design"],"falsifier":"Take a set of candidates already scored by the real evaluator in one Janus run, cluster them by code-embedding cosine distance, and compare within-cluster variance in real utility to between-cluster variance. If within-cluster variance is comparable to between-cluster variance, then the region-conditioned portfolio and novelty term are assigning meaning to code distances that do not correspond to real behavioral differences.","tokens_in":16271,"feed_emoji":"🧪","tokens_out":8742,"duration_ms":75033,"temperature":0.7,"pith_summary":"Janus's central claim is that the evaluator, not just the target program, should be evolved when real evaluations are expensive. Instead of fitting a fixed surrogate to scarce search-biased labels, it asks an LLM to generate executable proxy evaluators from domain descriptions, calibrates their parameters on real outcomes, and keeps a portfolio of region-specific proxies whose credits are updated online. Proxy predictions are used only to prioritize which candidates get real evaluation; a candidate must pass real validation before it can enter the population or update the incumbent. Across five scientific and engineering design tasks, Janus reports a larger best-so-far improvement over the real-evaluation budget and higher final performance than a matched algorithm-only baseline, reaching 99% of the baseline's final improvement with 59.1% fewer real evaluations on average. If the claim holds, it would let LLM-driven discovery work where trust is scarce and each evaluation costs hours or days.","feed_headline":"Evolving proxy evaluators cut real evaluations by 59.1%","feed_subtitle":"Same final improvement with 59.1% fewer expensive evaluations across five scientific design tasks.","key_machinery":"The load-bearing mechanism is the evaluator-as-program representation combined with real-anchored promotion. An evaluator is a pair $(g_\\phi, \\theta_\\phi(\\mathcal{A}))$: a program structure generated by an LLM from the task description and an archive of real outcomes, plus fitted parameters calibrated on that archive. The framework clusters the current target-program population by cosine distance between code embeddings, assigns each new candidate to the nearest region, and scores it with the top-credit evaluators in that region under a promotion-aligned meta-objective that rewards recovering near-optimal archive programs in the predicted top-$k$. Online credit updates adjust each evaluator's regional weight from the real outcomes of the candidates it helped promote. A candidate's acquisition score combines predicted quality, predicted validity, portfolio uncertainty, and embedding-region novelty, and only the real evaluator can confirm or reject the candidate.","core_discovery":"The paper's central claim is that co-evolving evaluators with target programs transfers the benefits of evaluator-guided LLM search to settings with expensive real evaluation. The key move is to represent each proxy evaluator as an LLM-generated executable program with structure and fitted parameters, to score evaluators by how well their top-ranked candidates recover the best archive programs (a promotion-aligned objective), and to maintain region-conditioned portfolios with online credit updates as real outcomes arrive. Because proxies remain fallible, Janus never lets a proxy prediction promote a candidate alone: candidates are screened and ranked by the portfolio, but the real evaluator is the only authority that admits a program to the population or updates the incumbent. The evidence is five program-design domains with real evaluations ranging from full battery-cycle simulations to packet-level network simulations, where Janus reports higher area under the best-so-far improvement curve and higher terminal performance than the algorithm-only baseline, reaching the baseline's endpoint with 59.1% fewer real-evaluator calls on average. Evolved evaluators also rank promising candidates more accurately than their seed versions.","pith_inferences":["Inference: the paper's critique of its fixed-embedding Gaussian-process baseline also applies to its own region logic: cosine distance between code embeddings is never shown to track task-specific physical behavior. If similar code does not imply similar real-evaluator behavior, region-conditioned portfolios and novelty bonuses can misallocate the budget.","Inference: the reported savings are in real-evaluator calls; Janus spends 1.8–4.5 times as many LLM tokens per completed call as the algorithm-only baseline. The practical gain depends on real evaluations being so expensive that token cost is negligible, so on cheaper evaluation tasks the method could lose its edge.","Inference: a direct test of the framework's hypothesis would be to replace code-embedding similarity with behavior-based similarity from cheap low-fidelity simulations, which would separate the contribution of co-evolving evaluators from the contribution of the embedding similarity assumption."],"forward_implications":["On the five test domains, Janus beats the algorithm-only baseline in both budget-averaged and terminal improvement; the gap in budget-averaged improvement ranges from 1.6 to 45.1 percentage points across domains.","Janus reaches 99% of the baseline's final improvement with 16.7%–82.8% fewer real-evaluator calls, with a mean reduction of 59.1%.","Evolved evaluators rank held-out candidates more accurately than their seed versions in the three reported domains, with NDCG@3 gains of 0.282, 0.694, and 0.311.","Ablations that remove evaluator evolution, the portfolio, the promotion-aligned objective, or online credit updates all reduce both metrics, indicating the four components address complementary failure modes.","In an objective-reweighting test, a previously evolved evaluator transferred without new labels and achieved an NDCG@3 of 0.959, showing the proxy's component structure is not tied to a single objective."],"supporting_citations":[{"why":"Supplies the island-based LLM program-search loop Janus builds on, including the real-validated population and LLM offspring generation.","marker":"[6]"},{"why":"Establishes evaluator-guided program search with LLM generation, the paradigm Janus extends to expensive evaluation.","marker":"[5]"},{"why":"Represents the fixed-surrogate approach (the Embed-GP baseline) Janus is compared against and argues is vulnerable to distribution shift.","marker":"[24]"},{"why":"Provides the Direct-LLM baseline protocol in which the LLM predicts candidate quality from labeled history.","marker":"[27]"},{"why":"Supplies the battery fast-charging benchmark, one of the five real-evaluation tasks.","marker":"[13]"},{"why":"Supplies the network active-queue-management benchmark and its packet-level simulation as the real evaluator.","marker":"[14]"},{"why":"Supplies the photonic demultiplexing benchmark with FDTD simulation as the real evaluator.","marker":"[15]"},{"why":"Supplies the exothermic reactor-control benchmark with stiff-ODE integration as the real evaluator.","marker":"[16]"},{"why":"Supplies the perishable inventory-routing benchmark with stochastic simulation as the real evaluator.","marker":"[17]"}],"fun_headline_variants":["Janus: co-evolving evaluators cuts real eval cost by 59%","Co-evolve evaluators and programs: 59% fewer real evals","Janus: co-evolve programs and evaluators to save 59% real evals","Evolve evaluators alongside programs: 59% fewer real evaluations","Co-evolving proxies with programs: 59% less expensive evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that code similarity, measured as cosine distance between program embeddings, tracks the behavior the real evaluator cares about; Janus uses this distance to form regions, choose evaluators, and reward novelty, and the paper does not validate it.","fun_headline_variants_meta":{"raw":{"variants":["Janus: co-evolving evaluators cuts real eval cost by 59%","Co-evolve evaluators and programs: 59% fewer real evals","Janus: co-evolve programs and evaluators to save 59% real evals","Evolve evaluators alongside programs: 59% fewer real evaluations","Co-evolving proxies with programs: 59% less expensive evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001054,"raw_usage":{"total_tokens":4474,"prompt_tokens":1042,"completion_tokens":3432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":3330}},"tokens_in":658,"tokens_out":3432,"duration_ms":22939,"temperature":1.0,"reasoning_tokens":3330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:17:25.175360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of candidates already scored by the real evaluator in one Janus run, cluster them by code-embedding cosine distance, and compare within-cluster variance in real utility to between-cluster variance. If within-cluster variance is comparable to between-cluster variance, then the region-conditioned portfolio and novelty term are assigning meaning to code distances that do not correspond to real behavioral differences.","supporting_citations":[{"cited_title":"Mathematical discoveries from program search with large language models,","cited_arxiv_id":null,"evidence_quote":"Establishes evaluator-guided program search with LLM generation, the paradigm Janus extends to expensive evaluation."},{"cited_title":"Fast charging of Li-ion batteries via learning and optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the battery fast-charging benchmark, one of the five real-evaluation tasks."},{"cited_title":"DESiRED: Dynamic, enhanced, and smart iRED: A P4-AQM with deep reinforcement learning and in-band network telemetry,","cited_arxiv_id":null,"evidence_quote":"Supplies the network active-queue-management benchmark and its packet-level simulation as the real evaluator."},{"cited_title":"Inverse-designed low-crosstalk CWDM (de)multiplexer assisted by photonic crystals,","cited_arxiv_id":null,"evidence_quote":"Supplies the photonic demultiplexing benchmark with FDTD simulation as the real evaluator."},{"cited_title":"A hierarchical multi-parametric programming approach for dynamic risk-based model predictive quality control,","cited_arxiv_id":null,"evidence_quote":"Supplies the exothermic reactor-control benchmark with stiff-ODE integration as the real evaluator."},{"cited_title":"Inventory management and delivery of perishable products with stochastic demands and risks consideration,","cited_arxiv_id":null,"evidence_quote":"Supplies the perishable inventory-routing benchmark with stochastic simulation as the real evaluator."}],"review_version":1}