{"id":"63e46799-4578-4669-a68a-b47d2575cb87","arxiv_id":"2608.04336","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"COMPAS jointly optimizes model, prompt, and decoding settings separately for easy, medium, and hard code tasks, lifting LiveCodeBench pass@1 from 45.9% to 52.8% at 7.4x lower cost.","lead":"COMPAS is a new method that searches over which AI model, what prompt, and how the model samples code, then picks the best combination for each task difficulty level. On standard coding benchmarks it improves pass rate by about seven percentage points while cutting cost by more than seven times, and it also works on repository-level coding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's fresh-batch gate does not prevent overfitting under doubled search budget, and the default 1M budget was chosen by inspecting the v6-only test split; transferability of the learned fronts is therefore not yet established.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption they identify is the same one I would stress. My read of Table 7 and the surrounding text is that the paper itself demonstrates the Eq. 2 gate fails to prevent overfitting when the search budget is doubled: more search does not help, it actively hurts, because the gate commits to overfit candidates. That alone shows the method's transfer property is not guaranteed by construction. The additional aggravating point is methodological: the default budget was chosen by comparing 0.5M/1M/2M on the v6-only test split. Adaptive tuning of a core hyperparameter on the test set means the headline comparison is not a clean out-of-sample evaluation, even if the underlying method is sound. I therefore agree with the reader's assessment and do not move the verdict. The requested concrete test—a validation-split budget selection followed by a single v6-only evaluation—would settle whether the concern is real. If the gain persists, the paper's central claim stands; if not, the claim should be conditioned on validation-based budget selection.","tokens_in":15238,"tokens_out":5631,"duration_ms":54774,"concrete_test":"Choose the offline budget (0.5M/1M/2M) and all COMPAS defaults using only a validation split of release-v5 (e.g., 80/20), never touching v6-only; then run the full five-method comparison once on v6-only with the chosen budget. If the validation-selected budget is not 1M, or if COMPAS no longer significantly beats EcoTune/RouteLLM (paired Wilcoxon p<0.05), the headline 52.8% is at least partly a test-set-selection artifact. As a secondary check, re-run Double-Budget with five fresh seeds and report per-group overfitting to see whether the Eq. 2 gate self-corrects on any budget.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that per-group quality-cost fronts learned offline transfer to unseen tasks. COMPAS's only mechanism for this is the accept/reject gate in Eq. 2, which evaluates each candidate on a freshly resampled mini-batch and a freshly resampled held-out batch. The paper's own sensitivity analysis (Table 7, Budget) shows this mechanism is budget-fragile: doubling the offline budget from 1M to 2M tokens lowers Pass@1 from 52.8% to 46.6%, with two groups at 31.7% and 18.0% Pass@1 attributed to overfitting their search samples. Thus the gate does not generically prevent overfitting; it merely happens to at 1M tokens. Moreover, the 1M default was selected after inspecting the same v6-only test split used for the headline comparison ('1M tokens scores highest... we fix 1M tokens as COMPAS's default offline budget'), so the reported 6.9-point gain over EcoTune is not an evaluation on a truly unseen test set. The five-seed and random-split robustness results mitigate but do not remove this concern, because those runs also use the test-selected budget. A fair transfer claim requires selecting the budget on validation data and then evaluating once.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes COMPAS, a two-phase offline/online method for code-generation optimization. It first reports three empirical observations about prompt/decoding interaction, model-specific tuning effects, and difficulty-dependent best configurations. The offline phase performs a cheap model-selection probe followed by LLM-reflection-based joint search over prompts and decoding settings, with an accept/reject gate (Eq. 2) that evaluates candidates on freshly resampled mini-batches and a held-out pool; each difficulty(-type) group receives its own quality-cost Pareto front. Online, test tasks are assigned to groups via benchmark difficulty labels and routed to a configuration on that group's front without further search. The evaluation compares COMPAS against Default, RouteLLM, EcoTune, and PromiseTune under matched token budgets on LiveCodeBench v6-only, a random split, five seeds, SWE-bench Verified-mini, and Devstral/Qwen3.5 model families, reporting improved pass@1 and reduced cost in most settings.","tokens_in":15535,"tokens_out":6640,"duration_ms":69279,"significance":"If the transferability claim is established, COMPAS is a valuable contribution: it jointly optimizes model, prompt, and decoding choices in a difficulty-aware way and demonstrates a substantially better quality-cost trade-off than global tuners and routers under a controlled budget. The paper's strengths are its thorough evaluation: matched token budgets, paired Wilcoxon tests with bootstrap CIs and Vargha-Delaney A12, five independent seeds, ablations, sensitivity analysis around several design axes, and a reproducibility artifact. The main caveat is that the default budget is selected on the test split and the anti-overfitting mechanism is budget-fragile, so the central transfer claim is not yet fully supported. The contribution remains significant if these issues are addressed with a proper validation-based selection and a budget-independent robustness check.","major_comments":[{"comment":"The default offline budget B=1M tokens is chosen after observing the v6-only test split: the text states '1M tokens scores highest... we fix 1M tokens as COMPAS's default offline budget,' and this same v6-only split is then used for the headline 52.8% result in Table 3. This is test-set-based hyperparameter selection, so the reported 6.9-point gain over EcoTune is not a single evaluation on a truly unseen test set. The five-seed and random-split robustness runs reuse the same selected budget and therefore do not remove the selection. Please select B (and the other deployer-set choices in Table 7) on a validation split disjoint from the final test split, or report the headline as the best of several budgets with an explicit correction for selection.","section":"Sensitivity Analysis (Table 7); Main Effectiveness (Table 3)"},{"comment":"Equation (2) is introduced as the mechanism that prevents overfitting, with the text claiming that fresh mini-batches and a held-out pool stop candidates from passing by overfitting M_g. However, Table 7 shows that doubling the budget to 2M tokens lowers Pass@1 from 52.8% to 46.6%, and the paper attributes this to two groups overfitting their search samples (31.7% and 18.0% Pass@1). Thus the gate does not generically prevent overfitting; it only does so at the budget that was selected on the test split. Additionally, H_g is a finite pool of the same training tasks and is sampled from throughout the search, so the reference to Dwork et al. (2015) does not by itself provide a reusable-holdout guarantee for this procedure. Please characterize when the gate fails (e.g., pool size, number of rounds, acceptance rate, group difficulty) and add a budget-independent safeguard or an early-stopping/validation rule.","section":"Search Stage 2: Joint Tuning, Eq. (2); Sensitivity Analysis (Table 7, Budget)"},{"comment":"The text states that the per-group weights alpha_g=(0.2,0.5,0.8) are 'validated against other alternatives in Sensitivity Analysis,' but Table 7 contains no axis that varies alpha_g. Since alpha_g is used both in model selection (Eq. 1) and in online routing (Eq. 3), its effect on the reported quality-cost trade-off is currently untested. Please add a sensitivity run for alpha_g or remove the validation claim.","section":"Search Stage 1: Fast Model Selection, Eq. (1); Sensitivity Analysis (Table 7)"}],"minor_comments":[{"comment":"The sentence 'doubling the budget to 2M does not significantly improve on 1M' is misleading because the observed value at 2M (46.6%) is lower than at 1M (52.8%); it should say it does not significantly differ or is numerically worse.","section":"Sensitivity Analysis (Table 7, Budget)"},{"comment":"On Devstral, COMPAS (27.1%) is below EcoTune (29.3%), and the explanation is that the Stage 1 probe cannot distinguish the two Devstral models. This limitation is acknowledged, but the 'Generalization Across Model Family' framing in the section title and takeaway is stronger than the evidence; consider explicitly labeling this as a boundary condition of the model-selection probe.","section":"Generalization Across Model Family (Table 5, Devstral)"},{"comment":"The notation d_quality and d_cost is overloaded: in Eq. (1) these are normalized across the model pool for a fixed seed configuration, while in Eq. (3) they are normalized over the front F_g. Please use distinct symbols or a clarifying sentence.","section":"Equations (1) and (3)"},{"comment":"The conclusion says 'each experiment only tested COMPAS on two same-family LLMs,' but the experiments cover three model families (DeepSeek, Devstral, Qwen3.5), each with two models; the wording should be 'two models per family.'","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.SE and the empirical effort is substantial. The decisive issue is methodological: the default budget is selected on the same v6-only split used for the headline comparison, and the anti-overfitting gate fails when that budget is doubled. These concerns are fixable with a proper validation-based budget choice and a budget-independent robustness analysis, so I do not recommend rejection, but the transfer claim should not be accepted as stated. The missing alpha_g sensitivity is a smaller but clear gap that should also be closed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: COMPAS is the first thing I've seen that jointly searches model, prompt, and decoding settings per difficulty group and keeps separate quality-cost fronts, and the empirical work around it is unusually careful. But the central transfer claim has a real soft spot: the default budget was picked after looking at the v6-only test split, and the accept/reject gate doesn't prevent overfitting when the budget doubles. I'd send it to review, but I wouldn't take the 6.9-point gain at face value.\n\nWhat's genuinely new: the three observations (prompt/decoding interaction, model-specific tuning effects, per-group best config) are clearly motivated, and the ablation study supports each design choice. The method itself — cheap model probe, then reflection-based joint prompt/decoding search with per-group Pareto fronts and online routing — is a real combination not covered by RouteLLM, GEPA, or EcoTune. The evaluation is above the usual bar: matched token budgets, paired Wilcoxon, bootstrap CIs, five seeds, per-group breakdowns, a random split, and an ablation that actually shows each component earns its keep. The SWE-bench transfer is a useful extra even if the numbers aren't leaderboard-comparable. The paper is honest about where it fails (Devstral) and what it didn't test (cross-family pools). The artifact is promised but not auditable from the text, so that's still a promise.\n\nThe soft spots, in proportion. The stress-test note is right: the 1M budget was selected after observing the v6-only sensitivity results, and the same split is used for the headline comparison. That means the 52.8% vs 45.9% gap is partly the result of tuning on the test set. The 2M budget run dropping to 46.6% with two groups overfitting their search samples shows Eq. 2's fresh-batch gate is not a robust anti-overfitting mechanism — it works at 1M tokens but fails at 2M. This doesn't kill the paper, but it means the transfer claim is not yet established. A fair evaluation would pick the budget on a validation split and then evaluate once. The random-split and five-seed runs mitigate the concern because they also use the same budget, so they inherit the issue. On the plus side, the per-group results are striking (97.6% Easy/functional vs 30.6% for the best baseline), so the effect is concentrated somewhere real; it's the magnitude that's uncertain.\n\nWho this is for: anyone working on LLM inference optimization or code-gen pipelines will want to know about COMPAS even if they end up re-benchmarking it. Deserves a serious referee. If I were the editor I'd send it out with the clear request that the budget-selection issue be addressed, either by a validation-then-test protocol or by relaxing the transfer claim. Recommendation: send to peer review.","headline":"Genuinely novel joint search with a careful evaluation, but the headline gain is undercut by the budget being tuned on the test split; worth a serious referee, not yet trustworthy at face value.","tokens_in":16062,"tokens_out":2105,"would_cite":true,"duration_ms":20208,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COMPAS claims joint difficulty-aware search over model, prompt, and decoding settings beats routers and global tuners for code generation.","keywords":["code generation","LLM inference optimization","prompt optimization","decoding settings","difficulty-aware routing","Pareto quality-cost fronts","LiveCodeBench","SWE-bench"],"falsifier":"Re-run the same search on a fresh LiveCodeBench-style split with a search budget chosen by any rule other than the fixed 1M tokens (for example, 1.5M or a budget proportional to group size). If the held-out Pass@1 falls below the 1M-budget result, or if accepted candidates show materially higher pass rates on the search pool than on the held-out pool, then the gate's anti-overfitting guarantee does not transfer across budgets.","tokens_in":1957,"feed_emoji":"🤖","tokens_out":3572,"duration_ms":81823,"temperature":0.7,"pith_summary":"COMPAS is a two-phase method for LLM code generation that treats model choice, prompt, and decoding settings as one joint configuration and optimizes a separate quality-cost front for each task-difficulty group. The paper argues that existing optimizers leave most of the search space unexplored: routers pick only a model, global tuners pick one configuration for all tasks, and prompt optimizers fix model and decoding. Under a matched search budget on LiveCodeBench, COMPAS reaches 52.8% pass@1 versus 45.9% for the best baseline while cutting cost from $36.57 to $4.92, and on SWE-bench Verified-mini it resolves 76.0% of tasks versus 70.0%. The method matters because code-generation pipelines could in principle route every query to a configuration tuned for that query's difficulty, at no additional online search cost.","feed_headline":"Difficulty-aware search lifts code pass@1 from 45.9% to 52.8%","feed_subtitle":"COMPAS tunes model, prompt, and decoding per difficulty group, beating routers and global tuners at lower cost.","key_machinery":"The load-bearing object is the per-group quality-cost Pareto front F_g: the set of accepted configurations (model, prompt, decoding settings) for difficulty group g such that no other accepted configuration is at least as good on both quality and cost. The front is built by a two-stage offline search. Stage 1 probes each candidate model on a small random sample S_g and fixes the winner by a cost-aware rule α_g · quality + (1 - α_g) · inverse cost, with α_g = 0.2, 0.5, 0.8 for Easy, Medium, Hard. Stage 2 runs an LLM-reflection loop that proposes a new candidate each round, re-evaluates it on a random mini-batch M_g and a freshly resampled held-out batch H_g, and accepts it only if it passes the rule on both (Eq. 2). Online routing (Eq. 3) is then a pure lookup: match the task's difficulty group, pick the front point maximizing the same normalized quality-cost trade-off. The reuse of the same α_g weights in search and routing, and the fresh-resampling gate that guards against adaptive overfitting of the holdout, are what make the fronts transferable rather than mere curve-fits to the training tasks.","core_discovery":"The central claim is that the joint space of model, prompt, and decoding settings contains group-specific configurations that dominate both single global configurations and model-only routing, and that these configurations can be found offline and transferred to unseen tasks. The paper supports this with three empirical observations: prompts and decoding settings interact, the same prompt-decoding update affects different models differently, and the best configuration depends on task difficulty. COMPAS operationalizes them by splitting training tasks into difficulty groups, spending a small probe budget to pick a model per group, then running an LLM-reflection loop that jointly mutates prompt and decoding settings, keeping only candidates that pass an accept/reject gate on both a mini-batch and a freshly resampled held-out batch. The output per group is a Pareto front of configurations trading quality against cost; at test time, each task is mapped to its group's front and a configuration is chosen by the same quality-cost rule used in search.","pith_inferences":["This paper's results suggest a testable extension: instead of a fixed 8-problem model probe, spend the probe budget adaptively, stopping once the model ranking stabilizes; this would likely recover the Devstral gap.","Because the best configuration varies by difficulty, the same per-group front idea could be applied along axes other than difficulty, such as task type, programming language, or domain, provided the benchmark carries a usable label.","The 2M-budget overfitting result implies the anti-overfitting gate is budget-sensitive; on other workloads the search budget should be treated as a hyperparameter with its own validation procedure, not a fixed constant.","A direct economic consequence is that COMPAS's cheapest runs cost about one-tenth of the best baseline's, so the method changes the break-even point for when it is worth doing offline search before serving a code-generation API."],"forward_implications":["If a deployment already pays for offline search tokens, COMPAS's per-group fronts can be reused for any number of test tasks with zero online search cost.","Routing by task difficulty can beat both a single global configuration and a quality-only router, so benchmark protocols should report quality-cost fronts, not just pass@1.","The same pipeline transfers from function-level generation to repository-level agentic settings: COMPAS resolves 76.0% on SWE-bench Verified-mini versus 70.0% for the best baseline.","The gain is not tied to one model family: on Qwen3.5 COMPAS leads at 40.6% pass@1 and $1.01 cost, while on Devstral it improves over the default only when the two pool models are separable by the cheap probe."],"supporting_citations":[{"why":"Model-only router baseline; COMPAS must beat it under matched budget.","marker":"Ong et al. 2025"},{"why":"Global decoding tuner baseline that searches the same dimensions; it is the strongest pass@1 baseline at $36.57.","marker":"Xu, Chen, and Wen 2025"},{"why":"Provides the reflection-loop mechanism COMPAS adapts for joint prompt-decoding mutation.","marker":"Agrawal et al. 2025"},{"why":"Supplies LiveCodeBench's contamination-free v5/v6 releases and per-problem difficulty ratings used for grouping.","marker":"Jain et al. 2025"},{"why":"Supplies SWE-bench tasks and fix-time difficulty labels used for the repository-level transfer test.","marker":"Jimenez et al. 2024"},{"why":"Establishes why the reusable holdout must be freshly resampled; the paper's Eq. 2 gate applies this principle.","marker":"Dwork et al. 2015"},{"why":"Provides the tested ranges for temperature, top-p, and frequency penalty in code generation.","marker":"Arora et al. 2024"},{"why":"Cost-aware router that motivates the α_g quality-cost trade-off rule.","marker":"Chen, Zaharia, and Zou 2023"},{"why":"Scaling laws underpinning the assumption that within-family model pairs are orderable by a cheap probe.","marker":"Kaplan et al. 2020"}],"fun_headline_variants":["Difficulty-aware code gen search lifts pass@1 by 6.9 points","Joint model-prompt-decoding search cuts cost 7x while boosting pass@1","Difficulty-aware routing beats global configs at 1/7 cost","Per-difficulty configuration search improves code gen pass@1 to 52.8%"],"cache_read_input_tokens":18176,"weakest_assumption_plain":"The offline accept/reject gate (Eq. 2) with freshly resampled mini-batches and held-out batches is assumed to prevent overfitting so the learned per-group fronts transfer to unseen test tasks; the paper's own sensitivity analysis shows this assumption is fragile, since doubling the search budget drops Pass@1 from 52.8% to 46.6%.","fun_headline_variants_meta":{"raw":{"variants":["Difficulty-aware code gen search lifts pass@1 by 6.9 points","Joint model-prompt-decoding search cuts cost 7x while boosting pass@1","Difficulty-aware routing beats global configs at 1/7 cost","Per-difficulty configuration search improves code gen pass@1 to 52.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3479,"prompt_tokens":979,"completion_tokens":2500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2414}},"tokens_in":595,"tokens_out":2500,"duration_ms":19557,"temperature":1.0,"reasoning_tokens":2414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:29:37.217856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same search on a fresh LiveCodeBench-style split with a search budget chosen by any rule other than the fixed 1M tokens (for example, 1.5M or a budget proportional to group size). If the held-out Pass@1 falls below the 1M-budget result, or if accepted candidates show materially higher pass rates on the search pool than on the held-out pool, then the gate's anti-overfitting guarantee does not transfer across budgets.","supporting_citations":[{"cited_title":"LiveBench: A Challenging, Contamination-Limited","cited_arxiv_id":null,"evidence_quote":"Supplies LiveCodeBench's contamination-free v5/v6 releases and per-problem difficulty ratings used for grouping."},{"cited_title":"Science , volume=","cited_arxiv_id":null,"evidence_quote":"Establishes why the reusable holdout must be freshly resampled; the paper's Eq. 2 gate applies this principle."}],"review_version":1}