Pith. sign in

REVIEW 3 major objections 4 minor 80 references

COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read COMPAS claims joint difficulty-aware search over model, prompt, and decoding settings beats routers and global tuners for code generation.

desk verdict Genuinely novel joint search with a careful evaluation, but the headline gain is undercut by the budget being tuned on the test split; worth a serious referee, not yet trustworthy at face value. read the letter →

arxiv 2608.04336 v1 pith:D2653UJ5 submitted 2026-08-05 cs.SE cs.AI

Jingzhi Gong , Jie M. Zhang , Gunel Jahangirova , Dong Huang , Mohammad Reza Mousavi , Mark Harman This is my paper · ORCID
classification cs.SEcs.AI
keywords codegenerationLLMinferenceoptimizationpromptdecodingsettingsdifficulty-awareroutingParetoquality-costfrontsLiveBenchSWE-bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

COMPAS is a two-phase method for LLM code generation that treats model choice, prompt, and decoding settings as one joint configuration and optimizes a separate quality-cost front for each task-difficulty group. The paper argues that existing optimizers leave most of the search space unexplored: routers pick only a model, global tuners pick one configuration for all tasks, and prompt optimizers fix model and decoding. Under a matched search budget on LiveCodeBench, COMPAS reaches 52.8% pass@1 versus 45.9% for the best baseline while cutting cost from $36.57 to $4.92, and on SWE-bench Verified-mini it resolves 76.0% of tasks versus 70.0%. The method matters because code-generation pipelines could in principle route every query to a configuration tuned for that query's difficulty, at no additional online search cost.

What carries the argument

The load-bearing object is the per-group quality-cost Pareto front F_g: the set of accepted configurations (model, prompt, decoding settings) for difficulty group g such that no other accepted configuration is at least as good on both quality and cost. The front is built by a two-stage offline search. Stage 1 probes each candidate model on a small random sample S_g and fixes the winner by a cost-aware rule α_g · quality + (1 - α_g) · inverse cost, with α_g = 0.2, 0.5, 0.8 for Easy, Medium, Hard. Stage 2 runs an LLM-reflection loop that proposes a new candidate each round, re-evaluates it on a random mini-batch M_g and a freshly resampled held-out batch H_g, and accepts it only if it passes the rule on both (Eq. 2). Online routing (Eq. 3) is then a pure lookup: match the task's difficulty group, pick the front point maximizing the same normalized quality-cost trade-off. The reuse of the same α_g weights in search and routing, and the fresh-resampling gate that guards against adaptive overfitting of the holdout, are what make the fronts transferable rather than mere curve-fits to the training tasks.

What would settle it

Re-run the same search on a fresh LiveCodeBench-style split with a search budget chosen by any rule other than the fixed 1M tokens (for example, 1.5M or a budget proportional to group size). If the held-out Pass@1 falls below the 1M-budget result, or if accepted candidates show materially higher pass rates on the search pool than on the held-out pool, then the gate's anti-overfitting guarantee does not transfer across budgets.

Watch

Extended reading notes

Core claim

The central claim is that the joint space of model, prompt, and decoding settings contains group-specific configurations that dominate both single global configurations and model-only routing, and that these configurations can be found offline and transferred to unseen tasks. The paper supports this with three empirical observations: prompts and decoding settings interact, the same prompt-decoding update affects different models differently, and the best configuration depends on task difficulty. COMPAS operationalizes them by splitting training tasks into difficulty groups, spending a small probe budget to pick a model per group, then running an LLM-reflection loop that jointly mutates prompt and decoding settings, keeping only candidates that pass an accept/reject gate on both a mini-batch and a freshly resampled held-out batch. The output per group is a Pareto front of configurations trading quality against cost; at test time, each task is mapped to its group's front and a configuration is chosen by the same quality-cost rule used in search.

Load-bearing premise

The offline accept/reject gate (Eq. 2) with freshly resampled mini-batches and held-out batches is assumed to prevent overfitting so the learned per-group fronts transfer to unseen test tasks; the paper's own sensitivity analysis shows this assumption is fragile, since doubling the search budget drops Pass@1 from 52.8% to 46.6%.

Editorial extensions

If this is right

  • If a deployment already pays for offline search tokens, COMPAS's per-group fronts can be reused for any number of test tasks with zero online search cost.
  • Routing by task difficulty can beat both a single global configuration and a quality-only router, so benchmark protocols should report quality-cost fronts, not just pass@1.
  • The same pipeline transfers from function-level generation to repository-level agentic settings: COMPAS resolves 76.0% on SWE-bench Verified-mini versus 70.0% for the best baseline.
  • The gain is not tied to one model family: on Qwen3.5 COMPAS leads at 40.6% pass@1 and $1.01 cost, while on Devstral it improves over the default only when the two pool models are separable by the cheap probe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This paper's results suggest a testable extension: instead of a fixed 8-problem model probe, spend the probe budget adaptively, stopping once the model ranking stabilizes; this would likely recover the Devstral gap.
  • Because the best configuration varies by difficulty, the same per-group front idea could be applied along axes other than difficulty, such as task type, programming language, or domain, provided the benchmark carries a usable label.
  • The 2M-budget overfitting result implies the anti-overfitting gate is budget-sensitive; on other workloads the search budget should be treated as a hyperparameter with its own validation procedure, not a fixed constant.
  • A direct economic consequence is that COMPAS's cheapest runs cost about one-tenth of the best baseline's, so the method changes the break-even point for when it is worth doing offline search before serving a code-generation API.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes COMPAS, a two-phase offline/online method for code-generation optimization. It first reports three empirical observations about prompt/decoding interaction, model-specific tuning effects, and difficulty-dependent best configurations. The offline phase performs a cheap model-selection probe followed by LLM-reflection-based joint search over prompts and decoding settings, with an accept/reject gate (Eq. 2) that evaluates candidates on freshly resampled mini-batches and a held-out pool; each difficulty(-type) group receives its own quality-cost Pareto front. Online, test tasks are assigned to groups via benchmark difficulty labels and routed to a configuration on that group's front without further search. The evaluation compares COMPAS against Default, RouteLLM, EcoTune, and PromiseTune under matched token budgets on LiveCodeBench v6-only, a random split, five seeds, SWE-bench Verified-mini, and Devstral/Qwen3.5 model families, reporting improved pass@1 and reduced cost in most settings.

Significance. If the transferability claim is established, COMPAS is a valuable contribution: it jointly optimizes model, prompt, and decoding choices in a difficulty-aware way and demonstrates a substantially better quality-cost trade-off than global tuners and routers under a controlled budget. The paper's strengths are its thorough evaluation: matched token budgets, paired Wilcoxon tests with bootstrap CIs and Vargha-Delaney A12, five independent seeds, ablations, sensitivity analysis around several design axes, and a reproducibility artifact. The main caveat is that the default budget is selected on the test split and the anti-overfitting mechanism is budget-fragile, so the central transfer claim is not yet fully supported. The contribution remains significant if these issues are addressed with a proper validation-based selection and a budget-independent robustness check.

major comments (3)
  1. [Sensitivity Analysis (Table 7); Main Effectiveness (Table 3)] The default offline budget B=1M tokens is chosen after observing the v6-only test split: the text states '1M tokens scores highest... we fix 1M tokens as COMPAS's default offline budget,' and this same v6-only split is then used for the headline 52.8% result in Table 3. This is test-set-based hyperparameter selection, so the reported 6.9-point gain over EcoTune is not a single evaluation on a truly unseen test set. The five-seed and random-split robustness runs reuse the same selected budget and therefore do not remove the selection. Please select B (and the other deployer-set choices in Table 7) on a validation split disjoint from the final test split, or report the headline as the best of several budgets with an explicit correction for selection.
  2. [Search Stage 2: Joint Tuning, Eq. (2); Sensitivity Analysis (Table 7, Budget)] Equation (2) is introduced as the mechanism that prevents overfitting, with the text claiming that fresh mini-batches and a held-out pool stop candidates from passing by overfitting M_g. However, Table 7 shows that doubling the budget to 2M tokens lowers Pass@1 from 52.8% to 46.6%, and the paper attributes this to two groups overfitting their search samples (31.7% and 18.0% Pass@1). Thus the gate does not generically prevent overfitting; it only does so at the budget that was selected on the test split. Additionally, H_g is a finite pool of the same training tasks and is sampled from throughout the search, so the reference to Dwork et al. (2015) does not by itself provide a reusable-holdout guarantee for this procedure. Please characterize when the gate fails (e.g., pool size, number of rounds, acceptance rate, group difficulty) and add a budget-independent safeguard or an early-stopping/validation rule.
  3. [Search Stage 1: Fast Model Selection, Eq. (1); Sensitivity Analysis (Table 7)] The text states that the per-group weights alpha_g=(0.2,0.5,0.8) are 'validated against other alternatives in Sensitivity Analysis,' but Table 7 contains no axis that varies alpha_g. Since alpha_g is used both in model selection (Eq. 1) and in online routing (Eq. 3), its effect on the reported quality-cost trade-off is currently untested. Please add a sensitivity run for alpha_g or remove the validation claim.
minor comments (4)
  1. [Sensitivity Analysis (Table 7, Budget)] The sentence 'doubling the budget to 2M does not significantly improve on 1M' is misleading because the observed value at 2M (46.6%) is lower than at 1M (52.8%); it should say it does not significantly differ or is numerically worse.
  2. [Generalization Across Model Family (Table 5, Devstral)] On Devstral, COMPAS (27.1%) is below EcoTune (29.3%), and the explanation is that the Stage 1 probe cannot distinguish the two Devstral models. This limitation is acknowledged, but the 'Generalization Across Model Family' framing in the section title and takeaway is stronger than the evidence; consider explicitly labeling this as a boundary condition of the model-selection probe.
  3. [Equations (1) and (3)] The notation d_quality and d_cost is overloaded: in Eq. (1) these are normalized across the model pool for a fixed seed configuration, while in Eq. (3) they are normalized over the front F_g. Please use distinct symbols or a clarifying sentence.
  4. [Conclusion] The conclusion says 'each experiment only tested COMPAS on two same-family LLMs,' but the experiments cover three model families (DeepSeek, Devstral, Qwen3.5), each with two models; the wording should be 'two models per family.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline pass@1 and resolve-rate results are genuine holdout evaluations, and the paper's design choices are motivated by non-test measurements rather than defined into the test outcomes.

full rationale

COMPAS's claimed improvements do not reduce to its inputs by construction. The offline search builds per-group fronts on training splits (LCB release-v5, SWE-bench 44-task search pool) and evaluates on excluded test tasks (v6-only, an independent random split, and Verified-mini), so the headline values are measured holdout outcomes rather than fitted values renamed as predictions. The three motivating observations are empirical measurements on non-test data and do not define the test results. Equation (2)'s accept/reject gate and Equation (3)'s routing lookup are computational procedures, not identities that force the reported gains. The paper contains self-citations (e.g., Gong et al. 2025, Cheung et al. 2025), but these are not load-bearing: the relevant claims are also directly tested in this paper on release-v5 data. The legitimate concerns raised around the paper—selecting the 1M-token budget and the difficulty×type grouping on the v6-only split, and using DeepSeek-V4-Pro both as a candidate model and as the reflection LLM—are statistical-validity and confounding issues, not circularity: no result is equivalent to an input by definition or by a self-citation chain. The sensitivity analysis's 2M-budget overfitting is a robustness failure, not a circular derivation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method's central claim rests on benchmark difficulty labels, transferability of per-group fronts across releases and benchmarks, and the anti-overfitting behavior of the accept/reject gate; the gate is shown fragile when the budget doubles. Free parameters (alpha_g, token budgets, budget weights, context lengths, probe cap) are hand-chosen, and several were tuned with the v6-only test split. No invented entities are introduced.

free parameters (6)
  • alpha_g (per-group quality-cost weight) = 0.2, 0.5, 0.8
    Used in Eq. (1) for model selection and Eq. (3) for online routing; hand-chosen, text claims sensitivity validation but Table 7 does not vary alpha_g.
  • offline token budget B = 1M tokens (LiveCodeBench), 4M tokens (SWE-bench)
    Calibrated from preliminary traces and fixed after comparing 0.5M/1M/2M on the v6-only test split (Table 7); affects which fronts are learned.
  • per-group budget weights = 1:2:3 for Easy:Medium:Hard
    Hand-chosen allocation of search budget across difficulty groups; not varied in sensitivity analysis.
  • reflection context size = 10 failure + 10 success examples
    Hand-set; Short-Context (2+2) drops Pass@1 to 44.0%, so the value materially affects results.
  • Stage 1 probe cap = 8 problems per group
    Caps model-probe budget so Stage 2 budget is preserved; on Devstral two models are indistinguishable at this probe size, causing a wrong model choice.
  • search/holdout split ratio = 80/20 per group
    Hand-chosen split for each group's search pool and held-out pool; not varied in the paper.
assumptions (5)
  • domain assumption Official difficulty labels are available for every test task and are meaningful grouping signals.
    LCB Elo thresholds and SWE-bench fix-time thresholds are used to route test tasks in the main results; the XGBoost-Router sensitivity run (49.7% vs 52.8%) partially addresses label dependence.
  • ad hoc to paper The accept/reject gate in Eq. (2) prevents the offline search from overfitting to the search pool.
    This is the mechanism meant to keep per-group fronts transferable; the Double-Budget run (2M tokens, 46.6% vs 52.8%) shows the gate can fail and overfit, so the assumption is workload-dependent.
  • domain assumption Optimal configurations are stable from the training release to the test release and across benchmarks.
    Fronts built on release v5 (or 44 SWE-bench tasks) are applied to v6-only (or Verified-mini); the main results are the evidence for this transfer assumption.
  • domain assumption Within a model family, two models can be ranked reliably by a cheap probe at a fixed seed config.
    The Devstral result (COMPAS 27.1% vs EcoTune 29.3%) shows the probe can fail when the two models score too close together.
  • domain assumption DeepSeek-V4-Pro acts as an unbiased reflection optimizer while also being a candidate model.
    The reflection LLM proposes new candidates from the same pool being optimized; no test isolates this potential self-preference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation." pith.science (2026). https://pith.science/paper/D2653UJ5

@misc{pith2026260804336,
  author       = {Pith},
  title        = {Pith review of: COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2653UJ5}},
  note         = {Machine review of arXiv:2608.04336}
}
abstract

Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one fixed configuration for all tasks: global optimizers search one configuration for all tasks, routers choose only a model, and prompt optimizers keep the model and decoding settings fixed. This leaves their joint, group-specific interactions unclear. We therefore examine how these choices interact and observe that prompts and decoding settings interact, tuning effects vary by model, and the best configuration varies by task difficulty. Guided by these observations, we introduce COMPAS (Code-generation Optimization over Models, Prompts, And Decoding Settings), a difficulty-aware method that learns group-specific quality-cost fronts through low-cost model selection and joint prompt-decoding search, then routes each test task to its matching front online without further search. Under a matched search budget on LiveCodeBench, COMPAS improves pass@1 from 45.9% for the best baseline to 52.8% while reducing cost from $36.57 to $4.92. This also transfers to repository-level code generation on SWE-bench, resolving 76.0% of tasks versus 70.0% for the best baseline. Code and the reproducibility artifact are available at https://github.com/gjz78910/COMPAS.

Figures

Figures reproduced from arXiv: 2608.04336 by the authors.

Figure 1
Figure 1. COMPAS offline search and online routing workflow. Empirical Motivations for COMPAS To examine these interactions without using the primary test tasks, we evaluate DeepSeek-V3 and DeepSeek-V3.2 on the 880-task release-v5 set with varied prompts and decoding settings. We report mean Pass@1 and bootstrap 95% confi￾dence intervals; [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Three observations from the non-test empirical study that motivate the key designs of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 51 canonical work pages

  1. [1]

    Information and software Technology , volume=

    Search-based software engineering , author=. Information and software Technology , volume=. 2001 , publisher=

    work page 2001

  2. [2]

    Advances in computers , volume=

    The algorithm selection problem , author=. Advances in computers , volume=. 1976 , publisher=

    1976

  3. [3]

    Journal of artificial intelligence research , volume=

    SATzilla: portfolio-based algorithm selection for SAT , author=. Journal of artificial intelligence research , volume=

  4. [4]

    Journal of Artificial Intelligence Research , volume=

    Autofolio: An automatically configured algorithm selector , author=. Journal of Artificial Intelligence Research , volume=

    work page

  5. [5]

    and Kadous, M

    Ong, Isaac and Almahairi, Amjad and Wu, Vincent and Chiang, Wei-Lin and Wu, Tianhao and Gonzalez, Joseph E. and Kadous, M. Waleed and Stoica, Ion , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  6. [6]

    arXiv preprint arXiv:2305.05176 , year=

    Frugalgpt: How to use large language models while reducing cost and improving performance , author=. arXiv preprint arXiv:2305.05176 , year=

    arXiv

  7. [7]

    International Conference on Learning Representations , volume=

    Hybrid llm: Cost-efficient and quality-aware query routing , author=. International Conference on Learning Representations , volume=

    work page

  8. [8]

    AutoML Conference 2023 , year=

    Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference , author=. AutoML Conference 2023 , year=

    work page 2023

Show all 80 references
  1. [9]

    Router-R1: Teaching

    Haozhen Zhang and Tao Feng and Jiaxuan You , booktitle=. Router-R1: Teaching. 2025 , url=

    2025

  2. [10]

    RouterArena: An Open Platform for Comprehensive Comparison of

    Yifan Lu and Rixin Liu and Jiayi Yuan and Xingqi Cui and Shenrun Zhang and Hongyi Liu and Jiarong Xing , booktitle=. RouterArena: An Open Platform for Comprehensive Comparison of. 2026 , url=

    2026

  3. [11]

    Advances in Neural Information Processing Systems , volume=

    Automix: Automatically mixing language models , author=. Advances in Neural Information Processing Systems , volume=

  4. [12]

    Cost-Aware Contrastive Routing for

    Reza Shirkavand and Shangqian Gao and Peiran Yu and Heng Huang , booktitle=. Cost-Aware Contrastive Routing for. 2025 , url=

    2025

  5. [13]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Intention chain-of-thought prompting with dynamic routing for code generation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  6. [14]

    LLMR outer B ench: A Massive Benchmark and Unified Framework for LLM Routing

    Li, Hao and Zhang, Yiqun and Guo, Zhaoyan and Wang, Chenxu and Tang, Shengji and Zhang, Qiaosheng and Chen, Yang and Qi, Biqing and Ye, Peng and Bai, Lei and Wang, Zhen and Hu, Shuyue. LLMR outer B ench: A Massive Benchmark and Unified Framework for LLM Routing. Findings of th...

    2026 doi

  7. [15]

    AutoML Conference 2023 (Journal Track) , year=

    Marius Lindauer and Katharina Eggensperger and Matthias Feurer and Andr. AutoML Conference 2023 (Journal Track) , year=

    2023

  8. [16]

    Advances in Neural Information Processing Systems , volume =

    Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost Tobias and Blum, Manuel and Hutter, Frank , title =. Advances in Neural Information Processing Systems , volume =. 2015 , url =

    2015

  9. [17]

    and Desai, Nishkrit and Bae, Juhan and Lorraine, Jonathan and Ba, Jimmy , title =

    Zhang, Michael R. and Desai, Nishkrit and Bae, Juhan and Lorraine, Jonathan and Ba, Jimmy , title =. 2023 , eprint =

    2023

  10. [18]

    The Twelfth International Conference on Learning Representations , year=

    Large Language Models to Enhance Bayesian Optimization , author=. The Twelfth International Conference on Learning Representations , year=

  11. [19]

    2018 , editor =

    Falkner, Stefan and Klein, Aaron and Hutter, Frank , booktitle =. 2018 , editor =

    2018

  12. [20]

    International Conference on Learning Representations , year=

    Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization , author=. International Conference on Learning Representations , year=

  13. [21]

    48th International Conference on Software Engineering , year=

    PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning , author=. 48th International Conference on Software Engineering , year=

  14. [22]

    Forty-third International Conference on Machine Learning , year=

    AgentConductor: Topology Evolution for Multi-Agent Competition-Level Code Generation , author=. Forty-third International Conference on Machine Learning , year=

  15. [23]

    Li, Shen and Huang, Li and Zhan, Shaoxiong and Sun, Weifeng and Yin, Tao and Liu, Zhongxin and Yan, Meng , title =. Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Six...

    2026 doi

  16. [24]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    EcoTune: Token-Efficient Multi-Fidelity Hyperparameter Optimization for Large Language Model Inference , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  17. [25]

    Science , volume=

    Dwork, Cynthia and Feldman, Vitaly and Hardt, Moritz and Pitassi, Toniann and Reingold, Omer and Roth, Aaron , title=. Science , volume=. 2015 , doi=

    2015

  18. [26]

    2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC) , year=

    Donato, Benedetta and Mariani, Leonardo and Micucci, Daniela and Riganelli, Oliviero , title=. 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC) , year=

    2025

  19. [27]

    Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results? , year=

    Arora, Chetan and Sayeed, Ahnaf Ibn and Licorish, Sherlock and Wang, Fanyu and Treude, Christoph , booktitle=. Optimizing LLMs for Code Generation: Which Hyperparameter Settings Yield the Best Results? , year=

  20. [28]

    arXiv preprint arXiv:2506.13131 , year=

    Alphaevolve: A coding agent for scientific and algorithmic discovery , author=. arXiv preprint arXiv:2506.13131 , year=

    arXiv

  21. [29]

    The Fourteenth International Conference on Learning Representations , year=

    ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution , author=. The Fourteenth International Conference on Learning Representations , year=

  22. [30]

    arXiv preprint arXiv:2512.09108 , year=

    Evolving Excellence: Automated Optimization of LLM-based Agents , author=. arXiv preprint arXiv:2512.09108 , year=

  23. [31]

    Issue Localization via LLM-Driven Iterative Code Graph Searching , year=

    Jiang, Zhonghao and Ren, Xiaoxue and Yan, Meng and Jiang, Wei and Li, Yong and Liu, Zhongxin , booktitle=. Issue Localization via LLM-Driven Iterative Code Graph Searching , year=

  24. [32]

    GA4GC: Greener Agent for Greener Code via Multi-objective Configuration Optimization

    Jingzhi Gong and Yixin Bian and \ de la Cal\ , Luis and Giovanni Pinna and Anisha Uteem and David Williams and Mar Zamorano and Karine Even-Mendoza and Langdon, \ William B.\ and Menendez, \ Hector D.\ and Federica Sarro. GA4GC: Greener Agent for Greener Code via Multi-objecti...

    2026 doi

  25. [33]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Decoding decoded: Understanding hyperparameter effects in open-ended text generation , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  26. [34]

    Forty-second International Conference on Machine Learning , year=

    Optimizing Temperature for Language Models with Multi-Sample Inference , author=. Forty-second International Conference on Machine Learning , year=

  27. [35]

    Exploring the Impact of Temperature on Large Language Models: Hot or Cold? , journal =

    Lujun Li and Lama Sleem and Niccolo’ Gentile and Geoffrey Nichil and Radu State , keywords =. Exploring the Impact of Temperature on Large Language Models: Hot or Cold? , journal =. 2025 , note =. doi:https://doi.org/10.1016/j.procs.2025.07.135 , url =

    2025 doi

  28. [36]

    International Conference on Learning Representations (ICLR) , year =

    The Curious Case of Neural Text Degeneration , author =. International Conference on Learning Representations (ICLR) , year =

  29. [37]

    Princis, Henrijs and Sharma, Arindam and David, Cristina , title =. Proc. ACM Program. Lang. , month = jun, articleno =. 2026 , issue_date =. doi:10.1145/3808347 , abstract =

    2026 doi

  30. [38]

    LiveBench: A Challenging, Contamination-Limited

    Colin White and Samuel Dooley and Manley Roberts and Arka Pal and Benjamin Feuer and Siddhartha Jain and others , booktitle=. LiveBench: A Challenging, Contamination-Limited. 2025 , url=

    2025

  31. [39]

    arXiv preprint arXiv:2412.02906 , year=

    Does few-shot learning help LLM performance in code synthesis? , author=. arXiv preprint arXiv:2412.02906 , year=

    arXiv

  32. [40]

    IEEE Transactions on Software Engineering , year=

    Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models , author=. IEEE Transactions on Software Engineering , year=

  33. [41]

    ACM Transactions on Software Engineering and Methodology , year=

    Structured Chain-of-Thought Prompting for Code Generation , author=. ACM Transactions on Software Engineering and Methodology , year=

  34. [42]

    ACM Transactions on Software Engineering and Methodology , year=

    Self-Planning Code Generation with Large Language Models , author=. ACM Transactions on Software Engineering and Methodology , year=

  35. [43]

    Demystifying

    Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming , journal=. Demystifying. 2025 , doi=

    2025

  36. [44]

    Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks , year=

    Sharifloo, Amir Molzam and Heydari, Maedeh and Kazerooni, Parsa and Maninger, Daniel and Mezini, Mira , booktitle=. Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks , year=

  37. [45]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  38. [46]

    arXiv preprint arXiv:2605.23491 , year=

    CoSPlay: Cooperative Self-Play at Test-Time with Self-Generated Code and Unit Test , author=. arXiv preprint arXiv:2605.23491 , year=

    arXiv

  39. [47]

    2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=

    Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation , author=. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=. 2025 , organization=

    2025

  40. [48]

    Trinity: An Evolved

    Jinglue Xu and Qi Sun and Peter Schwendeman and Stefan Nielsen and Edoardo Cetin and Yujin Tang , booktitle=. Trinity: An Evolved. 2026 , url=

    2026

  41. [49]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  42. [50]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  43. [51]

    The Thirteenth International Conference on Learning Representations , year=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. The Thirteenth International Conference on Learning Representations , year=

  44. [52]

    arXiv preprint arXiv:2606.19348 , year=

    Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

  45. [53]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  46. [54]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    arXiv

  47. [55]

    2025 , month =

    Introducing: Devstral 2 and Mistral Vibe CLI , author =. 2025 , month =

    2025

  48. [56]

    2026 , month =

    Qwen3.5: Towards Native Multimodal Agents , author =. 2026 , month =

    2026

  49. [57]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  50. [58]

    2024 , url=

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle=. 2024 , url=

    2024

  51. [59]

    2026 , url=

    Linzheng Chai and Jian Yang and Jiajun Wu and Ensheng Shi and Xianglong Liu , booktitle=. 2026 , url=

    2026

  52. [60]

    Holistic Agent Leaderboard: The Missing Infrastructure for

    Sayash Kapoor and Benedikt Stroebl and Peter Kirgis and Nitya Nadgir and Zachary S Siegel and Boyi Wei and Tianci Xue and Ziru Chen and Felix Chen and others , booktitle=. Holistic Agent Leaderboard: The Missing Infrastructure for. 2026 , url=

    2026

  53. [61]

    2025 , booktitle=

    SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution , author=. 2025 , booktitle=

    2025

  54. [62]

    Proceedings of the 2026 International Workshop on Agentic Engineering , pages =

    Tripathy, Arihant and Harshit, Ch Pavan and Vaidhyanathan, Karthik , title =. Proceedings of the 2026 International Workshop on Agentic Engineering , pages =. 2026 , isbn =. doi:10.1145/3786167.3788406 , abstract =

    2026

  55. [63]

    arXiv preprint arXiv:2602.07900 , year=

    Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents , author=. arXiv preprint arXiv:2602.07900 , year=

    arXiv

  56. [64]

    The Fourteenth International Conference on Learning Representations , year=

    Darwin G\"odel Machine: Open-Ended Evolution of Self-Improving Agents , author=. The Fourteenth International Conference on Learning Representations , year=

  57. [65]

    2024 , url=

    John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press , booktitle=. 2024 , url=

    2024

  58. [66]

    Xu and Xiangru Tang and Mingchen Zhuge and others , booktitle=

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and others , booktitle=. OpenHands: An Open Platform for. 2025 , url=

    2025

  59. [67]

    arXiv preprint arXiv:2604.25850 , year=

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses , author=. arXiv preprint arXiv:2604.25850 , year=

    arXiv

  60. [68]

    2025 , url=

    Lakshya A Agrawal and Shangyin Tan and Dilara Soylu and Noah Ziems and Rishi Khare and Krista Opsahl-Ong and Arnav Singhvi and Herumb Shandilya and Michael J Ryan and Meng Jiang and Christopher Potts and Koushik Sen and Alex Dimakis and Ion Stoica and Dan Klein and Matei Zahar...

    2025

  61. [69]

    Biometrics Bulletin , volume=

    Individual Comparisons by Ranking Methods , author=. Biometrics Bulletin , volume=. 1945 , doi=

    1945

  62. [70]

    The Annals of Statistics , volume=

    Bootstrap Methods: Another Look at the Jackknife , author=. The Annals of Statistics , volume=. 1979 , doi=

    1979

  63. [71]

    , journal=

    Vargha, Andras and Delaney, Harold D. , journal=. A Critique and Improvement of the. 2000 , doi=

    2000

  64. [72]

    Psychological Bulletin , volume=

    Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions , author=. Psychological Bulletin , volume=. 1993 , doi=

    1993

  65. [73]

    IEEE Transactions on Software Engineering , volume=

    The Impact of Automated Parameter Optimization on Defect Prediction Models , author=. IEEE Transactions on Software Engineering , volume=. 2019 , doi=

    2019

  66. [74]

    Tantithamthavorn, Chakkrit , year=

  67. [75]

    The Hitchhiker ' s Guide to Testing Statistical Significance in Natural Language Processing

    Dror, Rotem and Baumer, Gili and Shlomov, Segev and Reichart, Roi. The Hitchhiker ' s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018...

    2018 doi

  68. [76]

    Revisit Self-Debugging with Self-Generated Tests for Code Generation

    Chen, Xiancai and Tao, Zhengwei and Zhang, Kechi and Zhou, Changzhi and others. Revisit Self-Debugging with Self-Generated Tests for Code Generation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.1...

    2025 doi

  69. [77]

    Zhang, Jie and Bu, Qingwen and Xie, Xiaofei and Chen, Junjie and Cui, Heming , title =

    Huang, Dong and M. Zhang, Jie and Bu, Qingwen and Xie, Xiaofei and Chen, Junjie and Cui, Heming , title =. ACM Trans. Softw. Eng. Methodol. , month = dec, articleno =. 2025 , issue_date =. doi:10.1145/3724117 , abstract =

    2025 doi

  70. [78]

    2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages =

    Gong, Jingzhi and Giavrimis, Rafail and Brookes, Paul and Voskanyan, Vardan and Wu, Fan and Ashiga, Mari and Truscott, Matthew and Basios, Mike and Kanthan, Leslie and Xu, Jie and Wang, Zheng , title =. 2025 40th IEEE/ACM International Conference on Automated Software Engineer...

    2025

  71. [79]

    Proceedings of the 1st International Workshop on Responsible Software Engineering , pages =

    Cheung, Kuen Sum and Kaul, Mayuri and Jahangirova, Gunel and Mousavi, Mohammad Reza and Zie, Eric , title =. Proceedings of the 1st International Workshop on Responsible Software Engineering , pages =. 2025 , isbn =. doi:10.1145/3711919.3728678 , abstract =

    2025

  72. [80]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling Laws for Neural Language Models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.