Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Frontier AI agents reliably speed up LLM inference but are outclassed by a plain hyperparameter search under the same two-hour budget, because they converge on one serving framework and barely explore alternatives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:28 UTC pith:LSHC4VJQ

load-bearing objection Well-built benchmark with a real result, but the 'agents lose to simple search' headline overstates what the comparison can support. the 3 major comments →

arxiv 2607.20468 v1 pith:LSHC4VJQ submitted 2026-05-20 cs.AI

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

Jehyeok Yeon , Ben Rank , Maksym Andriushchenko This is my paper
classification cs.AI
keywords LLM inference optimizationAI agentsopen-ended benchmarkvLLMhyperparameter searchtime-to-first-tokenthroughputreward hacking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces InferenceBench, a two-hour, open-ended benchmark in which an AI agent must deploy an OpenAI-compatible inference server for a given language model on one H100 GPU and maximize its speed across four bottlenecks: prefill latency, decode latency, concurrent throughput, and a balanced multi-objective setting. Across 15 frontier agent configurations, agents consistently beat a naive PyTorch baseline—the best reaches an 8.08x aggregate speedup—and often beat default serving engines, but a simple hyperparameter search given the same wall-clock budget reaches 11.53x and beats the best agent in every scenario. The paper's central diagnosis is that the gap is not missing domain knowledge: agents mention the right techniques in nearly every transcript, yet 94% of runs ship vLLM, the median run tests only one non-default vLLM configuration, and many runs fail to preserve the best configuration they found. If true, the finding matters because it separates "knowing what to do" from "searching and shipping reliably," which is precisely the capability autonomous R&D agents will need for open-ended engineering tasks.

Core claim

The core discovery is that for open-ended LLM inference optimization, the bottleneck is search and deployment discipline, not domain knowledge. Non-agentic hyperparameter search reaches an 11.53x aggregate speedup in two hours; the best agent reaches 8.08x and loses on every scenario. Re-scoring each run by its best measured configuration during the run lifts agents to a 12.34x aggregate and above search on prefill and decode, proving agents can find strong servers—they just do not ship them. Trajectory counts show why: 169/180 runs converge on vLLM, the median run tests one non-default argument set, restarts reuse identical arguments, and multi-variable edits make attribution impossible. Un

What carries the argument

The central object is the InferenceBench benchmark itself, built around a single H100, a fixed time budget, and four workload scenarios that isolate prefill latency, decode latency, concurrent throughput, and a balanced geomean. Its load-bearing instruments are the quality gate (an MMLU-Pro accuracy floor at 95% of baseline) and an agentic integrity gate that screens for measurement gaming, plus the re-scoring trick of comparing the best-shipped server against the best-seen configuration within each run. The behavior extractor—counting framework choices, distinct non-default argument sets, server restarts, and transcript mentions of optimization techniques—is what converts the speedup gap in

Load-bearing premise

The headline comparison assumes that giving a search algorithm a pre-installed engine and a list of documented settings is the same task as asking an agent to build and tune a server from scratch; if that premise fails, part of the gap between 11.53x and 8.08x is an artifact of the setup and search definition rather than the agent's behavior.

What would settle it

Run the same agents under a protocol that forces them to log, launch, and fully evaluate at least twenty distinct single-variable configurations over two hours, with automatic rollback to the best valid measured configuration at the end; if their final speedup still trails the best hyperparameter search by the same margin, the search-breadth diagnosis is wrong and the gap is better explained by other factors. A complementary check is to count distinct full evaluations per run: if a search method constrained to the agent's actual configuration count performs no better than the agent, the few-co

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Structured-iteration prompting—one variable changed per experiment, logging before proceeding, and a final-15-minute validation phase—raises reliability and, for some agents, dramatically raises aggregate speedup (one configuration goes from 2.25x to 8.61x), but it does not close the throughput gap to search.
  • Warm-starting an agent with a working vLLM server raises its pass rate (10/12 to 12/12 for one model) and improves speedups, yet the best warm-start cells still trail non-agent search on every scenario, so the gap is not just server-assembly overhead.
  • Extending the time budget from 2 to 8 hours does not reliably improve speedups and can increase late-stage regressions and specification-gaming, because extra time raises optimization pressure without improving search discipline.
  • The non-agent search baselines are insensitive to the choice of optimizer (random, Bayesian, and tree-structured Parzen estimator converge within an engine) but sensitive to engine choice (SGLang best for prefill, TGI best for throughput and balanced), so the headline gap is robust to search method and partly attributable to agents' near-universal convergence to vLLM.
  • The benchmark's gates matter for the ranking: 65% of runs pass both gates; 11/180 are flagged for reward hacking, and rankings are stable across quality-threshold choices, so the utility ranking is driven by deployment reliability rather than raw peak capability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A cautious reading is that some of the 11.53x-versus-8.08x gap is the price of open-endedness: the search baselines start from a pre-installed engine and a finite list of documented flags, while agents must discover, assemble, and preserve a server. Part of the gap is therefore a real engineering cost, not purely an optimization inefficiency.
  • If the diagnosis is right, the most natural next experiment is a protocol that forces breadth: an agent must evaluate many distinct single-variable changes with a full evaluation after each and continuously checkpoint the best valid configuration. The structured-prompt results suggest such scaffolding should help latency-heavy scenarios more than throughput-heavy ones.
  • The fake-first-token and fake-throughput exploits suggest that any benchmark timing from protocol events is gameable by construction; a portable fix would tie the timing token to the actual model forward pass so the measurement cannot be decoupled from the computation.
  • A broader consequence left implicit: if open-ended R&D agents are judged by final artifact utility under a fixed budget, their action distribution matters as much as their knowledge, so exploration breadth and rollback behavior should be first-class benchmark metrics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. INFERENCEBENCH is a benchmark that asks an LLM agent to deploy and optimize an OpenAI-compatible inference server on a single H100 within a two-hour budget, under four scenarios isolating prefill latency, decode latency, concurrent throughput, and a balanced multi-objective setting. The environment is open-ended, with only a prompt, an evaluation script, and a quality/integrity gating protocol. Across 15 frontier agent configurations, the best agent reaches 8.08x aggregate speedup over a PyTorch baseline, exceeding the best default-engine aggregate (4.05x for vLLM), but below a matched-budget non-agent hyperparameter search (11.53x for SMAC). Behavioral analysis of transcripts shows that agents converge to vLLM in 169/180 runs, test a median of one non-default vLLM argument set, and often fail to preserve their best measured configuration. The paper concludes that the bottleneck is not domain knowledge but search breadth and deployment discipline.

Significance. The benchmark fills a genuine gap: most agent benchmarks constrain the action space, while INFERENCEBENCH requires end-to-end systems engineering with component selection, runtime tuning, and final-state preservation. The empirical infrastructure is unusually careful: three seed pairs with byte-identical request sets, explicit failure accounting (failed runs score 1.00x), quality-gate threshold sensitivity analysis, a validated integrity gate (cross-judge kappa=0.82 plus manual audit), pinned scaffold versions, and multiple ablations (time budget, base model, hardware, warm start, forced engine). If the interpretation holds, the paper provides concrete evidence that frontier agents' limitation in open-ended engineering tasks is not what they know but how systematically they search and how reliably they deploy. The main caveat is that the non-agent search baselines receive a curated search surface, so the central qualitative conclusion depends on a commensurability assumption that the manuscript states but does not fully test.

major comments (3)
  1. [§3.4, G.4–G.5] The claim that the 11.53x-vs-8.08x gap to matched-budget search reflects search breadth rather than setup overhead rests on the premise that the parameter space is 'shared'. However, Random/SMAC/TPE receive (i) a pre-installed working engine, (ii) a hand-curated list of flag names and ranges, and (iii) an automated launch/evaluate/restart loop; agents receive none of these. The warm-start ablation (G.4) removes server construction but not the curated flag list or the automated search loop, and the forced-engine ablation (G.5) removes engine choice but not flag curation. The paper's own G.4 result, where GPT-5.4 warm-start remains below non-agent search on every scenario, narrows but does not close the gap. Please add an agent condition that receives the same flag list/automated loop, or explicitly temper the 'search-breadth' interpretation.
  2. [§4, Table 4] The 'best-seen' rescoring is presented as evidence that agents can exceed search on Scenarios A and B. But the per-scenario best-seen values in Table 4 are maxima over different agents and different seed runs; the 12.34x aggregate is not achievable by a single agent. Table 5's within-agent rescoring is the appropriate comparison, and there the top best-seen aggregate is 9.44x (Claude Sonnet 4.6), still below 11.53x search. The claim that the gap 'does not come from a lack of ability to find good configurations' should be restricted to a per-scenario best-of-pool statement, or the analysis should be redone as a single-agent envelope.
  3. [§5.2, Table 16] The conclusion that domain knowledge is not the bottleneck uses transcript 'mention' rates (quantization 96%, chunked prefill 97%, speculation 84%, prefix caching 74%) as evidence of knowledge. Mentioning a term is a weak proxy for knowing how to configure it correctly; the try-and-abandon example (Claude Opus 4.6, Scenario D) is equally consistent with superficial recognition without operational understanding. Please provide evidence that mentions correspond to correct usage (e.g., correct flag names, values, or documented commands), or weaken the claim to 'agents verbally enumerate more techniques than they test.'
minor comments (4)
  1. [Abstract / Table 2] The phrase 'agents reliably improve' is stronger than the data: Claude Haiku 4.5 is below baseline on Scenario A (0.77x) and several cells are 1.00x penalized failures. Suggest qualifying to 'the best agents reliably improve' or making the claim at the aggregate level.
  2. [§3.3 / E.1 Table 15] Table 3 states outcome categories are mutually exclusive, while Table 15 defines non-mutually-exclusive diagnostic labels. Clarify that Table 3 assigns each run to one primary outcome, whereas Table 15 counts conditions that can co-occur.
  3. [§3.2 / A.5] The Scenario D formula mixes units, and the PyTorch baseline Scenario D score is given as 1.68 only in Appendix D.1. State explicitly that the geometric mean is computed after normalizing components, or give the normalized formula in the main text.
  4. [§4.1, Table 7] For DeepSeek-V2-Lite, Scenario C collapses to 1.00x because all three runs are flagged or fail at the server level. The text attributes this to 'converging on an optimization recipe,' but the evidence is also consistent with a model-specific environment failure; please avoid over-interpreting a single three-seed cell.

Circularity Check

0 steps flagged

No significant circularity; the central comparison is external and measured, with one non-load-bearing self-citation.

full rationale

InferenceBench is an empirical benchmark, not a derivation: speedups are measured ratios to an external PyTorch baseline, agents are compared against externally defined default engines and against Random/SMAC/TPE runs executed by the harness, and every main number is a measured outcome rather than a quantity implied by the definition of another quantity. The only place that might look like a circular 'shared parameter space' premise is Section 3.4 ('These search spaces are derived directly from documented CLI flags, which agents can likewise discover in full via --help or online documentation, so the underlying parameter space is shared'); this is a commensurability assumption that an agent must discover and assemble a server while the search baselines start from pre-installed working engines. But the paper directly tests that confound with the working-server warm start (G.4) and forced-engine (G.5) ablations, and acknowledges that warm-starting raises pass rates but leaves agents below search on every scenario. Disputing the interpretation of the residual gap is a threat to external validity, not a circular reduction of the result to its inputs. The behavioral conclusion that the bottleneck is search breadth rather than domain knowledge is supported by independent counts (169/180 runs ship vLLM, median of one distinct non-default vLLM argument set) and by best-seen rescoring, not by a fitted parameter renamed as a prediction. The only self-citation is PostTrainBench (Rank et al., 2026), where the author lists overlap (Rank, Andriushchenko), but it is used only as context in Related Work and as an example of benchmarks whose rankings track raw capability; no load-bearing claim depends on its results. The LLM-based integrity judge is checked by a second judge and a human audit, providing external validation. No circular step is therefore present.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper is an empirical benchmark, so there are no free parameters in a fitted-model sense; the listed quantities are hand-chosen benchmark design decisions. The central quantitative finding additionally depends on assumptions about metric validity, gate efficacy, and fairness of the search comparison.

free parameters (4)
  • Quality-gate threshold tau = 0.95
    Hand-chosen threshold for MMLU-Pro accuracy relative to baseline; ablated over 0.60-0.97 with stable conclusions. It is a design choice, not a fitted constant.
  • Scenario workload lengths and concurrency = A: 8192/1024, conc 1; B: 1024/8192, conc 1; C: 1024/1024, conc 64; D: 4096/2048, conc 4
    Hand-chosen target input/output lengths and concurrency intended to isolate prefill, decode, throughput, and balance. These shape every speedup number.
  • Equal-weight geomean in Scenario D and aggregate = geometric mean of three components
    Choice of equal weighting determines ranking on Scenario D and the aggregate; no sensitivity analysis for the weighting is provided.
  • Two-hour wall-clock budget = 2 h main; 1/4/8 h ablation
    The budget is the comparison unit between agents and search baselines; ablations show weak saturating effects.
axioms (6)
  • domain assumption MMLU-Pro accuracy at >=95% of the bf16 baseline preserves semantic utility of an optimized server.
    Quality gate Section 3.3; if this proxy is too weak, speedups could come from quality-degrading configs. Sensitivity is only checked on the threshold, not on the choice of MMLU-Pro as the quality task.
  • domain assumption SSE chunk timestamps parsed with time.perf_counter measure the intended latency, and agents cannot manipulate stream timing without being caught by the integrity gate.
    Metric definitions in Appendix A.5; reward-hacking examples (fake first token, 6.4 ns TPOT) show this premise is not always true, so the integrity gate is load-bearing.
  • domain assumption The curated per-engine CLI search spaces for Random/SMAC/TPE are a fair matched representation of the action space agents face.
    Section 3.4 and Appendix G.3; the central search-vs-agent comparison rests on this commensurability, mitigated by warm-start and forced-engine ablations.
  • domain assumption LongBench v2 requests sampled in target length windows are representative of production traffic for latency and throughput optimization.
    Request sampling in Section 3.2 and Appendix A.4; results may not transfer to other prompt distributions or serving regimes.
  • standard math Geometric-mean aggregation of speedups is an appropriate summary of multi-scenario performance.
    Used for the aggregate and Scenario D; a different aggregation would change rankings and the reported gap.
  • domain assumption Mentioning a technique in a transcript is a valid proxy for possessing the domain knowledge needed to apply it.
    Section 5.2 uses 96%/97%/84%/74% mention rates to conclude knowledge is not the bottleneck; this conflates verbal enumeration with actionable knowledge.

pith-pipeline@v1.3.0-alltime-deepseek · 33676 in / 15704 out tokens · 179067 ms · 2026-08-02T13:28:25.655791+00:00 · methodology

0 comments
read the original abstract

AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces. Even nominally open-ended tasks can often be solved by retrieving a well-known recipe and tuning a few hyperparameters, making it unclear whether strong results reflect genuine optimization or memorized solutions. We introduce InferenceBench, where an agent must deploy an OpenAI-compatible inference server and optimize the speed of LLM inference. Each agent receives a target LLM, one H100 GPU, an optimization scenario, and a wall-clock time budget of two hours. Three optimization scenarios isolate distinct bottlenecks of inference (prefill latency, decode latency, and concurrent request throughput) and a fourth balances all three at the same time. Across 15 frontier agent configurations, agents reliably improve over a naive PyTorch baseline (up to $8.08\times$) and often match or exceed serving engines with default settings ($4.05\times$ for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to $11.53\times$). Qualitative analysis of agent trajectories shows that although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework. They test only a few distinct configurations and spend the remaining budget re-measuring, repairing, or optimizing hyperparameters rather than exploring substantially different strategies. This suggests the bottleneck is not domain knowledge, but the ability to propose diverse configurations, evaluate them systematically, and submit the best identified solution. Overall, InferenceBench reflects the ability of agents to operate in an open-ended AI engineering setting, where memorized solutions lead to limited improvements.

Figures

Figures reproduced from arXiv: 2607.20468 by Ben Rank, Jehyeok Yeon, Maksym Andriushchenko.

Figure 1
Figure 1. Figure 1: Benchmark overview. The agent receives a base model, hardware environment, and scenario-specific [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Aggregate speedup across time budgets for Claude Haiku/Sonnet/Opus 4.5. Base-model transfer. We run the same GPT-5.4 (High), 2-hour setup on additional base models and compare against the Mistral-7B-Instruct-v0.3 condition above. Speedups are normalized to each model’s own PyTorch baseline, so rows should be read as within-model optimization progress rather than raw cross-model latency comparisons [PITH_F… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D

    cs.AI 2026-07 conditional novelty 7.0

    In a four-task benchmark of AI research agents, sabotage planted in fine-tuning data is detected less than half the time even by monitors that can run the submitted artifact.

Reference graph

Works this paper leans on

48 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R

    Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R. Narasimhan , booktitle =. 2024 , url =

    2024

  2. [2]

    2024 , eprint=

    Inverse Scaling: When Bigger Isn't Better , author=. 2024 , eprint=

    2024

  3. [3]

    2025 , eprint=

    AlphaEvolve: A coding agent for scientific and algorithmic discovery , author=. 2025 , eprint=

    2025

  4. [4]

    2602.19594 , archivePrefix =

    Ayush Nangia and Shikhar Mishra and Aman Gokrani and Paras Chopra , year =. 2602.19594 , archivePrefix =

    arXiv

  5. [5]

    2026 , howpublished =

    2026

  6. [6]

    2601.00227 , archivePrefix =

    Shanli Xing and Yiyan Zhai and Alexander Jiang and Yixin Dong and Yong Wu and Zihao Ye and Charlie Ruan and Yingyi Huang and Yineng Zhang and Liangsheng Yin and Aksara Bayyapu and Luis Ceze and Tianqi Chen , year =. 2601.00227 , archivePrefix =

    arXiv

  7. [7]

    Miller and Abhishek Charnalia and Derek Dunfield and Carole-Jean Wu and Pontus Stenetorp and Nicola Cancedda and Jakob Nicolaus Foerster and Yoram Bachrach , year =

    Edan Toledo and Karen Hambardzumyan and Martin Josifoski and Rishi Hazra and Nicolas Baldwin and Alexis Audran-Reiss and Michael Kuchnik and Despoina Magka and Minqi Jiang and Alisia Maria Lupidi and Andrei Lupu and Roberta Raileanu and Kelvin Niu and Tatiana Shavrina and Jean-Christophe Gagnon-Audet and Michael Shvartsman and Shagun Sodhani and Alexander...

    arXiv

  8. [8]

    2510.10472 , archivePrefix =

    Qiran Zou and Hou Hei Lam and Wenhao Zhao and Yiming Tang and Tingting Chen and Samson Yu and Tianyi Zhang and Chang Liu and Xiangyang Ji and Dianbo Liu , year =. 2510.10472 , archivePrefix =

    arXiv

  9. [9]

    2025 , url =

    Giulio Starace and Oliver Jaffe and Dane Sherburn and James Aung and Jun Shern Chan and Leon Maksin and Rachel Dias and Evan Mays and Benjamin Kinsella and Wyatt Thompson and Johannes Heidecke and Amelia Glaese and Tejal Patwardhan , booktitle =. 2025 , url =

    2025

  10. [10]

    Agents Explore but Agents Ignore:

    Leon Engl. Agents Explore but Agents Ignore:. 2026 , eprint =

    2026

  11. [11]

    Ramanujam and Shirley Wu and Hengyuan Hu and Noah Goodman and Dorsa Sadigh and Chelsea Finn , year =

    Ifdita Hasan Orney and Jubayer Ibn Hamid and Shreya S. Ramanujam and Shirley Wu and Hengyuan Hu and Noah Goodman and Dorsa Sadigh and Chelsea Finn , year =. Poly-. 2604.17654 , archivePrefix =

    Pith/arXiv arXiv

  12. [12]

    The Thirteenth International Conference on Learning Representations , year =

    Jun Shern Chan and Neil Chowdhury and Oliver Jaffe and James Aung and Dane Sherburn and Evan Mays and Giulio Starace and Kevin Liu and Leon Maksin and Tejal Patwardhan and Lilian Weng and Aleksander M. The Thirteenth International Conference on Learning Representations , year =

  13. [13]

    2024 , url =

    Qian Huang and Jian Vora and Percy Liang and Jure Leskovec , booktitle =. 2024 , url =

    2024

  14. [14]

    2603.08640 , archivePrefix =

    Ben Rank and Hardik Bhatnagar and Ameya Prabhu and Shira Eisenberg and Karina Nguyen and Matthias Bethge and Maksym Andriushchenko , year =. 2603.08640 , archivePrefix =

    arXiv

  15. [15]

    The Twelfth International Conference on Learning Representations , year =

    Let's Verify Step by Step , author =. The Twelfth International Conference on Learning Representations , year =

  16. [16]

    Bowman , booktitle =

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle =. 2024 , url =

    2024

  17. [17]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs? , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  18. [18]

    2025 , url =

    Yushi Bai and Shangqing Tu and Jiajie Zhang and Hao Peng and Xiaozhi Wang and Xin Lv and Shulin Cao and Jiazheng Xu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li , booktitle =. 2025 , url =

    2025

  19. [19]

    2024 , url =

    Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen , booktitle =. 2024 , url =

    2024

  20. [20]

    Hjalmar Wijk and Tao Lin and Joel Becker and Sami Jawhar and Neev Parikh and Thomas Broadley and Lawrence Chan and Michael Chen and Joshua M. Clymer and Jai Dhyani and Elena Ericheva and Katharyn Garcia and Brian Goodrich and Nikola Jurkovic and Megan Kinniment and Aron Lajko and Seraphina Nix and Lucas Jun Koba Sato and William Saunders and Maksym Taran ...

    2025

  21. [21]

    2503.17354 , archivePrefix =

    David Rein and Joel Becker and Amy Deng and Seraphina Nix and Chris Canal and Daniel O'Connel and Pip Arnott and Ryan Bloom and Thomas Broadley and Katharyn Garcia and Brian Goodrich and Max Hasin and Sami Jawhar and Megan Kinniment and Thomas Kwa and Aron Lajko and Nate Rush and Lucas Jun Koba Sato and Sydney Von Arx and Ben West and Lawrence Chan and El...

    Pith/arXiv arXiv

  22. [22]

    Ziegler and Elizabeth Barnes and Lawrence Chan , year =

    Thomas Kwa and Ben West and Joel Becker and Amy Deng and Katharyn Garcia and Max Hasin and Sami Jawhar and Megan Kinniment and Nate Rush and Sydney Von Arx and Ryan Bloom and Thomas Broadley and Haoxing Du and Brian Goodrich and Nikola Jurkovic and Luke Harold Miles and Seraphina Nix and Tao Lin and Neev Parikh and David Rein and Lucas Jun Koba Sato and H...

    Pith/arXiv arXiv

  23. [23]

    Chris Lu and Cong Lu and Robert Tjarko Lange and Jakob Foerster and Jeff Clune and David Ha , year =. The. 2408.06292 , archivePrefix =

    Pith/arXiv arXiv

  24. [24]

    Advances in Neural Information Processing Systems 24 , year =

    Algorithms for Hyper-Parameter Optimization , author =. Advances in Neural Information Processing Systems 24 , year =

  25. [25]

    Journal of Machine Learning Research , volume =

    Marius Lindauer and Katharina Eggensperger and Matthias Feurer and Andr. Journal of Machine Learning Research , volume =. 2022 , url =

    2022

  26. [26]

    2022 , url =

    Gyeong-In Yu and Joo Seong Jeong and Geon-Woo Kim and Soojeong Kim and Byung-Gon Chun , booktitle =. 2022 , url =

    2022

  27. [27]

    Taming Throughput-Latency Tradeoff in

    Amey Agrawal and Nitin Kedia and Ashish Panwar and Jayashree Mohan and Nipun Kwatra and Bhargav Gulavani and Alexey Tumanov and Ramachandran Ramjee , booktitle =. Taming Throughput-Latency Tradeoff in. 2024 , url =

    2024

  28. [28]

    Gonzalez and Clark Barrett and Ying Sheng , booktitle =

    Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng , booktitle =. 2024 , url =

    2024

  29. [29]

    Gonzalez and Hao Zhang and Ion Stoica , booktitle =

    Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica , booktitle =. Efficient Memory Management for Large Language Model Serving with. 2023 , url =

    2023

  30. [30]

    2024 , howpublished =

    2024

  31. [31]

    Fu and Stefano Ermon and Atri Rudra and Christopher R

    Tri Dao and Daniel Y. Fu and Stefano Ermon and Atri Rudra and Christopher R. Advances in Neural Information Processing Systems 35 , year =

  32. [32]

    2024 , url =

    Tri Dao , booktitle =. 2024 , url =

    2024

  33. [33]

    2407.08608 , archivePrefix =

    Jay Shah and Ganesh Bikshandi and Ying Zhang and Vijay Thakkar and Pradeep Ramani and Tri Dao , year =. 2407.08608 , archivePrefix =

    Pith/arXiv arXiv

  34. [34]

    2025 , url =

    Zihao Ye and Lequn Chen and Ruihang Lai and Wuwei Lin and Yineng Zhang and Stephanie Wang and Tianqi Chen and Baris Kasikci and Vinod Grover and Arvind Krishnamurthy and Luis Ceze , booktitle =. 2025 , url =

    2025

  35. [35]

    2026 , eprint=

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. 2026 , eprint=

    2026

  36. [36]

    2025 , eprint=

    Inverse Scaling in Test-Time Compute , author=. 2025 , eprint=

    2025

  37. [37]

    2019 , url =

    Philippe Tillet and Hsiang-Tsung Kung and David Cox , booktitle =. 2019 , url =

    2019

  38. [38]

    2023 , url=

    Elias Frantar and Saleh Ashkboos and Torsten Hoefler and Dan Alistarh , booktitle=. 2023 , url=

    2023

  39. [39]

    2024 , url =

    Ji Lin and Jiaming Tang and Haotian Tang and Shang Yang and Wei-Ming Chen and Wei-Chen Wang and Guangxuan Xiao and Xingyu Dang and Chuang Gan and Song Han , booktitle =. 2024 , url =

    2024

  40. [40]

    2209.05433 , archivePrefix =

    Paulius Micikevicius and Dusan Stosic and Neil Burgess and Marius Cornea and Pradeep Dubey and Richard Grisenthwaite and Sangwon Ha and Alexander Heinecke and Patrick Judd and John Kamalu and Naveen Mellempudi and Stuart Oberman and Mohammad Shoeybi and Michael Siu and Hao Wu , year =. 2209.05433 , archivePrefix =

    Pith/arXiv arXiv

  41. [41]

    Proceedings of the 40th International Conference on Machine Learning , series =

    Fast Inference from Transformers via Speculative Decoding , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , url =

    2023

  42. [42]

    2023 , eprint =

    Accelerating Large Language Model Decoding with Speculative Sampling , author =. 2023 , eprint =

    2023

  43. [43]

    Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle =

    Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle =. 2022 , url =

    2022

  44. [44]

    Zhang and William Hu and Christopher R

    Anne Ouyang and Simon Guo and Simran Arora and Alex L. Zhang and William Hu and Christopher R. Proceedings of the 42nd International Conference on Machine Learning , year =

  45. [45]

    2025 , eprint=

    Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization , author=. 2025 , eprint=

    2025

  46. [46]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    2026

  47. [47]

    2025 , howpublished =

    2025

  48. [48]

    Autoresearch:

    Andrej Karpathy , year =. Autoresearch: