Pith. sign in

REVIEW 2 major objections 6 minor 37 references

Token reduction is not cost reduction: cache traffic and added turns set the bill.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:39 UTC pith:M6UIQJOL

load-bearing objection A rare empirical paper where the headline claim holds up: token reduction is not cost reduction, and the evidence is as careful as it gets for paid API campaigns. the 2 major comments →

arxiv 2607.12161 v2 pith:M6UIQJOL submitted 2026-07-13 cs.CL

Token Reduction Is Not Cost Reduction

Sarel Weinberger , Amir Hozez This is my paper
classification cs.CL
keywords token reductionbilled costprompt cachingcoding agentscontext compressioncost per successful executiontrajectory effectspaired evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that for API-based coding agents, how many tokens a context-reduction layer removes is a poor, sometimes actively misleading, predictor of the provider-billed cost of finishing a task. Across 2,848 paired, provider-billed runs on three model tiers, cache creation and cache reads accounted for about 87% of the reconstructed cost and about 80% of the actual bill. The decisive counter-example: an arm that removed an estimated 38.4% of raw tool-output tokens cost 6.8% more than baseline, because the agent is a closed loop and compressed context can prompt re-searches and extra turns, each of which re-transmits the entire cached prefix. Compression also destroyed byte-exact edit anchors in a single-shot study, cutting patch application from 27/40 to 15/40. The paper argues that context-reduction systems should be judged by success-adjusted billed cost per task, not by tokens removed.

Core claim

The central claim is that end-to-end billed cost for API-based coding agents is governed by prompt-cache traffic and trajectory length, not by the volume of tool output removed. With cache writes billed at 1.25x and cache reads at 0.1x the input price, a token already inside the cached prefix is cheap to re-read, and the real cost driver is how many turns re-transmit that prefix. The headline evidence is an arm that delivered 38.4% fewer estimated raw tool-output tokens yet had +6.8% paired billed cost (95% CI +2.8% to +11.3%), with per-task token reduction a weak predictor of cost change (Pearson r=0.15, CI crossing zero). Compression also destroyed byte-exact edit anchors in a single-shot

What carries the argument

The load-bearing object is a four-component cost model (uncached input, cache-creation, cache-read, generated output) that reconstructs each run's provider bill at published prices with cache write multiplier 1.25 and read multiplier 0.1, calibrated to a median per-run residual of about 1%. On this sits the distinction between a layer's addressable share, the fraction of cost its context can modify, and its realized saving, the paired change in billed cost; the gap is the trajectory term, because every added or removed turn re-transmits the entire cached prefix. The empirical instrument is a pre-specified, hash-frozen, block-randomized paired campaign (2,908 runs, 103 tasks, four arms from i

Load-bearing premise

The cost deltas and the ~87% cache share assume that randomizing arm order within blocks neutralizes provider-side prompt-cache carryover between arms; all consecutive runs in a block fall inside the five-minute cache TTL, and temporal separation cannot rule out carryover.

What would settle it

Re-run the paired protocol with cache isolation per arm (separate cache namespace, or gaps longer than five minutes between runs, or cache-disabled billing): if the +6.8% cost delta and the ~87% cache share vanish, cross-arm prompt-cache carryover drove the headline. Alternatively, instrument every cache write and read with source labels (system prompt, tool output, assistant history, hook text) to test whether the unattributed 8.7% residual is thinking-token billing and to measure which components are truly addressable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Component-level token savings alone cannot license efficiency claims; claims must be settled by paired, actually-billed runs with task success and trajectory length.
  • Trajectory-neutral compression is the only kind that pays: removing a token from an early turn saves its cache write plus future discounted reads, but any added turn repays the saving at full trajectory rates.
  • Content policy must distinguish dense evidence (tracebacks, test output, patch anchors, program-consumed streams) from redundant retrieval context; compressing the former measurably harms task execution.
  • A layer's effect is not portable: the same API-boundary proxy carried a large cost penalty on one agent harness and a reconstructed saving on another, so each deployment needs its own end-to-end measurement.
  • Cost per successful execution, not token reduction, is the decision-grade metric; ratio metrics such as 'successes per million tokens' can invert the verdict when success sets are tiny and non-overlapping.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Per-token addressability should depend on turn position: a token delivered on the first turn costs about 1.6x the input price (cache write plus about 3.5 discounted reads), versus 1.25x for a final-turn token; instrumenting per-token entry turn and reuse count would test this directly.
  • If the trajectory-repayment mechanism is right, then compression that holds trajectory length fixed should show savings near its addressable share, while any fraction of an added turn should push net cost upward; the paper's two hook-based arms are consistent with that boundary.
  • The unexplained billing residual scales with thinking-effort on one model but not another, suggesting provider usage fields do not uniformly expose thinking charges; cross-model token-efficiency comparisons ought to normalize for this or risk spurious savings.
  • Because the tested proxy moved billed cost while reporting zero compressed tokens in both directions, billed-cost measurement is a necessary screen for any optimization layer even when its internal counters show no activity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper reports a large empirical study of context-reduction layers for API-based coding agents. The primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs (2,848 analyzed) over 103 tasks, 7 repositories, and 3 models, comparing a baseline, two hook-based compression generations (RTK, RTK-ML), and an API-boundary proxy (Headroom). The authors calibrate a four-component cost model (eq. 1) to provider usage fields, finding cache creation and cache reads account for about 87% of reconstructed cost (about 80% of the bill), with an 8.7% dollar-weighted residual. They report that RTK-ML removed an estimated 38.4% of raw tool-output tokens yet incurred +6.8% paired billed cost (95% CI [+2.8,+11.3]), that per-task reduction is weakly correlated with cost change (r=0.15), that Headroom costs +48.4%, and that in a separate single-shot SWE-bench-derived Go study compression reduced patch application from 27/40 to 15/40. They propose a layered evidence taxonomy (L1–L8) and success-adjusted billed cost as the decision metric.

Significance. If the empirical claims hold, this is an important, field-relevant result. The study is unusually careful: hash-frozen manifests and analysis plans, provider-billed costs cross-checked against usage fields, task-clustered bootstrap CIs, holdout-only confirmations, order-sensitivity checks, disclosed exclusions and residuals, append-only ledgers, and offline-reproducible analysis scripts. The paper's strongest contribution is the cost anatomy showing cache traffic dominates in these workloads, and the demonstration that a plausible-looking token-reduction metric can invert the efficiency verdict. The Codex replication, though reconstructed-cost, is a useful stress test. However, the quantitative cost deltas are not fully secured: the prompt-cache carryover threat within the five-minute TTL is acknowledged but not resolved, and the headline reduction-cost comparison is endogenous and concentrated. These issues make the central claims defensible in direction but uncertain in magnitude.

major comments (2)
  1. [§5.3, Table 5] The headline paired cost deltas (Table 7) are exposed to prompt-cache carryover: 100% of 2,136 consecutive run pairs within a block lie inside the provider's five-minute cache TTL, with a median gap of 10.8 s. The order-sensitivity check is underpowered to exclude the threat. For RTK-ML the order difference is +7.49 pp (CI [-2.32, +18.01]), and the direction is exactly what asymmetric carryover would predict: when RTK-ML runs first, the later baseline can reuse its cached prefix, inflating the delta; when it runs after baseline, the reverse can deflate it. The position-based cache-write check addresses shared-prefix reuse, not arm-specific asymmetries, which are plausible because the arms differ in system prompts, hook layers, and request shapes. This is load-bearing for the +6.8% RTK-ML and +48.4% Headroom results; a TTL-separated or cache-flush control, or a clear downgrade of these de
  2. [§7.2, Fig. 2] The central decoupling claim rests on an endogenous and highly concentrated comparison. The 38.4% reduction is measured only for RTK arms using a local BPE tokenizer, and is not an externally assigned treatment dose; 87/100 Haiku tasks saw <0.1% reduction, the Pearson r=0.154 CI crosses zero, and the point estimate rises to 0.24 when the five highest-reduction tasks are removed. The paper does label the analysis descriptive, but the abstract and conclusion state 'component-level reduction did not predict end-to-end cost movement' as a finding. The manuscript should either supply a designed dose-response experiment (varying compression strength on matched tasks) or carry the concentration and endogeneity caveats into every place the r=0.15 result is cited.
minor comments (6)
  1. [§7.2] Typo: 'observable for the the RTK arms'.
  2. [Table 5] The column header 'n T' should be defined (number of tasks) in the caption or text.
  3. [§6.1] Spacing typo: 'implied+753to +1,283tokens'.
  4. [Abstract / §5.2] The abstract says '2,908 provider-billed runs, of which 2,848 analyzed'; consider stating '2,908 executed, 2,848 analyzed' to avoid confusion.
  5. [§9] The Codex replication uses reconstructed costs from token counts, not provider-billed costs; this is disclosed, but the section title 'Cross-Agent Replication' could be read more strongly than 'reconstructed-cost replication'.
  6. [§14] The separate exposure of thinking tokens is an important limitation; consider moving or cross-referencing the Haiku residual/thinking discussion from §6.1 into the limitations section so that the limitations section is self-contained.

Circularity Check

0 steps flagged

No significant circularity: headline claims are measured outcomes from provider-billed runs, not predictions of a fitted model.

full rationale

The paper's central claims are empirical measurements from 2,848 paired provider-billed runs: cost-composition shares reconstructed via eq. (1) with provider-published multipliers calibrated against the independent billed total; end-to-end paired cost deltas (RTK-ML +6.8%, Headroom +48.4%); per-task correlation r=0.15; and the single-shot grounding results. None of these reduces to an input by construction. Eq. (1) is an accounting identity with externally published prices; the calibration to total_cost_usd is a validation, not a fit of the outcome being predicted. The 38.4% reduction is measured from the hook-side ledger with a local tokenizer, and the +6.8% cost delta is the provider-billed paired difference; the two are separate measurements, not definitions. Section 7.2 explicitly labels the reduction as endogenous and reads r descriptively, so the paper does not fit a parameter and then call it a prediction. The paper contains no load-bearing self-citation: references to Anthropic pricing documentation and external benchmarks are independent, and internal references to earlier campaigns are provenance, not justifications. Acknowledged limitations (Section 5.3 cache carryover, Section 14 thinking-token exposure and residual) are validity threats that could bias the measurements, but they do not make any derivation equivalent to its inputs. The cost-anatomy claim depends on the calibration and the disclosed 8.7% residual, but the residual is explicitly quantified and the claim is reported against the actual bill, so this is a measurement with uncertainty, not circular reasoning.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are introduced. The central claims rest on external pricing semantics, randomized-block validity under cache carryover, judge-based success measurement, tokenizer comparability, and the diagnostic value of the small single-shot Go study. The only numerical inputs to the cost model are provider-published cache multipliers; no parameter is fitted to the outcome.

free parameters (1)
  • Prompt-cache price multipliers (write mu_w, read mu_r) = mu_w = 1.25, mu_r = 0.1 (five-minute TTL)
    Taken from Anthropic's published pricing schedule and confirmed by calibration against provider-billed totals (median per-run residual ≈1%); alternative tiers rejected. Not free in the sense of being tuned to the outcome, but assumed a priori and load-bearing for the 87% cache-share claim.
axioms (5)
  • domain assumption Provider usage fields (input_tokens, cache_creation_input_tokens, cache_read_input_tokens, output_tokens) and published list prices with mu_r=0.1, mu_w=1.25 describe most of the billed cost (eq. 1).
    Section 6.1: calibration reproduces individual bills with median residual ≈1%, but leaves 8.7% dollar-weighted aggregate residual; the decomposition is load-bearing for the cost-anatomy headline.
  • domain assumption Randomized block order and fresh per-run working copies make within-block comparisons unbiased despite provider-side prompt-cache carryover.
    Section 5.3: all consecutive pairs fall inside the five-minute cache TTL, so temporal separation cannot rule out carryover; order-sensitivity checks show null interactions but do not prove absence.
  • domain assumption Deterministic judges and test-based success scoring are a valid measure of task success.
    Section 5.2 and 14: near-ceiling success 96-98% limits sensitivity; graded quality, code maintainability, and human review effort were not measured.
  • domain assumption The local BPE tokenizer (tiktoken o200k_base) gives comparable estimates of raw vs delivered tool-output tokens across arms.
    Section 2.1: estimates, not provider token counts; the -38.4% reduction comparison uses the same tokenizer on both counts.
  • domain assumption Single-shot SEARCH/REPLACE performance on 40 SWE-bench-derived Go rows is diagnostic of compression's effect on edit-anchor availability.
    Section 7.4: small n, one model, one edit format; the authors explicitly label it mechanism evidence only.

pith-pipeline@v1.3.0-alltime-deepseek · 32704 in / 13326 out tokens · 139044 ms · 2026-08-02T06:39:29.083693+00:00 · methodology

0 comments
read the original abstract

Context-reduction layers for API-based coding agents, including command-output compressors, retrieval rankers, and API-boundary proxies, are commonly evaluated by how much context or tool output they remove. We ask a different question: which interventions actually reduce end-to-end billed cost while preserving task success? Our primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs, of which 2,848 were analyzed, covering 103 tasks, seven repositories, and three models. The campaign compared a baseline with two generations of hook-based compression and an API-boundary proxy within a broader measured program of roughly 5,500 billed executions. Three findings emerge. First, prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost (about 80% of the actual bill), with an 8.7% dollar-weighted residual not attributable from retained telemetry. Second, local payload reduction was not a reliable predictor of end-to-end billed cost. An arm that removed 38% of estimated raw tool-output tokens incurred 6.8% higher paired cost (95% CI: +2.8% to +11.3%), while per-task reduction showed only a weak association with cost change (Pearson r = 0.15). Third, aggressive compression can remove action-critical evidence: on SWE-bench-derived Go tasks, compression reduced successful patch application from 27/40 to 15/40 by corrupting verbatim edit anchors. We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.

Figures

Figures reproduced from arXiv: 2607.12161 by Amir Hozez, Sarel Weinberger.

Figure 1
Figure 1. Figure 1: Cost decomposition as shares of actual billed cost (n=2,848 analyzed runs; component prices as in eq. (1)). The hatched fifth segment is the unattributed billed-cost residual (billed minus reconstructed). Numbers inside segments are percentages; run-bootstrap 95% CIs for every segment are in table 6. Normalized four-component shares appear in Appendix A.3 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-task estimated raw tool-output reduction (local BPE tokenizer) vs. paired billed-cost change (RTK-ML arm, Haiku 4.5; each point one task, n = 100; task means over paired blocks; red squares: binned means with bootstrap 95% CIs). Pearson r = 0.154 [-0.051, +0.356]; Spearman ρ = 0.013 [-0.082, +0.339] (task bootstrap, 10,000 resamples, seed 7). Observed reduction is endogenous to the arm’s own trajectory… view at source ↗
Figure 3
Figure 3. Figure 3: Cost per successful execution by system and model (whiskers: task-bootstrap 95% CIs; 10,000 resamples, seed 7; success and run counts in table 9) [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Billed cost vs. trajectory length (assistant turns) per run, by model (n=2,848 runs; log-scaled cost). Each added turn re-transmits the cached prefix, so trajectory changes dominate per-turn savings [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Paired billed-cost change vs. baseline by model–effort cell (task-clustered 95% CIs). Unmeasured cells (Sonnet 5 / Opus 4.8 at medium, xhigh, max) are annotated, not imputed. with 131/132 task success. The qualitative pattern—no free lunch from compression on long sessions, large proxy overhead—is consistent with the authoritative campaign. 8 Heterogeneity by Model, Effort, and Task Family 8.1 Model × effo… view at source ↗
Figure 6
Figure 6. Figure 6: Paired billed-cost change by task family (task counts in parentheses; CIs where ≥3 tasks). Headroom is right of zero in every family; RTK effects change sign across families [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Generated-token composition by phase (characterization split; 10,376 turns). Coarse phase-token shares were similar across arms under the frozen turn-level taxonomy. than in the measured cost and context volume associated with those phases (delivered tool bytes and the cache traffic they induce). 10.3 Where savings are created and repaid The phase-resolved cost waterfall (table 16) shows both the RTK arms … view at source ↗
Figure 8
Figure 8. Figure 8: Left: median cache-read context at the first code edit (runs with an implementation phase, base splits, all models). Right: median estimated delivered tool tokens before the first edit (Haiku runs; bytes/4 estimate). Headroom arrives at the first edit ≈49% heavier; the RTK arms match baseline context at the same milestone. approximately 24% lower, with heavy upper tails (a frozen campaign report characteri… view at source ↗
Figure 9
Figure 9. Figure 9: Layered evidence taxonomy for token-efficiency claims. Every system in table 1 is placed at the highest layer its retained artifacts support. Embedding retrieval. The BGE embedding leg was enabled, health-checked, and outage-free throughout the authoritative campaign, and its component-level union configurations improved recall-at-savings on ContextBench slices; but no evaluated task shows a decisive embed… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 15 linked inside Pith

  1. [1]

    LoCoEval: Long-conversation evaluation pipeline.https://anonymous.4open.science/ r/LoCoEval, 2026

    Anonymous. LoCoEval: Long-conversation evaluation pipeline.https://anonymous.4open.science/ r/LoCoEval, 2026. Anonymized research artifact; accessed 2026-06

    2026

  2. [2]

    Prompt caching — claude developer platform documentation.https://platform.claude

    Anthropic. Prompt caching — claude developer platform documentation.https://platform.claude. com/docs/en/build-with-claude/prompt-caching, 2026. Accessed 2026-07

    2026

  3. [3]

    Claude code documentation.https://code.claude.com/docs, 2026

    Anthropic. Claude code documentation.https://code.claude.com/docs, 2026. Accessed 2026-07

    2026

  4. [4]

    Claude developer platform pricing

    Anthropic. Claude developer platform pricing. https://platform.claude.com/docs/en/pricing,

  5. [5]

    ast-grep: A CLI tool for code structural search, lint, and rewriting

    ast-grep contributors. ast-grep: A CLI tool for code structural search, lint, and rewriting. https: //ast-grep.github.io, 2026. Version 0.43.0; accessed 2026-07

    2026

  6. [6]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2308.14508

    Pith/arXiv arXiv 2024

  8. [8]

    FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. 28

    Pith/arXiv arXiv 2021

  10. [10]

    ContextBench dataset

    ContextBench contributors. ContextBench dataset. Hugging Face datasetContextbench/ContextBench, configuration contextbench_verified, 2026. Dataset revision hash not recorded in the retained artifacts; row identifiers listed in the reproducibility appendix

    2026

  11. [11]

    SWE-bench_pro dataset (go split)

    ContextBench contributors. SWE-bench_pro dataset (go split). Hugging Face dataset Contextbench/SWE-bench_Pro, test split, 2026. Accessed 2026-05; Go rows selected by repository language

    2026

  12. [12]

    CodeBERT: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. InFindings of the Association for Computational Linguistics: EMNLP 2020, 2020

    2020

  13. [13]

    ripgrep: Recursively search directories for a regex pattern.https://github.com/ BurntSushi/ripgrep, 2026

    Andrew Gallant. ripgrep: Recursively search directories for a regex pattern.https://github.com/ BurntSushi/ripgrep, 2026. Accessed 2026-07

    2026

  14. [14]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. InProceedings of Machine Learning and Systems (MLSys), 2024

    2024

  15. [15]

    Headroom: An API-boundary context optimization proxy

    Headroom Labs. Headroom: An API-boundary context optimization proxy. https://github.com/ headroomlabs-ai/headroom, 2026. Version 0.27.0; open-source distribution; accessed 2026-07

    2026

  16. [16]

    Code- SearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- SearchNet challenge: Evaluating the state of semantic code search.arXiv preprint arXiv:1909.09436, 2019

    Pith/arXiv arXiv 1909

  17. [17]

    LLMLingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  18. [18]

    LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  19. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770

    Pith/arXiv arXiv 2024

  20. [20]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023

    2023

  21. [21]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  22. [22]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  23. [23]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  24. [24]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 29

    2023

  25. [25]

    Introducing SWE-bench Verified

    OpenAI. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/, 2024. Accessed 2026-07

    2024

  26. [26]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  27. [27]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. arXiv:2312.03863

    Pith/arXiv arXiv 2024

  28. [28]

    Executable code actions elicit better LLM agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2402.01030

    Pith/arXiv arXiv 2024

  29. [29]

    Xu, Xiangru Tang, Mingchen Zhuge, et al

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, et al. OpenHands: An open platform for AI software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    C-Pack: Packaged resources to advance general Chinese embedding.arXiv preprint arXiv:2309.07597, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged resources to advance general Chinese embedding.arXiv preprint arXiv:2309.07597, 2023. BGE embedding model family

    Pith/arXiv arXiv 2023

  31. [31]

    Retrieval meets long context large lan- guage models

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large lan- guage models. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.03025

    Pith/arXiv arXiv 2024

  32. [32]

    InterCode: Standardizing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Standardizing and benchmarking interactive coding with execution feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.14898

    Pith/arXiv arXiv 2023

  33. [33]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2405.15793

    Pith/arXiv arXiv 2024

  34. [34]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  35. [35]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Multi-SWE-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, et al. Multi-SWE-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    18/21” and “21/21

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. 30 A Reproducibility Appendix A.1 Artifact map and ca...

    2023