Pith. sign in

REVIEW 6 minor 28 references

QuoteBench: How Matched Scores Can Hide Command-Path Failures

T0 review · 0 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Matched execution scores can hide command-path failures: replaying the same stored reply through one added parser costs 55.4–73.2 points, and disclosing the boundary recovers 30.4–60.7, so a near-zero matched gap can mask two large…

desk verdict The fixed-reply crossover design actually delivers: matched scores here demonstrably hide a large transport effect, and the paper is unusually honest about where it could still be wrong. read the letter →

arxiv 2608.13547 v1 pith:67TQULDD submitted 2026-08-13 cs.AI cs.SE

Shangao Li , Yao Zhang , Volker Tresp , Yuanyuan Yang This is my paper · ORCID
classification cs.AIcs.SE
keywords Bashquotingcommand-pathfailuresmatchedexecutionscorestransportdamagecontract-conditionedcompensationLLMcodingagentsfinal-statevalidationnested
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QuoteBench isolates a failure mode that ordinary leaderboard scores hide: a command that is correct on the direct shell path can fail when an interface layer wraps, quotes, or re-parses it. Across eight fixed model configurations, replaying the same stored command through one extra double-quoted parser lowers success by 55.4 to 73.2 percentage points, and telling the model about that boundary recovers 30.4 to 60.7 points for six of the eight. Because these two effects are large and opposite, a matched score can look nearly unchanged; GPT-5.6-sol's gap of $-3.6$ points is the sum of $-64.3$ points of transport damage and $+60.7$ points of compensation. If the paper is right, reporting only matched execution scores is insufficient: the command path, generation contract, and validator are part of the measured system, and model rankings can change when the deployment path changes.

What carries the argument

The crossed $2\times2$ design with cells $RR$, $RN$, $NR$, $NN$, where generation contract precedes transport, together with the identity $Y_{NN}-Y_{RR} = (Y_{RN}-Y_{RR}) + (Y_{NN}-Y_{RN})$. The "nested transport" is one deliberately unescaped double-quoted parser added to the command path, standing in for remote, container, or CI wrappers. Fixed-reply replay keeps the stored command constant so any change between $RR$ and $RN$ is attributable to the parser, and exact final-state validators score only the resulting files, argument vectors, JSON, directory state, or Git history. Escaping at the interpolation point reproduces raw-path outcomes, closing the attribution loop by showing the damage is removed when the boundary is handled correctly.

What would settle it

Re-run the 448 public fixed-reply pairs with correct escaping at the interpolation point: the paper's claim predicts that every raw-path success is restored. A single pair in which correct escaping does not restore the raw-path outcome, or a validator false accept or reject found by additional mutation classes, would break the attribution of the loss to the unescaped parser.

Watch

Extended reading notes

Core claim

The central discovery is an empirical decomposition: matched score equals transport damage plus contract-conditioned compensation, where damage is fixed-reply transport loss and compensation is the recovery from telling the model about the boundary. Using 56 tasks and exact final-state validation, the paper crosses generation contract (raw versus disclosed-boundary) with execution transport (raw versus a nested double-quoted parser). Fixed-reply replay, comparing $RN$ against $RR$, shows a loss in every configuration; the disclosed-boundary contract, comparing $NN$ against $RN$, recovers most of it in six configurations. Correct escaping at the interpolation point reproduces the raw-path outcome for all 448 public pairs, so the loss is attributable to the unescaped transform rather than to model error. The paper concludes that matched scores should be reported alongside the full command path, and that raw generation is nearly saturated at the frontier while boundary adaptation is what still separates models.

Load-bearing premise

The load-bearing premise is that the final-state validators and the execution harness correctly classify success on both the raw and nested paths for all 56 tasks; the paper acknowledges that other validator blind spots may remain, and a wrong accept or reject would shift the damage and compensation numbers.

Editorial extensions

If this is right

  • A matched execution score alone cannot tell whether a model generated a wrong command or a correct command that was mangled by the execution path.
  • Model leaderboards are path-dependent: one unambiguous reversal and four single-task-margin reversals appear when comparing raw and nested orderings, so rankings should be reported together with the deployment configuration.
  • Effort-ladder gains in matched score can be contract-conditioned compensation rather than improved cross-path portability; Opus-4.8's matched gap moves from $-48.2$ to $-3.6$ points while its nested-replay pass rate stays near its starting level.
  • Correct escaping at the interpolation point, or a temporary-script transport, restores every raw-path success in the public replays, so the transport damage is removable at the harness side.
  • Evaluators of command-issuing agents should report the generation contract, execution transport, operating point, and final-state validator, because raw generation is near saturation at the frontier and the remaining signal is boundary adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical diagnostic follows from the masking threshold: for any model with a small matched gap, report the two off-diagonal cells, since a near-zero gap is exactly the case where damage and compensation may both be large.
  • The same four-cell decomposition should transfer to any pipeline that transforms generated output before execution, so matched scores in tool-use, structured-action, and serialization benchmarks may carry the same ambiguity even where the paper does not test them.
  • Because validator completeness is the load-bearing premise, adding new mutation classes or independent final-state checkers on the private payloads is a testable way to check whether the absolute damage and compensation numbers shift while the sign pattern persists.
  • A further testable extension is to make the frozen 56-task core a regression audit: models that rank high on matched nested scores should also be required to show positive compensation and small transport damage, not just a high diagonal cell.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 6 minor

Summary. QuoteBench is a 56-task, one-shot Bash-command benchmark built from 14 incident-derived operation families, each with one benign control and three hazardous payload variants, scored by exact final-state validators. The paper's central contribution is a crossed design: for each of eight same-window model configurations it separates the generation contract (raw versus a disclosed-boundary contract that tells the model its reply will be interpolated inside double quotes) from the execution transport (raw versus a nested double-quoted parser), and replays stored replies across the four cells without additional model calls. The key empirical finding is that moving a fixed raw reply through the added parser lowers success by 55.4–73.2 percentage points in every configuration, while the disclosed-boundary contract recovers 30.4–60.7 points for six configurations; the matched gap is thus an identity sum of two large opposing components (Eq. (1)), illustrated by GPT-5.6-sol's -3.6 = -64.3 + 60.7. Robustness evidence includes a 197/197 validator mutation audit, oracle and naive-probe checks, a correct-escaping control that reproduces every raw-path outcome for all 448 public replay pairs, real-ssh grounding (seven of eight configurations match to the decimal), leave-one-family-out ranges with Holm-adjusted family-sign tests, three-draw repetitions, and a private 42-payload crossover.

Significance. Assuming the measurements are correct, this is a significant evaluation-methodology result. It provides a rare controlled decomposition of a matched score into transport damage and contract-conditioned compensation, with a fixed-reply design that avoids confounds from regeneration. The finding that raw-path scores are near-saturated at the frontier while nested-path scores separate models, and that one model pair reverses under the two paths, gives concrete evidence that leaderboard comparisons of command-issuing agents are path-dependent. The paper is unusually careful: the decomposition is an identity and is labeled as such; the primary mechanism analysis is restricted to same-window configurations; the 'compensation' is described as a realized contrast over stored generations, not a fitted parameter; and the limitations (purposive family set, observational native-tool campaign, single stored generations on effort ladders, unenumerated validator blind spots) are stated explicitly in Section 3.1, Appendix F.1, and the Limitations section. The release of the harness, exact prompts, SHA-256 manifest, and sanitized rollout archive supports reproducibility.

minor comments (6)
  1. [Abstract / Section 4.2] Please fix the typographical spacing errors, including 'QuoteBenchmeasures' in the abstract and 'writtenpoints' in Section 4.2, and clean up the missing spaces around minus signs and percent symbols throughout (for example, '-64.3damage', '+60 .7compensation', and '-3.6points').
  2. [Section 4.2 / Table 4] The sentence 'The realized generation-by-transport interaction ... ranges from -7.1 to +119.6 points across the eight configurations (Table 4)' cites Table 4, but Table 4 does not report the interaction column; either add that column or cite Appendix Table 14 instead.
  3. [Section 4.3] The family-level compensation values quoted in prose (json-write +50.0, sed-replace +46.9, find-glob -12.5, grep-count +15.6, hostile-filenames +18.8) are not tabulated anywhere in the manuscript; please add a family-level compensation table or point to a specific artifact table so the claim that compensation concentrates where the hazard is explicit can be checked.
  4. [Section 3.1 / Appendix F.1] Because the headline damage and compensation numbers depend on exact final-state labels, I suggest adding one sentence stating what a hypothetical validator blind spot would need to look like (systematically transport-correlated and large enough to remove most of the 55.4-73.2 pp damage) to threaten the conclusion; the existing controls already make that scenario implausible, but the reader should not have to assemble the argument from multiple sections.
  5. [Section 4.1] The sentence 'Raw generation itself is close to saturated at the frontier' should be scoped to 'the six frontier configurations', because Table 4's same-window raw scores range from 78.6% to 100% across all eight configurations, and the lower-scoring configurations still show substantial raw-path spread.
  6. [Appendix A.2] Appendix A.2 gives verbatim prompts for the raw and disclosed-boundary contracts but not for the native contract; please include the native-tool prompt or explicitly state that it is available in the artifact, for full reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's decomposition is an explicit identity and all contrasts are observed pass-rate differences, not fitted or self-citation-derived quantities.

full rationale

QuoteBench is an experimental measurement paper; its central claim is that matched execution scores can hide large transport damage. The only 'derivation' in the methods is Eq. (1), which the paper itself presents as an algebraic identity (YNN−YRR = (YRN−YRR)+(YNN−YRN)); it is used for bookkeeping, not to generate predictions. The damage and compensation values are directly observed pass-rate differences across fixed stored replies in the four crossed cells, with no fitted parameter that is later renamed as a prediction. The escape control (correct escaping reproduces raw-path outcomes for all 448 public pairs) is a logical control isolating the transport, not a redefinition of the input. The 'masked-fragility' cut is explicitly labeled 'illustrative reading aid (not a fitted criterion)' and does not feed back into the mechanism estimates. The paper contains no load-bearing self-citation: prior work references are contextual, and no uniqueness theorem or ansatz is imported from the authors' own prior results. Validator completeness is a correctness risk, not circularity, and the paper discloses remaining blind spots ('Unenumerated false positives remain possible'). Therefore the derivation chain is self-contained and no circular step is identifiable.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central results rest on no fitted constants. The only hand-chosen numerical construct is a descriptive masked-fragility threshold used as a reading aid in Appendix C.2. The main supporting premises are domain assumptions about shell semantics, validator correctness, and prompt equivalence; each is disclosed, and the first two are checked directly. No new theoretical entities are introduced: the nested transport is an experimental condition, not a postulated entity.

free parameters (1)
  • Masked-fragility thresholds = |matched gap| <= 5 pp; damage <= -40 pp; compensation >= 30 pp
    Hand-chosen cutoffs in Appendix C.2 used only as a descriptive reading aid to label configurations as 'descriptively masked'. They are not fitted to optimize any result and do not enter the main decomposition.
assumptions (4)
  • domain assumption Bash quoting and expansion semantics, and the pinned GNU/Linux coreutils environment, are deterministic and as documented by the Free Software Foundation.
    The benchmark's final states depend on exact shell behavior; cited to the Bash Reference Manual (Free Software Foundation, 2025) and pinned network-disabled container.
  • domain assumption The final-state validators correctly encode the target state for each of the 56 tasks.
    Supported by oracle solvability checks, naive probe tests, and 197/197 mutation rejections, but the paper acknowledges unenumerated blind spots may remain (Section 3.1).
  • domain assumption The raw and disclosed-boundary system prompts differ only in the stated execution-contract clause.
    Appendix A.2 reports a character-level diff isolating the one-sentence change; the compensation contrast relies on this equivalence.
  • domain assumption The family-sign symmetry null is a valid inferential basis for the reported p-values over the 14 constructed families.
    Appendix C defines the exact sign-flip enumeration; the resulting p-values quantify variation across the constructed families, not a sampled task population, as the paper states.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QuoteBench: How Matched Scores Can Hide Command-Path Failures." pith.science (2026). https://pith.science/paper/67TQULDD

@misc{pith2026260813547,
  author       = {Pith},
  title        = {Pith review of: QuoteBench: How Matched Scores Can Hide Command-Path Failures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67TQULDD}},
  note         = {Machine review of arXiv:2608.13547}
}
read the original abstract

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

Figures

Figures reproduced from arXiv: 2608.13547 by the authors.

Figure 1
Figure 1. Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract￾then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The RR and NN cells are matched. The RN… view at source ↗
Figure 2
Figure 2. Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The RR value is raw success, RN is the nested-replay pass rate of raw-conditioned replies, and NN is matched nested success. The NN − RN gap is compensation. Appendix [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched￾nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order.…
Figure 5
Figure 5. Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 14 canonical work pages

  1. [7]

    Accessed 2026-07-29

    URL https://github.com/google-gemini/gemini-cli/issues/1839. Accessed 2026-07-29. Vidar Holen. ShellCheck: A static analysis tool for shell scripts, 2012.https://www.shellcheck.net. 14 Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?I...

    work page 2026

  2. [8]

    ISBN 979-8-89176-390-6

    Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/ 2026.acl-long.607. URLhttps://aclanthology.org/2026.acl-long.607/. LangChain contributors. Langchain shell tool source at commit b3a6d9a. GitHub source repository,

    doi:10.18653/v1/ 2026

  3. [9]

    Accessed 2026-07-29

    URL https://github.com/langchain-ai/langchain/tree/b3a6d9a012681df8a8e33345c8255ca69ec0e437. Accessed 2026-07-29. Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. InProceedings of LREC,

    work page 2026

  4. [11]

    arXiv:2308.03688

    URL https://arxiv.org/abs/2308.03688. arXiv:2308.03688. Waldek Mastykarz. Don’t rewrite your CLI for agents. Microsoft for Developers, July

    arXiv

  5. [13]

    The released benchmark is Terminal-Bench 2.0

    URL https://arxiv.org/abs/2601.11868. The released benchmark is Terminal-Bench 2.0. Microsoft AutoGen contributors. Autogen docker code executor at commit 027ecf0. GitHub source repository,

    arXiv

  6. [14]

    Accessed 2026-07-29

    URL https://github.com/microsoft/autogen/tree/ 027ecf0a379bcc1d09956d46d12d44a3ad9cee14. Accessed 2026-07-29. OpenAI. Codex shell execution source at commit fa1d4c4. GitHub source repository,

    work page 2026

  7. [15]

    Accessed 2026-07-

    URLhttps: //github.com/openai/codex/tree/fa1d4c40d0e63eef2e0ba8a9e004ccd0a80b77f5. Accessed 2026-07-

    work page 2026

  8. [16]

    Accessed 2026-07-29

    URLhttps://github.com/openai/codex/issues/ 20875. Accessed 2026-07-29. OpenHands community. Use language server protocol to re-implement code editing. GitHub issue Open- Hands/OpenHands#1934,

    work page 2026

Show all 28 references
  1. [17]

    Moti- vated partly by weird issues from heredoc-plus-Bash editing; accessed 2026-07-29

    URL https://github.com/OpenHands/OpenHands/issues/1934. Moti- vated partly by weird issues from heredoc-plus-Bash editing; accessed 2026-07-29. 15 OpenHands contributors. Openhands argv command parser at commit 850bd64. GitHub source repository,

    1934

  2. [18]

    Accessed 2026-07-29

    URL https://github.com/All-Hands-AI/OpenHands/tree/ 850bd647b64c9a6b5d2bbf25d4d9e16a3c6f685e. Accessed 2026-07-29. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. InAdvances in Neural Information Pro...

    2026

  3. [21]

    Jiaqi Shao, Hanck Chen, Wei Zhang, Maxm Pan, and Bing Luo

    URLhttps://arxiv.org/abs/2310.11324. Jiaqi Shao, Hanck Chen, Wei Zhang, Maxm Pan, and Bing Luo. Do agent benchmarks measure capability? protocol validity in the age of agentic AI.arXiv preprint arXiv:2607.22368,

    arXiv

  4. [22]

    org/abs/2607.22368

    URLhttps://arxiv. org/abs/2607.22368. SWE-agent contributors. Swe-agent action execution source at commit 3ea751c. GitHub source repository,

    arXiv

  5. [23]

    Accessed 2026-07-29

    URL https://github.com/princeton-nlp/SWE-agent/tree/ 3ea751c087f32b16e039a2233dd6eefecef325d5. Accessed 2026-07-29. Terminal-Bench contributors. Terminal-bench tmux execution source at commit d28711d. GitHub source repository,

    2026

  6. [24]

    Accessed 2026-07-29

    URL https://github.com/laude-institute/terminal-bench/tree/ d28711d0da2675d0bb1d56de45ae5df6082438a3. Accessed 2026-07-29. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. InProceedings of t...

    2026

  7. [25]

    ISBN 979-8-89176-390-6

    Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1711. URLhttps://aclanthology.org/2026.acl-long.1711/. Warp community. Agent mode fails to create files with heredoc syntax—quote escaping issues. GitHub issue warpdotdev/Warp#7735,

    2026 doi

  8. [26]

    Accessed 2026-07-29

    URL https://github.com/warpdotdev/Warp/issues/7735. Accessed 2026-07-29. Finnian Westenfelder, Erik Hemberg, Stephen Moskal, Una-May O’Reilly, and Silviu Chiricescu. LLM- supported natural language to bash translation. InProceedings of NAACL, pp. 11135–11147,

    2026

  9. [27]

    URLhttps://aclanthology.org/2025.naacl-long.555/

    doi: 10.18653/v1/2025.naacl-long.555. URLhttps://aclanthology.org/2025.naacl-long.555/. David A. Wheeler. Fixing Unix/Linux/POSIX filenames,

    2025 doi

  10. [28]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html

    doi: 10.52202/079017-1650. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html. 16 John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Standardizing and bench- mar...

    2024 doi

  11. [29]

    URL https://proceedings.neurips.cc/paper_files/ paper/2023/hash/4b175d846fb008d540d233c188379ff9-Abstract-Datasets_and_Benchmarks.html

    doi: 10.52202/075280-1035. URL https://proceedings.neurips.cc/paper_files/ paper/2023/hash/4b175d846fb008d540d233c188379ff9-Abstract-Datasets_and_Benchmarks.html. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE...

    2023 doi

  12. [31]

    Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang

    URL https: //arxiv.org/abs/2406.12045. Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UTBoost: Rigorous evaluation of coding agents on SWE-bench. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Proceedings of the 63rd Annual Meeting of t...

    arXiv

  13. [33]

    Wenxiao Zhang, Yu Liu, Zhiwei Yang, Zhongyi Zhang, Hanqi Feng, Xinyu Wang, Peng Qiu, Yanbing Liu, Barnabas Poczos, and Jin B

    URL https://arxiv.org/abs/2606.27733. Wenxiao Zhang, Yu Liu, Zhiwei Yang, Zhongyi Zhang, Hanqi Feng, Xinyu Wang, Peng Qiu, Yanbing Liu, Barnabas Poczos, and Jin B. Hong. CARE: Pre-execution command verification for shell-executing LLM agents.arXiv preprint arXiv:2607.21642, 20...

    arXiv

  14. [34]

    arXiv:2307.13854

    URL https://arxiv.org/abs/2307.13854. arXiv:2307.13854. A Benchmark Construction, Coverage, and Contracts Two mechanism surveys guided the family design. The internal survey contains 86 de-identified incidents from author-owned coding-agent sessions: 50 Codex incidents and 36 ...

    2026 arXiv

  15. [2018]

    arXiv:1802.08979

    URL https://arxiv.org/abs/1802.08979. arXiv:1802.08979. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan S...

    arXiv

  16. [2021]

    Anthropic Claude Code community

    URLhttps://arxiv.org/ abs/2103.02523. Anthropic Claude Code community. Avoid shell quoting issues when passing markdown to cli tools like gh. GitHub issue anthropics/claude-code#29619, 2026a. URLhttps://github.com/anthropics/ claude-code/issues/29619. Accessed 2026-07-29. Anth...

    2026 arXiv

  17. [2023]

    MelanieSclar, YejinChoi, YuliaTsvetkov, andAlaneSuhr

    URLhttps://arxiv.org/abs/2307.16789. MelanieSclar, YejinChoi, YuliaTsvetkov, andAlaneSuhr. Quantifyinglanguagemodels’sensitivitytospurious features in prompt design or: How I learned to start worrying about prompt formatting. InInternational Conference on Learning Representati...

    arXiv

  18. [2024]

    Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T

    URLhttps://arxiv.org/abs/2407.21787. Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, and He Ye. TerminalWorld: Benchmarking agents on real-world terminal tasks.arXiv preprint arXiv:2605.22535,

    arXiv

  19. [2025]

    Free Software Foundation.Bash Reference Manual, Version 5.3,

    URL https: //arxiv.org/abs/2503.14443. Free Software Foundation.Bash Reference Manual, Version 5.3,

    arXiv

  20. [2026]

    URLhttps://arxiv.org/abs/2605.22535. Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, Qunhong Zeng, Qi Zhang, Xuanjing Huang, Qidi Xu, and Tao Gui. OctoBench: Benchmarking scaffold-aware instructi...

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.