Pith.

Artificial Intelligence · arXiv:2610.07781 v1 · submitted 6 Oct 2026 · 21 references

Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designs

Yuhe Hu

The paper says

“Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on how recovery is evaluated.”

more of the abstract

We compare 8-bit and 4-bit variants of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic tool-use tasks and five prompts. The 8-bit-4-bit recovery comparison changes direction across prompts and evaluation targets. On tasks that both variants complete without faults under the same prompt, the difference ranges from 0 to +20.2 percentage points for Llama and from -50.0 to +35.0 points for Qwen. Full-pipeline point estimates favor 8-bit Llama under all five prompts, whereas the Qwen comparison changes direction across prompts. The evaluation target can also reverse the result. For Llama under one prompt, scoring each variant only on its own clean-passing tasks favors 4-bit by 17.5 points; scoring the same tasks for both variants gives no difference, while scoring the full pipeline favors 8-bit by 28.3 points. Executor leniency is a third such choice. Rescoring the same logs with strict output parsing, which 8-bit Llama violates far more often than 4-bit Llama under that prompt, turns that +28.3 into -15.0 while leaving Qwen essentially unchanged. These findings show that one prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness. Evaluations should compare variants on matched tasks, report full-pipeline success for deployment decisions, state the scoring policy, and quantify uncertainty across tasks rather than injected fault sites.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 8 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Honest, well-scoped empirical warning that 8-bit versus 4-bit agent comparisons flip across prompts, task sets, and scoring policies; the central negative claim holds, but the strict-scoring reversal is a rescoring counterfactual and should be labeled as such. This line is wrong

2 major objections · 3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 21 references say.

The paper says, at §5.3, Table 2, abstract (b7)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.95):

“The strict-scoring analysis rescored existing logs without rerunning inference; a model told that its expression was rejected might behave differently.”

Pith says: Objection 1 of 2 · major

The strict-scoring reversal is computed by rescoring logs that were generated under the tolerant executor, and the clean screen is rescored under the same counterfactual policy; no model was rerun. The paper acknowledges this in Section 7 (b163), but the abstract's claim that "executor leniency is a third such choice" and the conclusion's statement that "strict scoring can reverse it" present the result as an evaluation-design finding without the same caveat.

more of objection 1

Because a model that is told its expression was rejected may change its behavior in a way that rescoring cannot capture, the reversal may not describe a strictly deployed system. I recommend rephrasing the headline claim to "rescoring the released logs under strict parsing reverses the measured comparison" and stating explicitly that the paper makes no behavioral claim about an actual strict executor; if a deployment claim under a strict executor is intended, that experiment would be needed.

The paper says, at §5.1, §5.3, Table 2

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The strongest direction-changing evidence rests on small matched-task cells: Qwen/P3 and Qwen/P4 use ten matched tasks, and the strict-scoring Llama/P4 cells use four to six tasks. No multiplicity correction is applied across the roughly sixty reported prompt-target-policy contrasts, and prompts P3 and P4 were written after early results were known.

more of objection 2

The per-comparison 95% intervals therefore overstate the strength of the across-prompt pattern. I am not asking for a different protocol, but the paper should add a sensitivity statement or adjusted intervals, or explicitly say that the direction changes are point estimates for these twenty tasks and that the significant opposite-signed intervals have not been corrected for selection across prompts and evaluation targets.

3 minor objections

The paper says, at Figure 1

Pith says: minor objection 1 of 3

The matched-task counts n are printed directly under panel B, and the caption says the number of matched tasks is shown below each prompt; make the placement visually unambiguous so readers do not confuse the n values with panel A's separate-set sample sizes.

The paper says, at Table 4

Pith says: minor objection 2 of 3

The Qwen/P4 value 0.425 has three decimal places while all other entries in the table have two; align the formatting for consistency.

show 1 more

The paper says, at §4, Uncertainty paragraph

Pith says: minor objection 3 of 3

The statement that deterministic reruns do not provide independent statistical samples is useful; consider adding one sentence noting that the 20,000 bootstrap resamples also do not add information beyond the task-level sample, since this is a common source of overconfident reading of bootstrap intervals.

What would settle it

Rerun Llama/P4 under a genuinely strict executor: same twenty tasks, same prompt, same temperature-0 decoding and fault-injection protocol, but with a harness that rejects any non-JSON turn and any submit_answer that is not a literal numeric amount. Recompute the clean screen under this policy before injecting faults, then recompute Table 2's strict columns for Llama/P4 (separate, matched, full pipeline) and compare with the rescored values (-42.9, -33.3 with n=6, and -15.0). If the actual strict runs reproduce negative differences of similar magnitude, the reversal is a measured evaluation-design effect; if the sign flips, vanishes, or changes materially, the strict-policy result should be reported only as a rescoring sensitivity, not as an executor-leniency reversal.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 8 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic partly agrees 1 min 22 s · 13,965 tokens read, 12,703 written
  3. Editor wrote the verdict above 17 s · 14,822 tokens read, 2,341 written
  4. Referee major revision: 2 major objections, 3 minor 1 min 4 s · 14,198 tokens read, 9,461 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Worth a look if you do quantized-agent evaluations. The actual new thing here is a decomposition of an 8-bit versus 4-bit tool-failure recovery comparison across five prompts, three evaluation targets, and two scoring policies, on two models. The main negative claim—neither precision has a consistent recovery advantage—is well supported as scoped. The Llama/P4 result is a nice demonstration: separate clean screening favors 4-bit by 17.5 points, matched tasks give zero, and the full pipeline favors 8-bit by 28.3 points. That trio cleanly shows why one evaluation target is not enough.

Credit where earned. The paper is unusually transparent: pinned llama.cpp commit, fixed decoding, per-episode logs, task-level bootstrap intervals, and clear definitions of separately screened, matched-task, and full-pipeline rates. Section 7 openly discloses the post-hoc P3/P4 prompts, the non-preregistered Qwen extension, deterministic reruns not counting as samples, the small all-prompt intersections, and the fact that strict scoring rescored logs without rerunning inference. No fitted parameters, no derivations, no circularity burden. If the supplementary ZIP ships what it claims, this is reproducible work.

Soft spots, in proportion. First, the twenty tasks come from two closely related templates in one synthetic environment, and several headline reversals rest on matched-task cells of 4 to 10 tasks. The paper acknowledges this, but it still means the reversals are demonstrations rather than stable effect-size estimates for real workloads. Second, no multiplicity correction across roughly sixty contrasts; I would not treat every interval that excludes zero as a confirmed effect. Third, and most important: the strict-scoring reversal in the abstract is the weakest load-bearing claim. Rescoring tolerant-executor logs is a scoring counterfactual, not a measurement under a strict executor. A model told its expression was rejected may change its behavior. Section 7 admits this, yet the paper still frames executor leniency as a third evaluation-design reversal. I would like that framing softened or tested with an actual strict executor before it carries so much weight. This does not damage the central tolerant-scoring conclusion, only the sharpest version of one secondary claim.

I could not open the supplementary ZIP from here, so the reproduction-script claims rest on the paper's word, though the in-paper artifact descriptions are specific enough to believe.

This paper is for people building or evaluating quantized LLM agents, especially deployment studies. It is a methodological caution, not a benchmark result, and it deserves a serious referee. I would send it out, with requests for a strict-executor rerun, a multiplicity-aware summary, and a clearer label on the rescoring result. I would also cite it in evaluation-methodology footnotes.

The strongest objection

The tolerant-scoring core of the paper—no consistent 8-bit vs 4-bit recovery advantage across models and prompts—is well supported, and Section 7 is candid about the narrow task suite and small intersections. The least secure load-bearing step is the strict-scoring reversal presented in the abstract and Section 5.3 (b7). Table 2's strict columns rescore logs collected under the tolerant executor, including the clean screen; the strict clean-task sets and matched-task cells are therefore counterfactual task populations, not measurements under a strict policy. The paper acknowledges in b163 that a model told that its expression was rejected might behave differently. What remains unresolved is the stronger use of the result: 'Executor leniency is a third such choice' and the conclusion that strict scoring can reverse a quantization comparison are framed as an evaluation-design finding. An actual strict executor changes the observations the model sees, so 8-bit Llama's expression-submission rate and clean-passing set could change; the +28.3 to -15.0 and 0 to -33.3 reversals may not describe a strictly deployed system. This does not overturn the tolerant-scoring conclusion, but it is the one load-bearing claim whose current evidence is a rescoring counterfactual.

Choosing the strict-executor issue as the load-bearing concern: the central negative claim is independent of strict scoring, but the paper's abstract and conclusions elevate executor leniency to one of three design choices that can reverse results, and the evidence for it is derived entirely from rescoring tolerant-policy logs. This is not a hidden flaw—Section 7 states it—but it is an unresolved acceptance condition because the paper's deployment recommendation ('use the deployed executor's policy') is exactly the setting where model adaptation to a strict executor matters. The reader's weakest_assumption lists the same rescoring issue alongside small-sample and multiplicity concerns; I agree with the conditions but prioritize them differently, so the reader's CONDITIONAL verdict does not need to move. The small matched-task cells and the lack of a multiplicity correction are real but disclosed, and they do not threaten the negative claim; wider intervals would only weaken positive attributions and strengthen 'no consistent advantage.' The strict-executor rerun is a single, cheap, well-defined check on one model/prompt cell and would settle whether the strict-policy reversal is a measured evaluation-design effect or a rescoring artifact.

The referee's report

The main tolerant-scoring conclusion is well supported and is a useful negative result for the quantized-agent evaluation literature: it documents, with released per-episode logs and pinned decoding settings, that a precision-recovery comparison can be an artifact of screening, evaluation-target, and scoring choices. The paper's reproducibility practices are a real strength: fixed seed, temperature-0 decoding, pinned llama.cpp commit, logged model-file revisions, and confirmatory scripts for the headline tables. The design is parameter-free in the relevant sense, and the task-bootstrap intervals make the intended uncertainty unit explicit. The strict-scoring reversal is the least secure advertised result: it is a rescoring counterfactual rather than measured behavior under a strict executor, and although Section 7 acknowledges this, the abstract and conclusion should carry the same qualification. Overall, if the strict-scoring and small-sample claims are framed more narrowly, the paper makes a credible methodological contribution.

Pith's reading of the paper (a machine's summary, not the authors')

Quantizing a deployed language-model agent to 4 bits roughly halves its memory footprint, but this paper asks whether the cheaper model still bounces back as well when a tool call temporarily fails. The answer it finds is that there is no model-independent answer: on twenty deterministic tool-use tasks with five prompts, the 8-bit-versus-4-bit recovery comparison changes direction across prompts and even reverses under different evaluation choices. For Llama under one prompt, the measured difference moves from $-17.5$ points when each variant is scored on its own clean-passing tasks, to $0$ on tasks both pass cleanly, to $+28.3$ on the full pipeline, and finally to $-15.0$ when output parsing is strict. The paper's point is methodological as much as empirical: one prompt, one screened task set, and one scoring policy cannot establish a stable conclusion about quantized-agent robustness, so evaluations should compare variants on matched tasks, report full-pipeline success for deployment, and state the scoring policy.

What carries the argument. The argument is carried by a clean screen and a choice of denominator, plus one scoring distinction. Let $S(a,p)$ be the tasks a configuration at precision $a$ under prompt $p$ completes without faults: the separately screened rate averages recovery over $S(a,p)$; the matched-task rate averages over $S(a_1,p) \cap S(a_2,p)$, the intersection of tasks both precisions pass cleanly under the same prompt; and the full-pipeline rate averages over all twenty tasks with a clean-screen failure scored as zero. These three denominators answer different questions, and the demonstrations show the sign of the 8-bit-minus-4-bit difference depends on which one is used. The second switch is the executor's parsing policy: the tolerant harness extracts the first JSON object from a non-JSON turn and evaluates submitted arithmetic expressions, whereas strict scoring requires strict JSON and a literal amount, changes which tasks pass the clean screen, and reverses several comparisons. Uncertainty is carried by task-level bootstrap intervals computed over the twenty tasks, with tasks rather than injected fault sites treated as the unit of evidence.

Core claim. The central claim, stated on the paper's own terms, is that neither 8-bit nor 4-bit quantization provides a consistent recovery advantage across models and prompts. On tasks that both precisions complete cleanly under the same prompt, Llama's 8-bit-minus-4-bit difference ranges from $0$ to $+20.2$ percentage points, while Qwen's ranges from $-50.0$ under one prompt to $+35.0$ under another, with both extreme intervals excluding zero. The evaluation target itself can decide the comparison: for Llama under prompt P4, separate screening favors 4-bit by $17.5$ points, the matched-task estimate is exactly zero, and the full-pipeline estimate favors 8-bit by $28.3$ points, because the 8-bit model simply passes many more tasks cleanly. The scoring policy is a third switch: 8-bit Llama relies on the harness's tolerance of arithmetic expressions far more than 4-bit does, and rescoring the same logs strictly, without rerunning inference, turns the $+28.3$ full-pipeline advantage into $-15.0$. Qwen's large opposite-sign differences under P3 and P4 survive task matching, so task selection cannot explain everything; the paper concludes that the measured variation is real for these configurations and that evaluation design must be reported explicitly.

The premise it rests on. The sign reversals rest on matched-task cells of 4 to 10 tasks drawn from a twenty-task, two-template synthetic suite; if those tasks do not represent how real tool-use workloads fail, the directions and sizes of the reported reversals will not transfer to deployment.

If the paper is right (Pith's inferences, not the authors')

Conditional claims about recovery require the same tasks in every compared configuration; a prompt-wise clean-task intersection supplies that comparison, even where the intersection is small or tilted toward easier tasks.

Deployment comparisons should also report full-pipeline success over the whole suite, because clean competence is part of system performance and an uncertified task is counted as a failure in the gated protocol.

The executor's scoring policy is part of the evaluated system and should be documented, logged per episode so results can be rescored, and matched to the deployed executor.

Tasks, not injected fault sites, are the unit of uncertainty: multiple fault sites within one task share its structure, and deterministic reruns verify reproducibility rather than adding statistical samples.

One prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness.

The matched-task estimator generalizes beyond quantization: any comparison of model variants on a robustness-relevant behavior should be re-checked on the intersection of clean-capable tasks before the difference is attributed to the variant rather than to task selection.

Because the strict-scoring reversal was produced by rescoring logs, the deployment-relevant experiment that remains is a live rerun under an executor that actually rejects non-literal answers; feedback might change behavior, and the paper itself notes this.

The two Qwen prompts with the largest opposite-sign differences were written after early results were known, so the prompt dependence is measured but causally open; a single-factor ablation changing only the retry criterion would identify whether retry policy or unrelated wording drives it.

The reversals sit on 4 to 10 matched tasks from a two-template suite, so the no-consistent-winner conclusion should be stress-tested on structurally different task families; the paper's own forty-task generator with multi-order and distractor-tool families has only been run under P0.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on how recovery is evaluated. We compare 8-bit and 4-bit variants of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic tool-use tasks and five prompts. The 8-bit-4-bit recovery comparison changes direction across prompts and evaluation targets. On tasks that both variants complete without faults under the same prompt, the difference ranges from 0 to +20.2 percentage points for Llama and from -50.0 to +35.0 points for Qwen. Full-pipeline point estimates favor 8-bit Llama under all five prompts, whereas the Qwen comparison changes direction across prompts. The evaluation target can also reverse the result. For Llama under one prompt, scoring each variant only on its own clean-passing tasks favors 4-bit by 17.5 points; scoring the same tasks for both variants gives no difference, while scoring the full pipeline favors 8-bit by 28.3 points. Executor leniency is a third such choice. Rescoring the same logs with strict output parsing, which 8-bit Llama violates far more often than 4-bit Llama under that prompt, turns that +28.3 into -15.0 while leaving Qwen essentially unchanged. These findings show that one prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness. Evaluations should compare variants on matched tasks, report full-pipeline success for deployment decisions, state the scoring policy, and quantify uncertainty across tasks rather than injected fault sites.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-07

The authors' main claim, as the classifier picked it out 0.94

“These findings show that one prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.79 · human participants 0.03 · controlled comparison 0.65 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.59 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on how recovery is evaluated. background 0.88 0.12 0.20 hedged possibility 0.59
We compare 8-bit and 4-bit variants of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic tool-use tasks and five prompts. method 0.97 0.96 0.03 asserted 0.92
The 8-bit-4-bit recovery comparison changes direction across prompts and evaluation targets. result 1.00 0.89 0.40 asserted 1.00
On tasks that both variants complete without faults under the same prompt, the difference ranges from 0 to +20.2 percentage points for Llama and from -50.0 to +35.0 points for Qwen. result 1.00 0.94 0.08 asserted 1.00
Full-pipeline point estimates favor 8-bit Llama under all five prompts, whereas the Qwen comparison changes direction across prompts. result 1.00 0.95 0.10 asserted 1.00
The evaluation target can also reverse the result. result 0.99 0.79 0.08 asserted 0.96
For Llama under one prompt, scoring each variant only on its own clean-passing tasks favors 4-bit by 17.5 points; scoring the same tasks for both variants gives no difference, while scoring the full pipeline favors 8-bit by 28.3 points. result 1.00 0.96 0.07 asserted 1.00
Executor leniency is a third such choice. result 0.47 0.47 0.03 asserted 0.99
Rescoring the same logs with strict output parsing, which 8-bit Llama violates far more often than 4-bit Llama under that prompt, turns that +28.3 into -15.0 while leaving Qwen essentially unchanged. result 1.00 0.95 0.06 asserted 1.00
These findings show that one prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness. interpretation 0.99 0.83 0.80 asserted 1.00
Evaluations should compare variants on matched tasks, report full-pipeline success for deployment decisions, state the scoring policy, and quantify uncertainty across tasks rather than injected fault sites. future work 0.90 0.17 0.51 asserted 0.98

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-07: 1 request, 8,343 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. The measured 8-bit–4-bit difference varies across models, prompts, and evaluation targets. A. Each variant’s own clean-passing tasks. B. Tasks that both variants pass cleanly under the same prompt; the number of matched tasks is shown below each prompt. C. All twenty tasks, with a clean failure scored as zero. Error bars are 95% task-bootstrap intervals. Panel A uses a two-sample bootstrap because the task sets can differ; Panels B and C use paired bootstraps. We evaluate Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct at 8 and 4 bits on twenty determinis￾tic order-management tasks,…
Figure 2 of the paper
Figure 2. One task and the three fault conditions. Each task is a deterministic tool chain with one verified answer, and a fault is injected at one call on the required path. The transient condition (ET) is the main experiment. Displaced (IT) and permanent (EP) faults are evaluated under P0 only (Appendix A). Fault episodes run only on tasks that pass that condition’s clean screen. Models and serving. The main experiments evaluate Llama-3.1-8B-Instruct [Grattafiori et al., 2024] at Q8_0, Q4_K_M, and Q3_K_M, and Qwen2.5-7B-Instruct [Yang et al., 2024] at Q8_0 and Q4_K_M. The P0-only appendix…
Figure 3 of the paper
Figure 3. The P0 ladder: success by quantization level under clean execution and the three fault conditions, each level on its own same-prompt clean-solvable set, with episode-level 95% Wilson intervals. The Q5_K_M-to-Q4_K_M transient-recovery decrease is conditional on P0; compare Fig￾ure 1
Figure 4 of the paper
Figure 4. Exploratory Llama-only prompt–precision result. A. Task-mean P0–P1 recovery dif￾ference on each precision level’s pairwise common clean set, with task-bootstrap 95% intervals. B. Terminal actions of all 32 failed Q3_K_M/P1 episodes: 31 declare the transient fault permanent. 10

Figures are the authors', reproduced from the paper.

References (21)
  1. [3] Peijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li, Xiaowen Chu, and Bo Li. Can compressed LLMs truly act? an empirical evaluation of agentic capabilities in LLM compression. arXiv preprint arXiv:2505.19433, 2025 on PithPith's reviewarXiv
  2. [4] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ : Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023 on Pith
  3. [5] Georgi Gerganov and contributors. llama.cpp: LLM inference in C/C++ . https://github.com/ggml-org/llama.cpp, 2023 on Pith
  4. [6] Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024 on PithPith's reviewarXiv
  5. [7] Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. Flaw or artifact? rethinking prompt sensitivity in evaluating LLMs . arXiv preprint arXiv:2509.01790, 2025 on PithPith's reviewarXiv
  6. [8] Jiwon Jang, Kisu Yang, Heuiseok Lim, and Hyunwoo Park. Flat score, amplified failures: How the error budget masks damage in quantized LLM agents. arXiv preprint arXiv:2607.27275, 2026 on PithPith's reviewarXiv
  7. [9] Angelika Romanou, Mark Ibrahim, Candace Ross, Chantal Shaib, Kerem Oktar, Samuel J. Bell, Anaelia Ovalle, Jesse Dodge, Antoine Bosselut, Koustuv Sinha, and Adina Williams. Brittlebench: Quantifying LLM robustness via prompt sensitivity. arXiv preprint arXiv:2603.13285, 2026 on PithPith's reviewarXiv
  8. [13] Dongsheng Zhu, Xuchen Ma, Yucheng Shen, Xiang Li, Yukun Zhao, Shuaiqiang Wang, Lingyong Yan, and Dawei Yin. When tools fail: Benchmarking dynamic replanning and anomaly recovery in LLM agents. arXiv preprint arXiv:2606.05806, 2026 on PithPith's reviewarXiv
  9. [14] Jang, Jiwon and Yang, Kisu and Lim, Heuiseok and Park, Hyunwoo , journal=. Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized on Pith
  10. [15] Zhu, Dongsheng and Ma, Xuchen and Shen, Yucheng and Li, Xiang and Zhao, Yukun and Wang, Shuaiqiang and Yan, Lingyong and Yin, Dawei , journal=. When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in on Pith
  11. [16] Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction , author=. arXiv preprint arXiv:2608.11772 , year= on PithPith's reviewarXiv
  12. [17] Grattafiori, Aaron and others , journal=. The on Pith
  13. [18] Gerganov, Georgi and contributors , howpublished=. llama.cpp: on Pith
  14. [19] Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan , booktitle= on Pith
  15. [20] The case for 4-bit precision: k-bit Inference Scaling Laws , author=. arXiv preprint arXiv:2212.09720 , year= on PithPith's reviewarXiv
  16. [21] -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year= on PithPith's reviewarXiv
  17. [22] ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. arXiv preprint arXiv:2506.07982 , year= on PithPith's reviewarXiv
  18. [23] Romanou, Angelika and Ibrahim, Mark and Ross, Candace and Shaib, Chantal and Oktar, Kerem and Bell, Samuel J. and Ovalle, Anaelia and Dodge, Jesse and Bosselut, Antoine and Sinha, Koustuv and Williams, Adina , journal=. Brittlebench: Quantifying on Pith
  19. [24] Hua, Andong and Tang, Kenan and Gu, Chenhe and Gu, Jindong and Wong, Eric and Qin, Yao , journal=. Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating on Pith
  20. [25] Dong, Peijie and Tang, Zhenheng and Liu, Xiang and Li, Lujun and Chu, Xiaowen and Li, Bo , journal=. Can Compressed on Pith
show 1 more
  1. [26] Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year= on PithPith's reviewarXiv
Cited by 1 Pith paper

Reviewed papers on Pith that cite this one. The full citation record

  1. PROVISO: Provenance-Informed Action Selection for Secure LLM Agents cs.AI · Oct 2026

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment