{"id":"56d071a2-d4c6-4b19-ad2e-ab29221d22d1","arxiv_id":"2610.07781","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Whether 8-bit or 4-bit quantized agents recover better from tool failures reverses across prompts, task-selection targets, and scoring policies, so single-configuration evaluations of quantized-agent robustness are not stable.","lead":"The measured difference between 8-bit and 4-bit AI agents in recovering from temporary tool failures flips sign depending on the prompt, which tasks are scored, and how strictly the output is parsed. The paper argues that no single evaluation design can support a stable claim that one precision is more robust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Strict-scoring reversal is computed by rescoring tolerant-executor logs, not by running a strict executor; Llama/P4's headline sign flip could disappear if behavior adapts when expressions are rejected.","rationale":"Choosing the strict-executor issue as the load-bearing concern: the central negative claim is independent of strict scoring, but the paper's abstract and conclusions elevate executor leniency to one of three design choices that can reverse results, and the evidence for it is derived entirely from rescoring tolerant-policy logs. This is not a hidden flaw—Section 7 states it—but it is an unresolved acceptance condition because the paper's deployment recommendation ('use the deployed executor's policy') is exactly the setting where model adaptation to a strict executor matters. The reader's weakest_assumption lists the same rescoring issue alongside small-sample and multiplicity concerns; I agree with the conditions but prioritize them differently, so the reader's CONDITIONAL verdict does not need to move. The small matched-task cells and the lack of a multiplicity correction are real but disclosed, and they do not threaten the negative claim; wider intervals would only weaken positive attributions and strengthen 'no consistent advantage.' The strict-executor rerun is a single, cheap, well-defined check on one model/prompt cell and would settle whether the strict-policy reversal is a measured evaluation-design effect or a rescoring artifact.","tokens_in":13965,"tokens_out":12703,"duration_ms":81684,"concrete_test":"Rerun Llama/P4 under a genuinely strict executor: same twenty tasks, same prompt, same temperature-0 decoding and fault-injection protocol, but with a harness that rejects any non-JSON turn and any submit_answer that is not a literal numeric amount. Recompute the clean screen under this policy before injecting faults, then recompute Table 2's strict columns for Llama/P4 (separate, matched, full pipeline) and compare with the rescored values (-42.9, -33.3 with n=6, and -15.0). If the actual strict runs reproduce negative differences of similar magnitude, the reversal is a measured evaluation-design effect; if the sign flips, vanishes, or changes materially, the strict-policy result should be reported only as a rescoring sensitivity, not as an executor-leniency reversal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The tolerant-scoring core of the paper—no consistent 8-bit vs 4-bit recovery advantage across models and prompts—is well supported, and Section 7 is candid about the narrow task suite and small intersections. The least secure load-bearing step is the strict-scoring reversal presented in the abstract and Section 5.3 (b7). Table 2's strict columns rescore logs collected under the tolerant executor, including the clean screen; the strict clean-task sets and matched-task cells are therefore counterfactual task populations, not measurements under a strict policy. The paper acknowledges in b163 that a model told that its expression was rejected might behave differently. What remains unresolved is the stronger use of the result: 'Executor leniency is a third such choice' and the conclusion that strict scoring can reverse a quantization comparison are framed as an evaluation-design finding. An actual strict executor changes the observations the model sees, so 8-bit Llama's expression-submission rate and clean-passing set could change; the +28.3 to -15.0 and 0 to -33.3 reversals may not describe a strictly deployed system. This does not overturn the tolerant-scoring conclusion, but it is the one load-bearing claim whose current evidence is a rescoring counterfactual.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The manuscript evaluates tool-failure recovery of 8-bit and 4-bit GGUF variants of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on twenty deterministic order-management tasks, using five prompt variants, three evaluation targets (separately screened task sets, prompt-matched tasks, full-pipeline success), and two scoring policies. It reports that the 8-bit-versus-4-bit recovery comparison is not consistent across models and prompts, that separate clean screening and the choice of evaluation target can change the sign of the comparison, and that rescoring the same logs under strict output parsing reverses the Llama/P4 comparison. The paper recommends reporting matched-task recovery for conditional statements, full-pipeline success for deployment, stating the scoring policy, and using tasks rather than fault sites as the unit of uncertainty.","tokens_in":14198,"tokens_out":9461,"duration_ms":64286,"significance":"The main tolerant-scoring conclusion is well supported and is a useful negative result for the quantized-agent evaluation literature: it documents, with released per-episode logs and pinned decoding settings, that a precision-recovery comparison can be an artifact of screening, evaluation-target, and scoring choices. The paper's reproducibility practices are a real strength: fixed seed, temperature-0 decoding, pinned llama.cpp commit, logged model-file revisions, and confirmatory scripts for the headline tables. The design is parameter-free in the relevant sense, and the task-bootstrap intervals make the intended uncertainty unit explicit. The strict-scoring reversal is the least secure advertised result: it is a rescoring counterfactual rather than measured behavior under a strict executor, and although Section 7 acknowledges this, the abstract and conclusion should carry the same qualification. Overall, if the strict-scoring and small-sample claims are framed more narrowly, the paper makes a credible methodological contribution.","major_comments":[{"comment":"The strict-scoring reversal is computed by rescoring logs that were generated under the tolerant executor, and the clean screen is rescored under the same counterfactual policy; no model was rerun. The paper acknowledges this in Section 7 (b163), but the abstract's claim that \"executor leniency is a third such choice\" and the conclusion's statement that \"strict scoring can reverse it\" present the result as an evaluation-design finding without the same caveat. Because a model that is told its expression was rejected may change its behavior in a way that rescoring cannot capture, the reversal may not describe a strictly deployed system. I recommend rephrasing the headline claim to \"rescoring the released logs under strict parsing reverses the measured comparison\" and stating explicitly that the paper makes no behavioral claim about an actual strict executor; if a deployment claim under a strict executor is intended, that experiment would be needed.","section":"§5.3, Table 2, abstract (b7)","authors_acknowledge":"The strict-scoring analysis rescored existing logs without rerunning inference; a model told that its expression was rejected might behave differently.","authors_acknowledge_p":0.95},{"comment":"The strongest direction-changing evidence rests on small matched-task cells: Qwen/P3 and Qwen/P4 use ten matched tasks, and the strict-scoring Llama/P4 cells use four to six tasks. No multiplicity correction is applied across the roughly sixty reported prompt-target-policy contrasts, and prompts P3 and P4 were written after early results were known. The per-comparison 95% intervals therefore overstate the strength of the across-prompt pattern. I am not asking for a different protocol, but the paper should add a sensitivity statement or adjusted intervals, or explicitly say that the direction changes are point estimates for these twenty tasks and that the significant opposite-signed intervals have not been corrected for selection across prompts and evaluation targets.","section":"§5.1, §5.3, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The matched-task counts n are printed directly under panel B, and the caption says the number of matched tasks is shown below each prompt; make the placement visually unambiguous so readers do not confuse the n values with panel A's separate-set sample sizes.","section":"Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Qwen/P4 value 0.425 has three decimal places while all other entries in the table have two; align the formatting for consistency.","section":"Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement that deterministic reruns do not provide independent statistical samples is useful; consider adding one sentence noting that the 20,000 bootstrap resamples also do not add information beyond the task-level sample, since this is a common source of overconfident reading of bootstrap intervals.","section":"§4, Uncertainty paragraph","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nWorth a look if you do quantized-agent evaluations. The actual new thing here is a decomposition of an 8-bit versus 4-bit tool-failure recovery comparison across five prompts, three evaluation targets, and two scoring policies, on two models. The main negative claim—neither precision has a consistent recovery advantage—is well supported as scoped. The Llama/P4 result is a nice demonstration: separate clean screening favors 4-bit by 17.5 points, matched tasks give zero, and the full pipeline favors 8-bit by 28.3 points. That trio cleanly shows why one evaluation target is not enough.\n\nCredit where earned. The paper is unusually transparent: pinned llama.cpp commit, fixed decoding, per-episode logs, task-level bootstrap intervals, and clear definitions of separately screened, matched-task, and full-pipeline rates. Section 7 openly discloses the post-hoc P3/P4 prompts, the non-preregistered Qwen extension, deterministic reruns not counting as samples, the small all-prompt intersections, and the fact that strict scoring rescored logs without rerunning inference. No fitted parameters, no derivations, no circularity burden. If the supplementary ZIP ships what it claims, this is reproducible work.\n\nSoft spots, in proportion. First, the twenty tasks come from two closely related templates in one synthetic environment, and several headline reversals rest on matched-task cells of 4 to 10 tasks. The paper acknowledges this, but it still means the reversals are demonstrations rather than stable effect-size estimates for real workloads. Second, no multiplicity correction across roughly sixty contrasts; I would not treat every interval that excludes zero as a confirmed effect. Third, and most important: the strict-scoring reversal in the abstract is the weakest load-bearing claim. Rescoring tolerant-executor logs is a scoring counterfactual, not a measurement under a strict executor. A model told its expression was rejected may change its behavior. Section 7 admits this, yet the paper still frames executor leniency as a third evaluation-design reversal. I would like that framing softened or tested with an actual strict executor before it carries so much weight. This does not damage the central tolerant-scoring conclusion, only the sharpest version of one secondary claim.\n\nI could not open the supplementary ZIP from here, so the reproduction-script claims rest on the paper's word, though the in-paper artifact descriptions are specific enough to believe.\n\nThis paper is for people building or evaluating quantized LLM agents, especially deployment studies. It is a methodological caution, not a benchmark result, and it deserves a serious referee. I would send it out, with requests for a strict-executor rerun, a multiplicity-aware summary, and a clearer label on the rescoring result. I would also cite it in evaluation-methodology footnotes.","headline":"Honest, well-scoped empirical warning that 8-bit versus 4-bit agent comparisons flip across prompts, task sets, and scoring policies; the central negative claim holds, but the strict-scoring reversal is a rescoring counterfactual and should be labeled as such.","tokens_in":14822,"tokens_out":2341,"duration_ms":16957},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper shows that whether an 8-bit or 4-bit quantized agent recovers better from tool failures has no stable answer: the comparison flips sign across prompts, across which tasks are scored, and across how strictly outputs are parsed.","keywords":["quantization","tool-failure recovery","LLM agents","evaluation design","prompt sensitivity","robustness evaluation","matched-task comparison","scoring policy"],"falsifier":"Apply the paper's own screening, fault-injection, and matched-task protocol to a larger, structurally different task set, for example the forty-task generator with multi-order and distractor-tool families that the paper built but used only in its appendix, and check whether the 8-bit-versus-4-bit comparison still changes sign across prompts. A second check targets the scoring reversal: rerun Llama under P4 with an executor that actually rejects arithmetic expressions at runtime, since a model told its answer was rejected may behave differently from logs that were merely rescored.","tokens_in":15329,"feed_emoji":"⚖️","tokens_out":12874,"duration_ms":69916,"temperature":0.7,"pith_summary":"Quantizing a deployed language-model agent to 4 bits roughly halves its memory footprint, but this paper asks whether the cheaper model still bounces back as well when a tool call temporarily fails. The answer it finds is that there is no model-independent answer: on twenty deterministic tool-use tasks with five prompts, the 8-bit-versus-4-bit recovery comparison changes direction across prompts and even reverses under different evaluation choices. For Llama under one prompt, the measured difference moves from $-17.5$ points when each variant is scored on its own clean-passing tasks, to $0$ on tasks both pass cleanly, to $+28.3$ on the full pipeline, and finally to $-15.0$ when output parsing is strict. The paper's point is methodological as much as empirical: one prompt, one screened task set, and one scoring policy cannot establish a stable conclusion about quantized-agent robustness, so evaluations should compare variants on matched tasks, report full-pipeline success for deployment, and state the scoring policy.","feed_headline":"One scoring rule flips a 28-point recovery gap to -15","feed_subtitle":"Changing which tasks you score or how strictly you parse output reverses which quantized model looks more robust.","key_machinery":"The argument is carried by a clean screen and a choice of denominator, plus one scoring distinction. Let $S(a,p)$ be the tasks a configuration at precision $a$ under prompt $p$ completes without faults: the separately screened rate averages recovery over $S(a,p)$; the matched-task rate averages over $S(a_1,p) \\cap S(a_2,p)$, the intersection of tasks both precisions pass cleanly under the same prompt; and the full-pipeline rate averages over all twenty tasks with a clean-screen failure scored as zero. These three denominators answer different questions, and the demonstrations show the sign of the 8-bit-minus-4-bit difference depends on which one is used. The second switch is the executor's parsing policy: the tolerant harness extracts the first JSON object from a non-JSON turn and evaluates submitted arithmetic expressions, whereas strict scoring requires strict JSON and a literal amount, changes which tasks pass the clean screen, and reverses several comparisons. Uncertainty is carried by task-level bootstrap intervals computed over the twenty tasks, with tasks rather than injected fault sites treated as the unit of evidence.","core_discovery":"The central claim, stated on the paper's own terms, is that neither 8-bit nor 4-bit quantization provides a consistent recovery advantage across models and prompts. On tasks that both precisions complete cleanly under the same prompt, Llama's 8-bit-minus-4-bit difference ranges from $0$ to $+20.2$ percentage points, while Qwen's ranges from $-50.0$ under one prompt to $+35.0$ under another, with both extreme intervals excluding zero. The evaluation target itself can decide the comparison: for Llama under prompt P4, separate screening favors 4-bit by $17.5$ points, the matched-task estimate is exactly zero, and the full-pipeline estimate favors 8-bit by $28.3$ points, because the 8-bit model simply passes many more tasks cleanly. The scoring policy is a third switch: 8-bit Llama relies on the harness's tolerance of arithmetic expressions far more than 4-bit does, and rescoring the same logs strictly, without rerunning inference, turns the $+28.3$ full-pipeline advantage into $-15.0$. Qwen's large opposite-sign differences under P3 and P4 survive task matching, so task selection cannot explain everything; the paper concludes that the measured variation is real for these configurations and that evaluation design must be reported explicitly.","pith_inferences":["The matched-task estimator generalizes beyond quantization: any comparison of model variants on a robustness-relevant behavior should be re-checked on the intersection of clean-capable tasks before the difference is attributed to the variant rather than to task selection.","Because the strict-scoring reversal was produced by rescoring logs, the deployment-relevant experiment that remains is a live rerun under an executor that actually rejects non-literal answers; feedback might change behavior, and the paper itself notes this.","The two Qwen prompts with the largest opposite-sign differences were written after early results were known, so the prompt dependence is measured but causally open; a single-factor ablation changing only the retry criterion would identify whether retry policy or unrelated wording drives it.","The reversals sit on 4 to 10 matched tasks from a two-template suite, so the no-consistent-winner conclusion should be stress-tested on structurally different task families; the paper's own forty-task generator with multi-order and distractor-tool families has only been run under P0."],"forward_implications":["Conditional claims about recovery require the same tasks in every compared configuration; a prompt-wise clean-task intersection supplies that comparison, even where the intersection is small or tilted toward easier tasks.","Deployment comparisons should also report full-pipeline success over the whole suite, because clean competence is part of system performance and an uncertified task is counted as a failure in the gated protocol.","The executor's scoring policy is part of the evaluated system and should be documented, logged per episode so results can be rescored, and matched to the deployed executor.","Tasks, not injected fault sites, are the unit of uncertainty: multiple fault sites within one task share its structure, and deterministic reruns verify reproducibility rather than adding statistical samples.","One prompt, one screened task set, and one scoring policy do not establish a stable conclusion about quantized-agent robustness."],"supporting_citations":[{"why":"Supplies the fault taxonomy (transient, displaced, permanent), the recovery framing, and the gated protocol of injecting failures only into tasks the system already solves cleanly, which the experiments adopt.","marker":"[Zhu et al., 2026]"},{"why":"Provides the serving stack and the GGUF quantization presets (Q8_0, Q4_K_M, Q3_K_M) that define what '8-bit' and '4-bit' mean in this study.","marker":"[Gerganov and contributors, 2023]"},{"why":"Is the source of one of the two evaluated models, Llama-3.1-8B-Instruct.","marker":"[Grattafiori et al., 2024]"},{"why":"Is the source of the second evaluated model, Qwen2.5-7B-Instruct.","marker":"[Yang et al., 2024]"},{"why":"Prior evaluation of agentic capabilities under compression; the paper contrasts its single screened success rate with matched-task and full-pipeline targets.","marker":"[Dong et al., 2025]"},{"why":"Shows aggregate scores can hide quantization-induced failures, motivating the need to look beyond one success rate.","marker":"[Jang et al., 2026]"},{"why":"Post-training quantization method that motivates the cost savings of low-bit checkpoints.","marker":"[Frantar et al., 2023]"},{"why":"Makes the case that 4-bit precision is attractive, framing the deployment question the paper tests.","marker":"[Dettmers and Zettlemoyer, 2022]"},{"why":"A recovery harness the paper argues would need the same matched-task analysis when used to compare model variants.","marker":"[Wang et al., 2026]"}],"fun_headline_variants":["Quantization recovery results flip across prompts and scoring rules","Scoring choices flip whether 8-bit or 4-bit looks more robust","Quantized-agent recovery gap swings from +28 to -15 by scoring","Which quantized model recovers better depends on evaluation design","Tool-failure recovery: 8-bit vs 4-bit changes with how you score"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The sign reversals rest on matched-task cells of 4 to 10 tasks drawn from a twenty-task, two-template synthetic suite; if those tasks do not represent how real tool-use workloads fail, the directions and sizes of the reported reversals will not transfer to deployment.","fun_headline_variants_meta":{"raw":{"variants":["Quantization recovery results flip across prompts and scoring rules","Scoring choices flip whether 8-bit or 4-bit looks more robust","Quantized-agent recovery gap swings from +28 to -15 by scoring","Which quantized model recovers better depends on evaluation design","Tool-failure recovery: 8-bit vs 4-bit changes with how you score"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"cost":0.000209356,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000209356,"upstream_inference_prompt_cost":0.000113876,"upstream_inference_completions_cost":9.548e-05},"total_tokens":1564,"prompt_tokens":1223,"completion_tokens":341,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":512,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":261}},"tokens_in":1223,"tokens_out":341,"cost_source":"openrouter_usage","duration_ms":3748,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":261,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-08T00:11:25.910402+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Apply the paper's own screening, fault-injection, and matched-task protocol to a larger, structurally different task set, for example the forty-task generator with multi-order and distractor-tool families that the paper built but used only in its appendix, and check whether the 8-bit-versus-4-bit comparison still changes sign across prompts. A second check targets the scoring reversal: rerun Llama under P4 with an executor that actually rejects arithmetic expressions at runtime, since a model told its answer was rejected may behave differently from logs that were merely rescored.","supporting_citations":[{"cited_title":"When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents","cited_arxiv_id":"2606.05806","evidence_quote":"Supplies the fault taxonomy (transient, displaced, permanent), the recovery framing, and the gated protocol of injecting failures only into tasks the system already solves cleanly, which the experiments adopt."},{"cited_title":"llama.cpp: LLM inference in C/C++","cited_arxiv_id":null,"evidence_quote":"Provides the serving stack and the GGUF quantization presets (Q8_0, Q4_K_M, Q3_K_M) that define what '8-bit' and '4-bit' mean in this study."},{"cited_title":"The Llama 3 Herd of Models","cited_arxiv_id":"2407.21783","evidence_quote":"Is the source of one of the two evaluated models, Llama-3.1-8B-Instruct."},{"cited_title":"Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents","cited_arxiv_id":"2607.27275","evidence_quote":"Shows aggregate scores can hide quantization-induced failures, motivating the need to look beyond one success rate."},{"cited_title":"GPTQ : Accurate post-training quantization for generative pre-trained transformers","cited_arxiv_id":null,"evidence_quote":"Post-training quantization method that motivates the cost savings of low-bit checkpoints."},{"cited_title":"When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents","cited_arxiv_id":"2606.05806","evidence_quote":"A recovery harness the paper argues would need the same matched-task analysis when used to compare model variants."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}