Pith.

Computation and Language · arXiv:2611.22835 v1 · submitted 17 Nov 2026 · 63 references

When Passing Checks Do Not Establish Completion: Evidence Interpretation in Coding Agents

Xueping Gao

The paper says

“Verification procedures in coding agents must preserve execution facts, interpret their scope and discharge checking obligations. We study these responsibilities through qualified repository checks, controlled evidence consumers and executable workflows.”

more of the abstract

A prospective thirteen-task factorial study finds that correct reported facts need not yield the requested finite verdict. Conversely, all 122 delivered labels in a held-out receipt-supply study are correct, while sixteen responses misreport facts and eleven others exceed authorized scope. Production repair supplies successful controls and finite-supported preservation failures. Across 118 repair episodes on 31 tasks, source interventions separate startup reachability from repair and witness production from the claimed operation. Lifetime and resource controls delimit what passing validation establishes. Generated-note comparisons yield local repair differences but can state false explanations; strong ordinary baselines, new tasks and Skill-transfer studies supply no consistent overall advantage. These results characterize evidence interpretation, applicability and fulfilled actions while limiting claims of general diagnostic superiority, Skill deterioration or deployed folding effects.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 18 Nov 2026 · read the full text (71,049 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A careful empirical study showing that passing checks in coding agents can misrepresent task completion, with detailed case evidence and explicit limitations. This line is wrong

no objections raised

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 71,049 characters. Not checked: figures, code, what the 63 references say.

What would settle it

Independently replicate the flush repair case (Django13516) with a different model or framework, using the same fixture and assertion, and verify whether the write-side flush event can satisfy the explicit-flush assertion without the explicit operation being effective. If the same pattern of witness misattribution is observed, the central claim is strengthened; if not, the generality may be limited, but the paper already acknowledges limited external validity.

from the skeptic's reading

How the review went

5 passes by deepseek-v4-flash · 18 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 5 s · 19,652 tokens read, 478 written
  3. Referee no objections raised 4 s · 19,857 tokens read, 245 written
  4. Check against the paper
  5. Editor wrote the verdict above 5 s · 20,178 tokens read, 452 written

Not run on this paper: the simulated authors' defence.

Pith checked this review's objections against the text of the paper it read.

The review in full

The review in brief, machine-written

This paper documents how production code changes in coding agents can distort validation evidence, so that passing a test suite does not guarantee the requested operation was performed. The central claim—that correct reported facts need not yield a correct verdict—is supported by specific case studies (flush forwarding, startup obstruction, witness misattribution, lifecycle dependencies) with source-level interventions and controlled reversions. The paper is unusually careful about stating its own limitations: the sample is small (31 tasks, 118 episodes), many diagnoses are post-hoc and source-informed, and external validity is explicitly bounded. The authors do not overclaim; they repeatedly note that their findings establish neither prevalence nor diagnostic superiority over ordinary testing.

The main strength is the granularity of the evidence. The flush example cleanly separates a passing assertion from the actual operation that satisfied it, and the one-pair separation argument (equal test triples, different behavioral outcomes) is a crisp logical point. The startup obstruction and lifecycle counterexamples are well-documented. The paper also includes null results and contrary controls, which is rare and commendable.

The soft spots are real but acknowledged. The sample is small and not random; the tasks were selected for mechanism relevance, not representativeness. The tracing methods rely on adapter invocation records and source reversions, which the paper notes are necessary but not sufficient. The generated-note comparisons are exploratory and do not show a consistent advantage. The experimental archive is currently private, which limits reproducibility but is stated as a planned release.

This paper is for researchers working on coding-agent evaluation, test-based verification, and program repair. It does not offer a new method or a benchmark, but it provides a useful conceptual framework and concrete counterexamples that should inform how the community interprets passing checks. It deserves a serious referee: the claims are well-scoped, the evidence is detailed, and the limitations are transparent. I would accept it for peer review with the expectation that the authors follow through on the public archive.

The strongest objection

The paper's central claim is that passing checks do not establish task completion because production code changes can distort validation evidence. This is supported by detailed case studies with explicit source-level interventions, tracing, and controls. The weakest assumption identified by the reader—that execution tracing and source-level interventions correctly attribute evidence to operations—is acknowledged by the paper itself (e.g., 'Entry means adapter invocation, not exhaustive coverage,' 'Preservation is necessary, not sufficient'). The paper does not claim general prevalence or diagnostic superiority; it explicitly limits external validity. The evidence presented is internally consistent and the reasoning is careful. I find no load-bearing concern that would undermine the central argument. The paper's claims are appropriately scoped and supported by the presented evidence.

The reader's weakest assumption (tracing accuracy and task representativeness) is acknowledged and mitigated by the paper's own limitations and multi-method approach. The paper does not overclaim; it explicitly states that the analyses are post-hoc, source-informed, and not estimates of prevalence. The central claim—that passing checks do not establish completion because production code can distort evidence—is well-supported by the case studies with concrete interventions (e.g., startup substitution, component reversions, lifecycle diagnostics). The paper also provides counterexamples (e.g., Devstral repairs that do pass the explicit flush check) that strengthen the argument by showing the distinction is not universal but depends on the specific production code path. No internal inconsistency or unsupported leap is evident. The private experimental archive is a reproducibility gap but does not undermine the qualitative findings as reported. Therefore, the verdict should remain ACCEPT (UNCHANGED).

The referee's report

If the findings hold, the paper makes a valuable contribution by systematically demonstrating that verification procedures in coding agents must account for production code dependencies. It provides concrete examples and a framework for distinguishing between different types of evidence. However, the study is limited in scale (31 tasks, 118 episodes), many diagnoses are post-hoc and source-informed, and the paper explicitly acknowledges that it does not establish prevalence or general diagnostic superiority. The significance is moderate: it raises important caveats for the field but does not provide a new method or large-scale empirical results.

Pith's reading of the paper (a machine's summary, not the authors')

This paper investigates whether a coding agent's passing of validation checks actually establishes that the agent completed the requested repair operation. Through a series of controlled experiments on 31 tasks with two model families, the authors demonstrate that production code edits can change which operation supplies the evidence that checks observe. A patch can pass all official checks while the intended operation remains ineffective, because the positive witness comes from a different source (e.g., writes rather than an explicit flush). The paper introduces a framework for distinguishing facts, verdicts, and authorized completion scope, and shows through source-level interventions that startup reachability, witness production, and evidence consumption are separable concerns. The central claim is that passing checks is insufficient evidence of task completion; one must examine which operation produced the validation witness.

What carries the argument. The central mechanism is the distinction between a validation check's outcome and the operation that produced the evidence for that outcome. The paper formalizes this with a finite evidence item (h,c,p,o,w): candidate source digest, selected check, observed phase, relevant operation, and recorded witness. The key analytical tool is the operation-isolating consumer: a controlled intervention that resets observations or changes context to test whether a specific operation (rather than a side effect) produced the witness. For example, resetting flush observations before the explicit flush call separates write-origin events from explicit-flush events. This allows the paper to show that equal check outcomes can arise from different operations, and that a passing check does not establish which operation was responsible.

Core claim. The paper establishes that a coding agent's passing of validation checks does not necessarily mean the agent completed the requested operation. The central discovery is that production code participates in validating its own edits, so a change to production code can alter which operation supplies the evidence that checks observe. For example, on Django13516, both Qwen conditions produce a patch that adds an unconditional flush to a write method, leaving the explicit flush ineffective. Both patches pass all official checks and both actors claim a supported fix, yet the explicit flush operation was never performed. The positive witness came from writes, not from the explicit flush. This demonstrates that a passing check can be satisfied by an unintended operation, and that the validation outcome alone does not identify which operation fulfilled the obligation.

The premise it rests on. The paper assumes that its execution tracing and source-level interventions correctly identify the causal relationship between production code changes and validation outcomes, specifically that adapter invocation records, call-stack distinctions, and controlled source reversions accurately attribute evidence to operations.

If the paper is right (Pith's inferences, not the authors')

If the paper is correct, then coding agent evaluations that rely solely on passing checks as evidence of task completion are systematically unreliable: a patch can pass all checks while the requested operation remains ineffective.

The paper's framework implies that validation procedures must trace evidence to its producing operation, not just record whether a check passed or failed. This would require changes to how coding agent benchmarks are designed and how completion is adjudicated.

The finding that production code participates in validating its own edits means that startup reachability and witness production must be analyzed separately from the repair itself. A patch that obstructs startup can cause all checks to fail without indicating anything about the repair's correctness.

The paper's counterexample on flush forwarding shows that a named boolean assertion (e.g., 'flush was called') can be satisfied by an unintended operation (writes) while the explicit operation adds nothing. This implies that check design must account for which operation can produce the observed evidence.

The paper's distinction between facts, verdicts, and authorized completion scope implies that even correct finite judgments can coexist with incorrect facts or excessive completion claims. Evaluation pipelines must separately verify factual accuracy and scope compliance, not just verdict correctness.

The paper's framework suggests that many existing coding agent benchmarks may overestimate repair success because they do not verify which operation produced the validation evidence. A systematic audit of prior evaluations using operation-isolating consumers could reveal a significant rate of 'passing but not completing' cases.

The paper's finding that production code changes can distort validation evidence implies that the same vulnerability applies to any system where the code under test also participates in test execution. This includes unit tests, integration tests, and end-to-end evaluations in software engineering, not just coding agents.

The paper's method of source-level interventions (reversions, startup substitution, operation-isolating consumers) could be automated into a diagnostic tool that, given a candidate patch and its checks, determines whether the checks are satisfied by the intended operation or by a side effect. Such a tool would improve the reliability of automated repair evaluation.

The paper's results suggest that the distinction between 'plausible' and 'correct' patches in program repair research (Qi et al., Smith et al., Ahmed et al.) may need a third category: 'passing but not completing,' where checks pass but the requested operation is not performed. This would refine the overfitting analysis by identifying cases where the patch overfits to the check's evidence rather t

Check this review

What was read

The passes were given all 71,049 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): 238a932da279fa462258a481a3d0ac81907f7c8e4efb1240c05e00b941fc1c42

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.22835

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0080 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0019 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0019 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Editor deepseek-v4-flash · medium effort · $0.0019 · instructions of the Editor, fingerprint e7211f6a3d27…
  5. Circularity check deepseek-v4-flash · medium effort · $0.0018 · instructions of the Circularity check, fingerprint e9946e39d67b…
  6. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0022 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.018 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.505d180dcaef

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 18 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): 86ec19f500219aadf468c0a2c88181c7180ff4bd398097619540f63a5b5e1594

curl -s 'https://pith.science/api/v1/verdicts/2611.22835?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Verification procedures in coding agents must preserve execution facts, interpret their scope and discharge checking obligations. We study these responsibilities through qualified repository checks, controlled evidence consumers and executable workflows. A prospective thirteen-task factorial study finds that correct reported facts need not yield the requested finite verdict. Conversely, all 122 delivered labels in a held-out receipt-supply study are correct, while sixteen responses misreport facts and eleven others exceed authorized scope. Production repair supplies successful controls and finite-supported preservation failures. Across 118 repair episodes on 31 tasks, source interventions separate startup reachability from repair and witness production from the claimed operation. Lifetime and resource controls delimit what passing validation establishes. Generated-note comparisons yield local repair differences but can state false explanations; strong ordinary baselines, new tasks and Skill-transfer studies supply no consistent overall advantage. These results characterize evidence interpretation, applicability and fulfilled actions while limiting claims of general diagnostic superiority, Skill deterioration or deployed folding effects.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-18

The authors' main claim, as the classifier picked it out 0.47

“Lifetime and resource controls delimit what passing validation establishes.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.93 · human participants 0.12 · controlled comparison 0.90 · randomized assignment 0.04 · states quantitative result 0.98 · compares with prior results 0.86 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Verification procedures in coding agents must preserve execution facts, interpret their scope and discharge checking obligations. background 1.00 0.07 0.12 asserted 0.98
We study these responsibilities through qualified repository checks, controlled evidence consumers and executable workflows. method 0.76 0.77 0.04 asserted 0.96
A prospective thirteen-task factorial study finds that correct reported facts need not yield the requested finite verdict. result 0.92 0.85 0.42 asserted 0.88
Conversely, all 122 delivered labels in a held-out receipt-supply study are correct, while sixteen responses misreport facts and eleven others exceed authorized scope. result 1.00 0.84 0.18 asserted 1.00
Production repair supplies successful controls and finite-supported preservation failures. result 0.88 0.58 0.12 asserted 0.95
Across 118 repair episodes on 31 tasks, source interventions separate startup reachability from repair and witness production from the claimed operation. result 0.73 0.82 0.09 asserted 0.99
Lifetime and resource controls delimit what passing validation establishes. interpretation 0.91 0.41 0.30 asserted 1.00
Generated-note comparisons yield local repair differences but can state false explanations; strong ordinary baselines, new tasks and Skill-transfer studies supply no consistent overall advantage. result 0.98 0.82 0.22 asserted 0.94
These results characterize evidence interpretation, applicability and fulfilled actions while limiting claims of general diagnostic superiority, Skill deterioration or deployed folding effects. interpretation 0.71 0.78 0.63 asserted 0.98

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-18: 1 request, 6,700 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (2)
Figure 1 of the paper
Figure 1. Production code participates in both the behavior under test and the production of validation evidence. A candidate can obstruct startup or change
Figure 2 of the paper
Figure 2. Receipt-supply output partition: 16 checks, nine task blocks and two

Figures are the authors', reproduced from the paper.

References (63)
  1. [1] Catch2 contributors. Assert Info reset need to also reset result disposition to normal to handle uncaught exception correctly. PR 2723,2023. https: //github.com/catchorg/Catch2/pull/2723 on Pith
  2. [2] Guava contributors. LoadingCache stops loading forever after race condition. Issue 2971. https://github.com/google/guava/issues/2971 on Pith
  3. [3] Superpowers contributors. Fix(systematic-debugging): find- polluter.sh find -path never matches (fixes2008). PR 2011, 2026. https://github.com/obra/superpowers/pull/2011. Third source snapshot: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b on Pith
  4. [4] zstd contributors. Add ZSTD_d_stableOutBuffer + fix single-pass mode for empty frames. Pull request2094, merged 29April2020. https://github. com/facebook/zstd/pull/2094 on Pith
  5. [5] X. Zhou, J. Liu, L. Li, R. A. Rossi and X. Hu. Counterfactual Trace Auditing of LLM Agent Skills. arXiv:2605.11946v2, 2026 on PithPith's reviewarXiv
  6. [6] G. Dong, Y . Gao, L. Li, T. Xu, Y . Hua and F. Yang. Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents. arXiv:2608.11888v1, 2026 on PithPith's reviewarXiv
  7. [7] J. Wang, Y . Zhao, T. Li, H. Chen, J. Chen, K. Peng and S. He. Agent Skill Evolution: How Revisions Affect Coding Agents. arXiv:2610.04832v1, 2026 on PithPith's reviewarXiv
  8. [8] D. Sarkis. Building agents: When to fold subagents into your main agent loop. Anthropic, 22 October 2026. https://claude.com/resources/articles/ building-agents-when-to-fold-subagents-into-your-main-agent-loop on Pith
  9. [9] Y . Kim et al. Towards a Science of Scaling Agent Systems. arXiv:2512.08296v3, 2026 on PithPith's reviewarXiv
  10. [10] H. Liu, H. Yang, T. Jiang, B. Tang, F. Xiong, Y . Luo and Z. Li. SkillsV ote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution. EMNLP, 2026. https://aclanthology.org/ 2026.emnlp-main.281/ on Pith
  11. [11] Z. Zhang, S. Lu, H. Qian, D. He and Z. Liu. AgentFactory: A Self- Evolving Framework Through Executable Subagent Accumulation and Reuse. arXiv:2603.18000v1, 2026 on PitharXiv
  12. [12] E. Su et al. BRIDGE: Selective Boundary Repair with Dependency Graph Evidence in Multi-Agent Systems. arXiv:2610.17805v1, 2026 on PitharXiv
  13. [13] W. Li, G. Li, D. Li, K. Maeda, T. Ogawa and M. Haseyama. Bench- marking Task Takeover Across Agent Handoffs. arXiv:2610.36523v1, 2026 on PithPith's reviewarXiv
  14. [14] X. Gao. Executable Code Knowledge: Code as a Native, Validation- Carrying Knowledge Representation for AI Coding Agents. arXiv:2608.16295v1, 2026 on PithPith's reviewarXiv
  15. [15] X. Gao. Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents. arXiv:2608.08793v1, 2026 on PithPith's reviewarXiv
  16. [16] X. Gao. Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose. arXiv:2606.18051v1, 2026 on PithPith's reviewarXiv
  17. [17] C. Jia, T. Zhao, Z. Xiao, W. Zhang and M. Zhou. VeriSkill: A Self-Evolution Framework for Program Verification Skills. arXiv:2607.27733v1, 2026 on PithPith's reviewarXiv
  18. [18] SWE-bench. Experiments: public predictions, execution logs and tra- jectories. Snapshot and content digests recorded in the supplementary artifact. https://github.com/SWE-bench/experiments on Pith
  19. [19] H. Ning et al. Executable but Unfaithful: Mock Object, Stub, and As- sertion Generation by LLMs and Coding Agents. arXiv:2610.14629v1, 2026 on PithPith's reviewarXiv
  20. [20] Z. Chen et al. Rethinking the Value of Agent-Generated Tests for LLM- Based Software Engineering Agents. arXiv:2602.07900v1, 2026 on PithPith's reviewarXiv
show 43 more
  1. [21] B. Mohammadi and L. Bindschaedler. Whose Oracle Is It? The Author- ity Problem in Deployed Agentic Program Repair. APR, 2026, pp.37–40. doi:10.1145/3843776.3844673 on Pithdoi:10.1145/3843776.3844673
  2. [22] Y . Guo, Y . Liu, J. M. Zhang, Y . Ma, Y . Lou and Z. Chen. SWE- Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests. arXiv:2607.00990v1, 2026. https://arxiv.org/abs/2607.00990 on PithPith's reviewarXiv
  3. [23] Y . Ma, B. Kereopa-Yorke and B. Schultz. Building to the Test: Coding Agents Deliver What You Check, Not What You Requested. arXiv:2606.28430v1, 2026. https://arxiv.org/abs/2606.28430 on PithPith's reviewarXiv
  4. [24] B. Wang et al. The Verification Horizon: No Silver Bullet for Coding Agent Rewards. arXiv:2606.26300v1, 2026. https://arxiv.org/abs/2606. 26300 on PithPith's reviewarXiv
  5. [25] X. Xu and W. Wu. Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? arXiv:2607.28871v1, 2026 on PithPith's reviewarXiv
  6. [26] J. Yin et al. VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents. arXiv:2610.28679v1, 2026. https://arxiv.org/ abs/2610.28679v1 on PithPith's reviewarXiv
  7. [27] Z. Zhong et al. SpecGuardBench: Can Coding Agents Construct Re- liable Validation Harnesses from Natural Language Specifications? arXiv:2610.16870v1, 2026 on PithPith's reviewarXiv
  8. [28] N. Mündler, M. N. Müller, J. He and M. Vechev. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. NeurIPS, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/ hash/94f093b41fc2666376fb1f667fe282f3-Abstract-Conference.html on Pith
  9. [29] Z. Qi, F. Long, S. Achour and M. Rinard. An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems. ISSTA, 2015. doi:10.1145/2771783.2771791 on Pithdoi:10.1145/2771783.2771791
  10. [30] E. K. Smith, E. T. Barr, C. Le Goues and Y . Brun. Is the Cure Worse Than the Disease? Overfitting in Automated Program Repair. ESEC/FSE, 2015. doi:10.1145/2786805.2786825 on Pithdoi:10.1145/2786805.2786825
  11. [31] T. Ahmed, J. Ganhotra, A. Shinnar and M. Hirzel. Investigat- ing Test Overfitting on SWE-bench. arXiv:2511.16858v3, 2026. doi:10.1145/3803437.3805574 on PithPith's reviewarXiv
  12. [32] C. Le Goues et al. The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. IEEE Transactions on Software Engineering, 2015. doi:10.1109/TSE.2015.2454513 on Pithdoi:10.1109/tse.2015.2454513
  13. [33] H. Li et al. CodeTracer: Towards Traceable Agent States. arXiv:2604.11641v1, 2026 on PithPith's reviewarXiv
  14. [34] S. Zheng et al. BenchShield: Formal Model-Backed Instrumenta- tion for Reward Integrity in LLM-Agent Evaluation Infrastructure. arXiv:2609.11028v1, 2026 on PithPith's reviewarXiv
  15. [35] Python Software Foundation. os.stat_result and timestamp resolu- tion, Python 3.10 documentation. https://docs.python.org/3.10/library/os. html#os.stat_result on Pith
  16. [36] Python Software Foundation. io—Core tools for working with streams, Python 3.10 documentation. https://docs.python.org/3.10/library/io.html on Pith
  17. [37] A. Hora and R. Robbes. Are Coding Agents Generating Over-Mocked Tests? An Empirical Study. MSR, 2026. arXiv:2602.00409v1. https:// arxiv.org/abs/2602.00409 on PitharXiv
  18. [38] P. Sahoo, G. Mittal, X. Li, S. Ma, B. Steenhoek, P. Lin, and Y . Hu. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evalu- ation. arXiv:2605.12925v4, 2026. https://arxiv.org/abs/2605.12925v4 on PithPith's reviewarXiv
  19. [39] Django contributors. Fixed1891/11707: Prevented duplicates with limit_choices_to on multi-value relations. Pull request13315, merged October27,2020. https://github.com/django/django/pull/13315 on Pith
  20. [40] R. Abou Assi, C. Trad, M. Maalouf and W. Masri. Coincidental Correctness in the Defects4J Benchmark. arXiv:1808.09233v4, 2019 on PithPith's reviewarXiv
  21. [41] W. Masri and R. Abou Assi. Prevalence of coincidental correctness and mitigation of its impact on fault localization. ACM Transactions on Software Engineering and Methodology, 23(1), Article8, 2014. doi:10.1145/2559932 on Pithdoi:10.1145/2559932
  22. [42] S. Bansal, F. Jiao, Y . Zhou, A. Xu, S. Joty and S. Yavuz. VibePass: Can Vibe Coders Really Pass the Vibe Check? arXiv:2603.15921v1, 2026 on PitharXiv
  23. [43] hardwork-xu. Agent Code Evidence (ContextProof): versioned source evidence for coding agents. Public software repository, accessed Octo- ber30,2026. https://github.com/hardwork-xu/agent-code-evidence on Pith
  24. [44] Django Software Foundation, “QuerySet API reference: order_by(),” Django3.1 documentation, https://docs.djangoproject.com/en/3.1/ref/ models/querysets/, accessed 30Oct.2026 on Pith
  25. [45] M. N. Rahman, C. Canbaz and S. B. Hossain. Covered Is Not Checked: Which Branch Outcomes Would Your Tests Notice? arXiv:2610.33708v1, 2026. https://arxiv.org/abs/2610.33708 on PithPith's reviewarXiv
  26. [46] P. J. Eby. PEP3333: Python Web Server Gateway Interface v1.0.1. Python Software Foundation. https://peps.python.org/pep-3333/ on Pith
  27. [47] Neon contributors. fix: stop leaking BackgroundPurges. PR8650, merged8August2024. https://github.com/neondatabase/neon/pull/8650 on Pith
  28. [48] Tokio contributors. Stronger close() semantics for TaskTracker. Is- sue6759, 2024. https://github.com/tokio-rs/tokio/issues/6759 on Pith
  29. [49] AnyIO contributors. MemoryObjectStream can drop items when the receiving end is cancelled. Issue728; fixed in4.4.0. https://github.com/ agronholm/anyio/issues/728 on Pith
  30. [50] OpenAI. Codex. 2026. https://openai.com/codex/ on Pith
  31. [52] K. Egashira, N. Mündler-Sasahara, M. N. Müller and M. Vechev. TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications. arXiv:2610.13065v1, 2026. https://arxiv.org/abs/ 2610.13065 on PithPith's reviewarXiv
  32. [53] Y . Lei et al. SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills. arXiv:2605.24117v1, 2026. https:// arxiv.org/abs/2605.24117 on PithPith's reviewarXiv
  33. [54] G. Nie, F. Zhu, S. Kai, X. Han, T. Zhong and M. Yuan. SkillEvoReg: Regularizing Agent Skill Evolution Against Overfitting. arXiv:2609.30861v1, 2026. https://arxiv.org/abs/2609.30861 on PithPith's reviewarXiv
  34. [55] B. Danglot, O. L. Vera-Pérez, B. Baudry and M. Monperrus. Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. arXiv:1811.08330, 2018. https://arxiv.org/abs/1811.08330 on PithPith's reviewarXiv
  35. [56] M. Haque, P. Babkin, F. Farmahinifarahani and M. Veloso. Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs. KnowledgeNLP, 2025, pp.160–179. doi:10.18653/v1/2025.knowledgenlp-1.17 on Pithdoi:10.18653/v1/2025.knowledgenlp-1.17
  36. [57] J. Wu, T. Wu, M. Zhang, Y . Dong and B. Shen. From Guessing to Seeing: Enhancing LLM-Based Program Repair via Trace-Guided Multi-strategy Debate. ASE, 2026, pp.2050–2062. doi:10.1145/3832783.3837458 on Pithdoi:10.1145/3832783.3837458
  37. [58] nlohmann/json contributors. Copy values before inserting an initializer list into an array. Merged PR5693, reference commit e444a662762420681487f369aa5fc8565de8366f. https: //github.com/nlohmann/json/pull/5693 on Pith
  38. [59] More-itertools contributors. Peekable: avoid caching entire iterable on negative slice with omitted stop. PR1322, merged commit69dbc3392928afe1cc892a387bf0e721522f21c2. https://github.com/more-itertools/more-itertools/pull/1322 on Pith
  39. [60] Boltons contributors. October PR sweep follow-up: #482, #483, #605, #607, #608. PR 610, merged com- mit d31d821d56c8f7756af9c30a864077f2e5855321. https: //github.com/mahmoud/boltons/pull/610 on Pith
  40. [61] Bidict contributors. Fix _dedup no-op detection for non- reflexive-equality keys/values (e.g. NaN). PR 377, merged commit 7ca069d231f61675ac7b6929225bba5a9b342f7d. https://github.com/jab/bidict/pull/377 on Pith
  41. [62] Google ADK contributors. Parallel calls to the same streaming tool leave an orphan task after live teardown. Issue7058. https://github.com/google/ adk-python/issues/7058 on Pith
  42. [63] Google ADK contributors. Track concurrent streaming tool calls. Un- merged PR7059; base d57c84f13baf53cfd910c2155449f8c4254e01c5, proposed reference 8fee86a8bfca641bd51e970d7b56ca3055cb14f1. https://github.com/google/adk-python/pull/7059 on Pith
  43. [2026] https://arxiv.org/abs/2608.08950 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment