Structured Feedback Improves Repair in an LLM Agent Loop

Ankit Goyal; Jaideep Ray

REVIEW 2 major objections 5 minor 11 references

Structured Feedback Improves Repair in an LLM Agent Loop

T0 review · 2 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured validator feedback that names the failure location, observed value, and admissible alternatives materially improves repair in an LLM agent loop, with most of the gain coming from the alternatives themselves.

desk verdict Robust within TextWorld; the recommendation to validators outruns the evidence once alternatives are not enumerable. read the letter →

arxiv 2607.14167 v1 pith:I36L5KLP submitted 2026-07-15 cs.SE cs.AI

Jaideep Ray , Ankit Goyal This is my paper

classification cs.SEcs.AI

keywords LLMagentsverifierfeedbackrepairloopstructuredadmissiblealternativesTextWorldvalidatordesignablationstudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the content of validator feedback—not just the fact of rejection—drives whether an LLM agent can repair a failed candidate. Under an equal four-call budget on 50 paired text-adventure games, feedback naming the failure location, the observed value, and admissible alternatives raised terminal success from 14/50 to 36/50 for one coder model and from 8/50 to 29/50 for another. Ablations show the admissible alternatives account for nearly all of the gain, while presenting the same information as a keyed JSON record instead of prose adds nothing within statistical noise. The conclusion matters because it gives agent designers a concrete rule: validators should return actionable repair values when they can enumerate them, and retry budget is best spent on new information.

What carries the argument

VeriHarness, a code-controlled agent loop where external validators (not the model) enforce acceptance, budgets, and traces. Its failure encoder maps a rejected action to a common interface: a stable label, the command index, the observed command, and up to 12 admissible commands in deterministic order. The contrast among four feedback policies—RawDiag, SameNL, LocObs, TypedFields—isolates the contribution of location, observed value, alternatives, and keyed presentation.

What would settle it

Replace the oracle's admissible-command list with a random list of valid-but-irrelevant commands; if success still rises by about 40 points, the gain is not from the content of the alternatives. Alternatively, run the same four-call loop on a real bug-repair benchmark whose validator reports only a failing test name; the paper's mechanism predicts little or no gain there.

Watch

Extended reading notes

Core claim

The paper claims that when an external validator rejects a candidate, telling the next model call exactly where the failure happened, what value was observed, and which values would have been admissible is much more effective than returning the raw validation message. In 50 paired TextWorld games with a four-call cap, this three-part feedback raised terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B and from 8/50 to 29/50 for Llama-3.1-8B. Ablations show that the admissible alternatives carry almost all of the effect: feedback with only location and observed value stays near the raw-diagnostic baseline, while rendering the same three values in natural-language prose performs within t

Load-bearing premise

The benefit presumes that a validator can produce a list of genuinely admissible alternatives, as the TextWorld oracle does; real validators often cannot enumerate such a list, so the effect may not transfer.

Editorial extensions

If this is right

Validators should return failure location, observed value, and expected or admissible alternatives when available; simply saying 'command N is invalid' is far less useful.
Under a four-call cap, adding alternatives improved terminal success by 42–44 points across two models, and the ordering held with sampled decoding.
Increasing the call budget from four to eight did not help the raw-diagnostic condition, but structured feedback continued to gain, so extra calls help mainly when each retry receives new information.
The keyed JSON representation gave no success advantage over prose with the same repair values (Qwen +2, Llama 0), so JSON has no demonstrated reasoning benefit, although it saved a few calls.
Repair feedback cannot act on failures the validator does not expose: when a visible test passed but a hidden test failed, no policy could recover, limiting the approach to failures the validator can detect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: agent-loop designers should invest in validators that can enumerate candidate fixes (e.g., type-correct alternatives, known API signatures) because retry budgets without new information appear to stall; raw diagnostics were flat from four to eight calls in the paper's 15-game subset.
The mechanism may transfer to software repair only where an enumerable set of admissible replacements exists; for bugs with open-ended fixes, the TextWorld result would not predict a similar jump, and the HumanEval scope check shows the boundary.
A testable prediction beyond the paper: replacing the oracle list with a random set of valid-but-irrelevant commands should shrink the gap, isolating whether the gain comes from the content of the alternatives or simply from receiving a command-shaped hint.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core result is real and well-supported within the TextWorld setup: giving the model the list of admissible alternatives is what moves terminal success, and JSON syntax is not the driver. The soft spot is that the paper's broader validator recommendation assumes alternatives are enumerable, which real validators often cannot provide.

What's new: prior Reflexion/Self-Refine/self-debugging work shows iterative feedback helps, but nobody has cleanly ablated location vs observed vs expected alternatives, or compared prose against a keyed record. The 50-game paired design with bootstrap intervals and exact McNemar tests is appropriate, and the ablations are convincing: SameNL beats RawDiag by 42 points for both models, TypedFields is within noise of SameNL, and LocObs sits near baseline. That is a clean story: the values matter, the keying doesn't.

The HumanEval scope check is a genuine boundary condition. On all 15 tasks the visible test passed, so no repair opportunity arose. That does not undermine the TextWorld claim, but it does mean the paper offers no evidence that this feedback works for code repair when a test fails without enumerating fixes. The authors are transparent about capping alternatives at 12 and using the environment's deterministic order, and they flag that they did not test ranked or randomized lists. What they do not test is the more basic availability assumption: can a validator produce alternatives at all? Since LocObs is near baseline, the advice to report location and observed value would not help without alternatives. That is a real external-validity gap, though not a flaw in the internal comparison.

Minor issues: the artifact is not linked or hashed, so independent verification of the traces isn't possible; one serving error was rerun, though this is disclosed. Neither affects the core result.

Recommendation: this deserves a serious referee. The internal validity is high and the external-validity question is exactly what a revision should address, either by broadening the experiments or by scoping the claim more carefully.

Referee Report

2 major / 5 minor

Summary. The paper introduces VeriHarness, a code-controlled agent loop that separates LLM candidate generation from external validation, and uses it to compare four validator-feedback policies on 50 TextWorld games with two models (Qwen2.5-Coder-14B and Llama-3.1-8B) under a four-call cap. The central finding is that feedback containing a failure location, the observed value, and admissible alternatives substantially improves terminal success (from 14/50 to 36/50 for Qwen, and from 8/50 to 29/50 for Llama), with paired bootstrap CIs and Holm-adjusted exact McNemar tests. Ablations show that the gain is almost entirely due to the admissible-alternatives field: prose and typed-JSON presentations of the same repair values perform nearly identically, while location-plus-observed feedback without alternatives remains near the raw-diagnostic baseline. Additional robustness lanes cover call budgets and sampled decoding, and a 15-task HumanEval scope check illustrates that feedback cannot help when the visible validator does not expose a failure.

Significance. If the results hold, the paper provides a concrete, controlled answer to an underspecified interface question in agent design: what should a validator return to the next LLM call. The study is well designed for internal validity: paired games, exact tests with correction, disclosed confounds, and a reproducible artifact containing 880 row-level results and 2,652 call traces. The main strength is that the evidence cleanly isolates the repair-value effect from the representation effect. The primary limitation is external validity: the benefit depends on a validator being able to enumerate admissible alternatives, a property that is true of the TextWorld oracle but often absent in real software validators. The paper is honest about many limitations, but the availability assumption itself is not tested or analyzed.

major comments (2)

[§3, §5, §4.4] The paper's practical conclusion that validators should return specific repair information rests on the availability of enumerable admissible alternatives. In §3, feedback is the oracle's list of up to the first 12 admissible commands in TextWorld's deterministic order. §4.2 shows that LocObs (location+observed, no alternatives) is near RawDiag, so the measurable benefit comes specifically from the alternatives field. The HumanEval scope check (§4.4) tests hidden failures, not the case where a validator can locate and observe but cannot enumerate alternatives; §6 admits untested list length/order/ranking but not this availability assumption. The abstract's closing claim and §5 recommendation therefore overgeneralize. Please either provide evidence on this availability condition (e.g., an analysis of how often the oracle list contains a successful replacement, or an experiment with a vali
[§4.2, §6] The RQ2 attribution — that alternatives account for most of the improvement — rests on the contrast between LocObs and TypedFields, but the paper never reports whether the truncated first-12 list actually contains a command the model adopts, nor whether the deterministic order influences the model's choice. If the correct replacement tends to appear early in TextWorld's enumeration, the effect could be partly an artifact of ordering. The disclosure in §6 that ranked/randomized lists were not tested is welcome, but a trace-level analysis (e.g., how many final accepted plans use a command from the oracle list and at what position) would directly support the claimed mechanism and would also strengthen the recommendation to validators.

minor comments (5)

[§2.3] The example feedback omits 'the rejected plan and common output instructions' that are part of the actual retry prompts. A full sample prompt for at least one policy would improve reproducibility and help readers assess prompt-level confounds.
[§4.1] The first-call policy-name confound is disclosed, and the reported first-call wins differ by only one per model. Still, a condition with a neutral policy label would allow a byte-identical pre-repair prompt and would remove any remaining doubt about the clean comparison.
[§4.4] The HumanEval scope check uses a single visible test per task and reports that all 15 first answers pass it. The selection of the 'one deterministically selected public assertion' deserves a sentence of detail; otherwise the reader cannot assess how representative the 15 tasks are.
[§4.2] The claim that TypedFields 'uses somewhat fewer calls' is presented alongside terminal success. Since call count is tied to early termination, the comparison would be more informative if restricted to solved games or accompanied by per-policy call distributions.
[Table 3] There is a typo in the header: 'Qwen2.5-Coder-14B-Instruct-A WQ' should be 'AWQ'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical ablation with no fitted parameters, derived equations, or load-bearing self-citation chain.

full rationale

The paper's central claim is an empirical comparison of feedback policies inside a fixed agent loop. There are no fitted parameters, normalization steps, or theoretical equations that could reduce the outcome to the input. The treatment feedback (location, observed value, admissible alternatives) comes from an external TextWorld oracle, and the outcome is terminal game success scored by the same environment; giving more informative feedback and observing higher success is the intended intervention, not a construction artifact. The ablations are properly controlled: RawDiag vs SameNL isolates repair values, TypedFields vs SameNL isolates the keyed format, and LocObs vs TypedFields isolates the alternatives. The HumanEval scope check is presented as a limitation of the transfer setting, not as a positive result, and the paper explicitly states the condition under which feedback can help. The self-citations in the background section are standard prior-work citations (Reflexion, Self-Refine, SWE-bench, etc.) and are not load-bearing for the present empirical result. The limitations section (§6) openly acknowledges untested ordering/ranking of admissible actions and the non-pure contrast between prose and keyed policies, but these are external-validity caveats, not circularity. No step in the paper's derivation chain reduces to its own inputs by definition or by self-citation, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rest on no fitted parameters or new scientific entities. Free parameters are design caps (12 commands, 4 calls) that bound the scope. Domain assumptions concern the representativeness of TextWorld and the oracle-generated alternatives. No circular derivation or self-citation underwrites the empirical result.

free parameters (2)

admissible_actions_cap = 12 (first 12 commands in environment order)
Invalid actions return up to 12 admissible commands; the list length and ordering are hand-chosen and untested, so the effect size is conditional on this cap.
primary_call_cap = 4 calls
The primary matrix uses a four-call cap; budget lanes (2,4,6,8) show ordering persists but magnitudes differ.

assumptions (3)

standard math Paired bootstrap and exact McNemar tests are valid for 50 paired games with planned contrasts after Holm correction.
Statistical protocol in §3 Analysis; assumes exchangeability across games and no unmodeled correlation from reusing budget-4 rows.
domain assumption TextWorld terminal win is a meaningful proxy for LLM repair quality.
The study uses 50 generated TextWorld games; the central claim is explicitly scoped to this environment, so this is an acknowledged scope assumption.
domain assumption The environment's admissible command list is the correct source of 'expected alternatives' for repair.
§3 states invalid actions return up to the first 12 admissible commands; the study assumes this oracle enumeration is representative of useful validator feedback. HumanEval results show this assumption fails when the validator cannot expose the failure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Structured Feedback Improves Repair in an LLM Agent Loop." pith.science (2026). https://pith.science/paper/I36L5KLP

@misc{pith2026260714167,
  author       = {Pith},
  title        = {Pith review of: Structured Feedback Improves Repair in an LLM Agent Loop},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I36L5KLP}},
  note         = {Machine review of arXiv:2607.14167}
}

read the original abstract

LLM agents often retry after external validation rejects a candidate, but the interface between validation and the next model call remains underspecified. We introduce VeriHarness, a code-controlled agent loop in which models generate candidates while external validators control acceptance, budgets, and traces. We use it to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games under a four-call cap, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 points). Ablations locate most of the gain in the admissible alternatives: feedback containing only the location and observed value remains near the raw diagnostic baseline. Presenting the complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. The ordering persists across the tested call budgets and one sampled-decoding setting.

Figures

Figures reproduced from arXiv: 2607.14167 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 4 linked inside Pith

[1]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. CoRRabs/2107.03374 (2021)

arXiv 2021
[2]

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. InInternational Conference on Learning Representations

2024
[3]

Marc-Alexandre Côté, Àkos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. TextWorld: A Learning Environment for Text-Based Games.CoRRabs/1806.11532 (2018)

arXiv 2018
[4]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. CoRRabs/2407.21783 (2024)

arXiv 2024
[5]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report.CoRRabs/2409.12186 (2024)

arXiv 2024
[6]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Representa- tions

2024
[7]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626. doi:10.1145/3600006.3613165

arXiv 2023
[8]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. InAdvances in Neural Information Processi...

2023

Show all 11 references

[9]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. InAdvances in Neural Information Processing Systems, Vol. 36

2023
[10]

Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InAdvances in Neural Information Processing Systems, Vol. 37

2024
[11]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations. 4

2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. CoRRabs/2107.03374 (2021)

arXiv 2021

[2] [2]

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. InInternational Conference on Learning Representations

2024

[3] [3]

Marc-Alexandre Côté, Àkos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. TextWorld: A Learning Environment for Text-Based Games.CoRRabs/1806.11532 (2018)

arXiv 2018

[4] [4]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. CoRRabs/2407.21783 (2024)

arXiv 2024

[5] [5]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report.CoRRabs/2409.12186 (2024)

arXiv 2024

[6] [6]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Representa- tions

2024

[7] [7]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626. doi:10.1145/3600006.3613165

arXiv 2023

[8] [8]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. InAdvances in Neural Information Processi...

2023