LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation

Brent Paulovicks; Ngoc Phuoc An Vo; Vadim Sheinin

REVIEW 4 major objections 6 minor 14 references

LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two new LLM-as-a-Judge metrics validate and refine Bash code without running it, matching execution-based evaluation up to 8% better than the ICE-Score baseline and boosting refinement accuracy by up to 24%.

desk verdict Two sensible new LLM-as-a-Judge metrics for Bash validation, but the empirical support is bar-chart-thin at n=50; the idea is worth refereeing, not yet worth citing. read the letter →

arxiv 2506.11237 v1 pith:KYLULEUH submitted 2025-06-12 cs.SE cs.CL

Ngoc Phuoc An Vo , Brent Paulovicks , Vadim Sheinin This is my paper

classification cs.SEcs.CL

keywords LLM-as-a-JudgeBashcodegenerationreference-lessevaluationvalidationrefinementbidirectionalfunctionalitymatchinglogicrepresentationITautomation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that an LLM judge can replace execution-based testing for Bash code in IT automation, by checking whether a generated script's described functionality and its extracted logic fully cover the functionalities the user asked for. Two metrics are proposed: bidirectional functionality matching, which compares required functionalities extracted from the problem against a generated functionality description of the code in both directions, and logic representation, which translates the code into pseudocode-like logic and tests whether that logic covers the required functions. Using execution-based pass/fail as ground truth on three Bash test suites, the authors report that both metrics agree with execution results and beat the ICE-Score baseline by up to 8% in validation accuracy. They then feed the metrics' judgments and feedback into reflection code agents and report up to 24% relative improvement in execution-based accuracy over the non-agentic baseline. A sympathetic reader would take the paper as evidence that reference-less LLM judging can be a cheap online substitute for test execution and a useful repair signal.

What carries the argument

The load-bearing object is the comparison step: after the LLM produces a functionality description of the code snippet and a list of required functionalities from the problem, the judge checks whether the code functionality satisfies and covers all required functionalities, and vice versa. For logic representation, the code is first translated into pseudocode-like logic representation, and the same coverage check is applied against the required functionality list. These two extracted representations are the proxies that carry the entire argument; if either extraction is wrong, the judgment is wrong.

What would settle it

Take a set of Bash scripts that pass execution-based tests but implement the task through an unusual strategy that an LLM would not naturally list among required functionalities; if the bidirectional probe rejects a large fraction of these passing scripts, the metric is not tracking execution correctness. A complementary test is to give the judge a problem whose required functionality list is deliberately corrupted by adding an unrequested function and check whether the verdict flips even though the code still executes correctly.

Watch

Extended reading notes

Core claim

The central claim is that semantic validation of natural-language-to-Bash code does not require reference solutions or a runtime environment: an LLM can extract the required functionalities from the problem, build a comprehensive functionality description (or logic representation) of the candidate code, and decide correctness by checking coverage in both directions. The bidirectional check matters because it catches both missing functionality (code does less than asked) and extraneous logic (code does things not asked for), while the logic representation bypasses surface syntax. The paper treats execution-based evaluation as ground truth and frames its contribution as better alignment with that ground truth than a strong LLM-judge baseline, plus the ability to return detailed feedback for correction.

Load-bearing premise

The whole pipeline assumes the LLM can reliably extract accurate required functionalities from the problem text and produce faithful functionality descriptions or logic representations of the code, so the comparison is actually testing correctness rather than paraphrasing ability.

Editorial extensions

If this is right

In an incident-remediation pipeline, a reference-less judge that agrees with execution-based evaluation can be run before executing any remediation script, saving time and avoiding the side effects of running untrusted Bash.
Because the metrics emit feedback rather than only a score, they can be wired directly into iterative code-generation agents as a critic signal.
If the reported validation gains hold, the metrics can be used to select among candidate generator models without maintaining expensive per-command test harnesses.
The refinement results suggest that the feedback component, not just the binary judgment, is the main added value over scoring-only judges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the judge could act as a fast filter in a two-stage pipeline that reserves execution-based validation for the ambiguous cases the LLM judge flags.
A natural stress test is to see whether the feedback remains useful when the generator and the judge are the same model, since self-generated correctness judgments can be circular.
The reported pattern that larger models improve more under feedback hints that judge quality, not the refinement loop itself, may be the binding constraint; this could be tested by freezing the generator and swapping judge models.
The bidirectional matching idea is essentially a semantic equivalence check and could be applied to other natural-language-to-code settings, provided the target language can be described functionally.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate, well-scoped attempt to make reference-less code validation work for Bash in IT automation, and the two metrics are a real step beyond ICE-Score in design. But the paper's central quantitative claim – that both metrics beat ICE-Score in agreement with execution-based ground truth – is not backed by the evidence as presented. The validation results are bar charts only, over 50 items per suite, with no numbers, no error bars, no significance tests. An 8-point gap at n=50 is roughly 4 items and about 1.4 standard errors; 'significantly outperformed' is doing a lot of work. The right test here is McNemar's paired exact test; it's not there.

The design itself is sound and worth credit. Bidirectional functionality matching checks both whether the code's functionality description covers the problem's required functionalities and vice versa, which is a sensible fix to the unidirectional comparisons in ICE-Score and CodeSift. The logic representation approach, translating code to pseudocode and then checking coverage, is a nice way to dodge syntax-level over-penalization. Using execution-based evaluation as ground truth is the correct yardstick, and the authors are upfront (in the limitations) that the whole pipeline depends on the LLM's extraction, translation, and comparison abilities. That honesty is genuinely good.

The weaker places beyond the missing stats: the judge LLM for Task 1 is never specified, only the generator (Granite-34b-code-instr). That's a basic reproducibility gap. One generator model only, so generalization is untested. The refinement loop, which is where the practical payoff is, relies on manual code extraction after reflection because the LLMs often return code mixed with reasoning and hallucinations. That turns 'automatic' into 'human-assisted' and could inflate the reported improvements in Table 1. No code or prompts are released, so I can't re-run anything.

What is not a problem: the circularity worry. The metrics are evaluated against execution-based ground truth defined independently of the metrics, so the evaluation setup isn't circular. The same-family judge/generator concern is minor unless the judge turns out to be Granite too.

Bottom line: this is a promising, honest paper from a team that knows the domain, but the evidence as written is not enough to support the headline. With a few tables, a McNemar test, judge model specification, and released prompts, it would be a solid contribution. It deserves a serious referee, but not a cite from me next year as it stands.

Referee Report

4 major / 6 minor

Summary. The paper proposes two reference-less LLM-as-a-Judge metrics for validating natural-language-to-Bash code generation: Bidirectional Functionality Matching and Logic Representation, plus a combined variant. Each metric extracts required functionalities from a problem and compares them with a generated functionality description or a pseudocode logic representation of a candidate Bash snippet. Using execution-based evaluation on the NL2Bash-EAbench benchmark as ground truth, the authors compare their metrics against ICE-Score on 50 test cases per suite and report that their metrics outperform the baseline. They then build self-reflection agents and reflection-with-dedicated-evaluator agents that use the proposed metrics' judgments and feedback for code refinement, reporting execution-accuracy improvements up to 24%. The paper's Limitations section candidly acknowledges that the approach depends on LLMs' ability to extract functionalities, translate to logic representations, reason about coverage, and follow instructions, and that manual extraction of code from LLM outputs is sometimes required.

Significance. If the empirical claims were established, the contribution would be practically useful: the two metrics are reference-less, provide not only binary judgments but also feedback for repair, and are evaluated against an external execution-based ground truth rather than in a circular manner. The use of a publicly available execution-based benchmark and the authors' transparent discussion of limitations are strengths. The central significance claim, however, is currently weakened by the lack of statistical support and by underspecified experimental conditions: the paper does not provide numeric validation results, confidence intervals, or paired significance tests, and it does not state which LLM served as the judge in Task 1. The abstract's wording that the proposed methods 'significantly outperformed' the baseline and achieved 'significant improvement' for refinement is therefore not yet warranted by the reported evidence.

major comments (4)

[§5, Figures 2–4] The claim that the proposed metrics 'significantly outperformed' ICE-Score is not statistically supported: the only evidence is bar charts for Accuracy, Precision, Recall, and F1 over 50 test cases per suite, with no numeric values, error bars, or significance tests. With n=50, an 8-percentage-point margin corresponds to four items, and near 80% accuracy the standard error is roughly 5.7 points, so the difference is within sampling noise. Please report confusion-matrix counts for each suite and metric and test paired differences with McNemar's exact test (or an equivalent), and state whether any multiple-comparison correction was applied.
[§4.1] Task 1 does not specify the LLM used as the judge. The paper names Granite-34b-code-instr only for code generation, but the two proposed metrics and the ICE-Score baseline are all implemented with an unspecified judge LLM; without this information the results are not reproducible and the comparison may be tied to one judge model. I also note that Task 1 uses only a single generator model, so the general framing about selecting the best model is broader than the experiments support.
[§4, 'Processing for Automatic Code Extraction'] For Task 2, the paper states that after reflection the authors 'have to manually examine and extract the code' from mixed output. Because the executed code is manually selected, the reported execution accuracy for the reflection agents may include human assistance, which weakens the automatic-refinement claim. Please report how often manual extraction was required, provide extraction results on raw model outputs, or automate the extraction step.
[§5, Table 1] The refinement improvements in Table 1 are presented without variance estimates, confidence intervals, or significance tests, and with only one run per condition. Several observed differences are small (for example, 72% vs. 68% or 58% vs. 54%), and the 24% improvement is a single cell across 3 models × 3 suites with no multiple-comparison correction, so the 'significant improvement' language in the abstract is not warranted by the reported evidence.

minor comments (6)

[References] In the reference list, 'V o et al.' contains a spurious space; the name should be 'Vo et al.'
[§2 Related Works] The description of ICE-Score says it works on 'four programming languages (Java, Python, C, C++, and JavaScript),' but five languages are listed; please correct the count or the list.
[§2 Related Works] The acronym 'G-EV AL' has inconsistent spacing and should be written as 'G-EVAL' throughout.
[§4 Experiments] The paper does not state temperature or sampling settings for any of the LLMs used, which makes the comparisons hard to reproduce.
[§5 Evaluations] The appendix examples in Figures 5 and 6 are helpful, but they show only one case per metric; adding a failure case where the metric disagrees with execution would clarify the metrics' behavior.
[§4.1] ShellCheck is used for syntactic pre-processing, but the paper does not specify what happens when ShellCheck rejects a snippet (for example, whether such snippets are excluded from evaluation or marked as failing).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the metrics are evaluated against external execution-based ground truth and are not defined in terms of it.

full rationale

The paper's two proposed metrics are algorithmic LLM-as-a-Judge pipelines: required functionalities are extracted from the problem, a functionality description or logic representation is generated from the code snippet, and the two are compared. The metrics do not ingest execution results, ground-truth labels, or the ICE-Score baseline. The validation experiments then compare the metrics' binary judgments against execution-based evaluation on the same items, and the refinement experiments measure final accuracy using the same execution-based ground truth after iterative refinement. This is an external, empirical evaluation design, not a definitional reduction. The dataset cited (NL2Bash-EAbench, Vo et al. 2024) is authored by the same researchers, but it is an independently constructed execution-based benchmark with validators, so it functions as real evidence rather than a circular premise. Similarly, the comparison to ICE-Score and CodeSift is positioning against prior work, not a load-bearing self-citation. No equation-level reduction, fitted-input-renamed-as-prediction, or uniqueness-imported-from-authors pattern is present. The remaining concerns—small sample size, missing significance tests, and potential bias from using LLMs as both generator and judge—are empirical validity issues, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces methods (metric algorithms, an agent architecture), not new physical or mathematical entities. It relies on three domain assumptions, all acknowledged or implicit. No numeric free parameters are fitted; the main unspecified choices are the judge model and prompt details, which are not quantitative parameters.

assumptions (3)

domain assumption LLMs can correctly extract required functionalities from a natural language problem statement.
This is the core step of both metrics (Algorithm 1 Step 1, Algorithm 2 Step 1) and is explicitly listed as a limitation in the paper.
domain assumption LLMs can generate comprehensive functionality descriptions of a code snippet and translate Bash code into a semantics-preserving logic representation.
The judge compares code functionality or logic representation against the problem requirements; if this translation is incorrect, the judgment is meaningless. Acknowledged in the Limitations section.
domain assumption Execution-based evaluation on the NL2Bash-EAbench test suites is a reliable ground truth for code correctness.
The paper uses execution-based results as the reference standard for computing accuracy and agreement, but the design of these tests is only described in a prior paper and their reliability is taken for granted.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation." pith.science (2026). https://pith.science/paper/KYLULEUH

@misc{pith2026250611237,
  author       = {Pith},
  title        = {Pith review of: LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYLULEUH}},
  note         = {Machine review of arXiv:2506.11237}
}

read the original abstract

In an effort to automatically evaluate and select the best model and improve code quality for automatic incident remediation in IT Automation, it is crucial to verify if the generated code for remediation action is syntactically and semantically correct and whether it can be executed correctly as intended. There are three approaches: 1) conventional methods use surface form similarity metrics (token match, exact match, etc.) which have numerous limitations, 2) execution-based evaluation focuses more on code functionality based on pass/fail judgments for given test-cases, and 3) LLM-as-a-Judge employs LLMs for automated evaluation to judge if it is a correct answer for a given problem based on pre-defined metrics. In this work, we focused on enhancing LLM-as-a-Judge using bidirectional functionality matching and logic representation for reference-less automatic validation and refinement for Bash code generation to select the best model for automatic incident remediation in IT Automation. We used execution-based evaluation as ground-truth to evaluate our LLM-as-a-Judge metrics. Results show high accuracy and agreement with execution-based evaluation (and up to 8% over baseline). Finally, we built Reflection code agents to utilize judgments and feedback from our evaluation metrics which achieved significant improvement (up to 24% increase in accuracy) for automatic code refinement.

Figures

Figures reproduced from arXiv: 2506.11237 by the authors.

**Figure 2.** Comparison among LLM-as-a-Judge Metrics for Bash-1 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Comparison among LLM-as-a-Judge Metrics for Bash-2 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Comparison among LLM-as-a-Judge Metrics for Bash-3 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Example of Bidirectional Functionality Matching Validation [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Example of Logic Representation Validation [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

[1]

Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. https://doi.org/10.1109/CLOUD62652.2024.00052 CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation . In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404--410, Los Alamitos, C...

arXiv 2024
[2]

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

2005
[3]

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

arXiv 2023
[4]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[5]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

arXiv 2023
[6]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

arXiv 2024
[7]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[8]

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297

arXiv 2020

Show all 14 references

[9]

Ngoc Phuoc An Vo, Brent Paulovicks, and Vadim Sheinin. 2024. https://arxiv.org/abs/2405.06807 Execution-based evaluation of natural language to bash and powershell for incident remediation . Preprint, arXiv:2405.06807

2024 arXiv
[10]

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826--23854

2023
[11]

Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint arXiv:2302.05527

2023 arXiv
[12]

Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2232--2242

2024
[13]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[14]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. https://doi.org/10.1109/CLOUD62652.2024.00052 CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation . In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404--410, Los Alamitos, C...

arXiv 2024

[2] [2]

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

2005

[3] [3]

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

arXiv 2023

[4] [4]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004

[5] [5]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

arXiv 2023

[6] [6]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

arXiv 2024

[7] [7]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002

[8] [8]

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297

arXiv 2020