evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments

Fatou Ndiaye Mbodji

REVIEW 4 major objections 4 minor 15 references

evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims LLM judges can evaluate smart contract comments, and prompt design determines how well they match human experts.

desk verdict A modular LLM-as-judge tool for smart contract comments, but the central claim about human alignment rests on hidden annotations and no statistics. read the letter →

arxiv 2507.20774 v1 pith:36ZKSXT4 submitted 2025-07-28 cs.AI

Fatou Ndiaye Mbodji This is my paper

classification cs.AI

keywords LLM-as-a-JudgesmartcontractcommentscodecommentgenerationautomaticevaluationpromptengineeringSoliditySCCLLMCCGIR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models can serve as practical judges of automatically generated smart contract comments, and that the choice of prompt changes how well an LLM judge agrees with human experts. It introduces evalSmarT, a modular framework that combines roughly 40 LLMs with 10 prompting strategies, yielding over 400 evaluator configurations. The authors report that, among the 40 configurations they tested, GPT-4 paired with a language-aware, question-answering prompt (P6) agreed best with human expert annotations on accuracy, completeness, clarity, and helpfulness. That configuration was then used to compare two comment generators, SCCLLM and CCGIR, and the framework's rankings aligned with human judgment. If these results hold, the framework offers a scalable and semantically rich way to evaluate and select generated comments without relying only on surface metrics such as BLEU and ROUGE or on costly human review.

What carries the argument

The load-bearing object is the evaluator tuple ⟨M, P⟩, a model-prompt pair, together with a prompt design matrix of ten strategies. The matrix varies domain knowledge (blockchain-specific logic), language features (Solidity constructs, modifiers, events), and evaluation framing (baseline vs. QA-based reasoning), so P6 is the language-aware plus QA variant. The mechanism is the LLM-as-a-Judge protocol: an LLM receives a code-comment pair and returns structured scores from 0 to 100 for accuracy, completeness, clarity, and helpfulness, where helpfulness is expressed as which stakeholder audiences would find the comment useful.

What would settle it

Take a held-out set of smart contract code-comment pairs with independent expert scores from at least two annotators, and rerun the 40 configurations; if a different model-prompt pair beats GPT-4 plus P6, or if the inter-annotator agreement is low enough that no configuration can align with both experts, the claim that P6 is best and that the framework's output aligns with human judgment would fail.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLM-based evaluation is viable for smart contract comments, and that prompt design is the deciding factor in alignment with human judgment. The specific finding is that, among the 40 configurations tested, the combination of GPT-4 with prompt P6, a strategy that injects Solidity and language awareness and frames the task as a question-answering exercise, produced the closest agreement with human expert annotations across all four measured dimensions. Using that evaluator, the framework found that SCCLLM generates substantially more accurate, complete, and clear comments than CCGIR, especially on contracts that differ from CCGIR's training distribution, and the generated audience-specific helpfulness tags separated technical from non-technical utility.

Load-bearing premise

The results depend on the unshown human expert annotations being reliable, unbiased, and representative enough that picking GPT-4 with P6 on them generalizes to the full comparison of SCCLLM and CCGIR.

Editorial extensions

If this is right

Comment-generation tools can be benchmarked at scale by running many model-prompt configurations and choosing the one that best matches human judgment, without a dedicated human panel per run.
Developers can use the framework to rank several candidate comments for the same function and pick the most informative one, with audience-specific helpfulness tags.
Prompt design becomes a first-class experimental variable: a change in framing, such as adding QA reasoning, can change which model aligns best with human judgment.
Evaluation can be reproduced and extended: new prompts and new LLMs can be added to the tuple space while keeping the same scoring dimensions.
The distinction between technical and non-technical helpfulness suggests that comment generators should be evaluated and tuned per audience, not by a single overall score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The choice of GPT-4 plus P6 as the default evaluator rests on a set of human annotations that the paper does not show; a natural extension would be to publish that set and measure inter-annotator agreement so the selection can be independently reproduced.
If LLM judges are used to rank comments produced by LLMs, there is a risk of model-specific bias; a testable extension would compare the framework's rankings against a blinded, larger human panel across several generator models.
The audience-helpfulness dimension could be turned into an automatic flag for comments that will not be understood by non-technical readers, a feature the paper does not develop.
The large gap between SCCLLM and CCGIR on out-of-distribution contracts suggests the same evaluator could serve as a generalization probe for future smart contract summarizers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

The paper describes evalSmarT, a modular LLM-as-a-judge framework for scoring automatically generated smart contract comments. The engineering is sensible and the dashboard of 40 models × 10 prompt strategies is a nice resource. But the headline claim — that prompt design significantly impacts alignment with human judgment — is asserted, not demonstrated. The paper never shows the human annotations, the correlation, or the significance test that would make that claim credible.

What is genuinely new is the application of LLM-as-a-judge to the smart contract domain, with prompt variants that inject Solidity/blockchain knowledge and QA framing. That is a reasonable extension of Wu et al. The demonstration on SCCLLM and CCGIR is a useful illustration of how the tool behaves, and the concrete scores in Tables III and IV are there for anyone to see. If the code and data links work, the tool itself may be reproducible.

The soft spot is not minor; it is load-bearing. Section V.A states that GPT-4 with prompt P6 'achieved the best alignment with human expert annotations' across four dimensions, but the annotations are never released, no guidelines are given, no inter-annotator agreement is reported, and no sample size or validation split is stated. The chosen evaluator is therefore a fitted parameter on unseen data. Using that evaluator to then compare SCCLLM and CCGIR is fine as a demonstration, but the later claim that the findings 'align with human judgment' is circular: the same hidden annotations were used both to pick the evaluator and to assert its validity. In addition, the abstract's causal statement about prompt design has no statistical support — no correlations, no error bars, no baseline comparison. The CCGIR numbers (accuracy 10, completeness 6) are striking, but without confidence intervals they could be noise from a small sample.

This is a tool paper wearing an empirical paper's clothes. For someone building a smart contract comment evaluator, the tool could be useful. For the scientific claim, the missing human study makes it unfalsifiable as written. I would not cite it for the claim, but I would not mind seeing it in a workshop demo.

Recommendation: send it back for major revision with a clear demand for the human annotation data, statistical analysis, and an explicit validation/evaluation split. That is the only way the central claim can be judged. It deserves a serious referee if the authors are willing to supply that; as it stands, the framework is interesting but the science is not there.

Referee Report

4 major / 4 minor

Summary. evalSmarT is a modular LLM-as-a-judge framework for evaluating generated smart contract comments. It combines approximately 40 LLMs with 10 prompting strategies to yield around 400 evaluator configurations, scoring comments on accuracy, completeness, clarity, and audience-specific helpfulness. The paper demonstrates the tool by selecting GPT-4 with prompt P6 (language-aware + QA framing) as the default evaluator on the basis of "best alignment with human expert annotations," and then uses this evaluator to compare comments generated by SCCLLM and CCGIR on real-world Etherscan contracts. The abstract claims that prompt design significantly impacts alignment with human judgment and that LLM-based evaluation offers a scalable, semantically rich alternative to existing methods.

Significance. If the central empirical claim were properly supported, evalSmarT would address a real gap: smart contract comment evaluation currently relies on surface-level metrics such as BLEU and ROUGE or on costly human judgment, and a reproducible, domain-aware LLM-based evaluator would be valuable to the community. The modular architecture, the four evaluation dimensions including audience-specific helpfulness, and the promised open-source code and data are clear strengths. However, as submitted, the empirical core is asserted rather than demonstrated: no human annotation dataset is provided, no inter-annotator agreement is reported, no statistical comparison across configurations is given, and no independent validation of the selected evaluator is performed. The paper itself states in the conclusion that this is a targeted demonstration and that a more comprehensive evaluation campaign is future work, which partially tempers the claims, but the abstract and Section V claim more than the presented evidence supports.

major comments (4)

[V.A] The claim that GPT-4 with prompt P6 achieved the best alignment with human expert annotations is the load-bearing result of the paper, but the human annotations are never described, released, or analyzed. No sample size, number of annotators, annotation guidelines, inter-annotator agreement, or validation split is reported. As written, the selection of the default evaluator is unfalsifiable, and the abstract's statement that prompt design significantly impacts alignment with human judgment cannot be checked by any reader.
[V.C] The claim that evalSmarT's findings "align with human judgment" is circular relative to the evidence presented: the same hidden human annotations are used to select the default evaluator in V.A and then invoked as validation of the results produced by that evaluator in Tables III and IV. No held-out human evaluation or independent ground truth is presented, so the agreement claim is not supported.
[V.B and Tables III-IV] The illustrative comparison between SCCLLM and CCGIR rests on an unspecified dataset described only as "real-world smart contract functions from Etherscan." No dataset size, sampling procedure, or contract selection criteria are given, and there is no baseline comparison against BLEU/ROUGE or direct human scores. Consequently, the strong conclusion in V.C that CCGIR degrades on out-of-distribution contracts is unsupported, since no distribution analysis of the test contracts or of CCGIR's training data is provided.
[Abstract and V.A] The causal statement that "prompt design significantly impacts alignment with human judgment" is not backed by any statistical evidence. The paper reports no correlation coefficients, no significance tests, and no comparison across the 40 configurations beyond the bare assertion that one configuration was best. The reader cannot determine whether the reported advantage of P6 is within noise or is systematic.

minor comments (4)

[V.B] In the paragraph following Table IV, the sentence about CCGIR's audience-specific helpfulness ends mid-way with "with only 40"; the numerical values appear truncated and should be completed.
[Abstract, III, IV.A] The number of evaluator configurations is inconsistently stated as "over 400" in the abstract, "Around 400" in Section III, and "up to 400" in Section IV.A; please reconcile these numbers.
[Resources] The anonymous code/data link "https://anonymous.4open.science/r/SC code summarization-4653" contains a space and is not usable as printed; the repository should also include the prompt templates and the human annotation data if the central claims are to be independently verified.
[Table I and Section II] The statement that 0% of existing methods employ LLM-based evaluation should clarify that this refers to LLMs as evaluators, not as generators, since the paper lists recent LLM-based comment generation methods in the same table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the default evaluator selection is a validation step rather than a circular derivation, though missing human annotations is a reproducibility concern.

full rationale

This paper is an empirical systems/tool paper with no mathematical derivation chain whose conclusions could reduce to their own inputs by construction. The selection of GPT-4 + P6 as the default evaluator in Section V.A is explicitly stated to be based on 'the best alignment with human expert annotations,' which is a standard validation step rather than a circular definition of the quantity being predicted. Applying that selected evaluator in Section V.B to compare SCCLLM and CCGIR is a typical fit-then-apply workflow; there is no equation or parameter in the paper that is defined in terms of the SCCLLM/CCGIR comparison it supports. The summary claim that the findings 'align with human judgment' is not supported by an independent human evaluation of the SCCLLM/CCGIR outputs, but that is a reproducibility and validity weakness, not a circularity that can be exhibited as a reduction within the paper. The only apparent self-citation, CCGIR [2], is used as a benchmark target, not as the justification for evalSmarT's validity or for the evaluator-selection result. No load-bearing uniqueness theorem, ansatz, or fitted parameter is smuggled in via self-citation, and no known result is merely renamed. Accordingly, no specific circular step can be quoted, and the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central evaluation claims rest on unstated assumptions about the quality of human annotations and the representativeness of the demonstration dataset. The framework itself has no mathematical axioms, but these domain assumptions are load-bearing for the empirical conclusions. The only fitted parameter is the default evaluator configuration, which was chosen based on the unseen human annotations.

free parameters (1)

Default evaluator configuration (model and prompt) = GPT-4 with prompt P6
Selected in Section V.A as the configuration with 'best alignment with human expert annotations' across the four evaluated dimensions. The underlying human data are not provided, so this is a hand-fitted choice.

assumptions (4)

domain assumption Human expert annotations are a reliable gold standard for smart contract comment quality.
Used implicitly throughout Section V to select the default evaluator and to claim findings align with human judgment; no inter-annotator agreement or annotation guideline is reported.
domain assumption The evaluation dimensions (accuracy, completeness, clarity, helpfulness) are meaningful and the overall score is a simple mean of three dimensions.
The overall score in Table III is the arithmetic mean of accuracy, completeness, and clarity, a choice not justified in the paper.
domain assumption LLMs can produce meaningful cardinal scores (0-100) for comment quality.
The framework relies on LLM outputs as numerical evaluations, an unvalidated assumption for this domain.
domain assumption The set of smart contract functions collected from Etherscan is representative of smart contract code in general.
Used to support claims about generalization of SCCLLM and CCGIR in Section V.C; no dataset statistics are provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments." pith.science (2026). https://pith.science/paper/36ZKSXT4

@misc{pith2026250720774,
  author       = {Pith},
  title        = {Pith review of: evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36ZKSXT4}},
  note         = {Machine review of arXiv:2507.20774}
}

read the original abstract

Smart contract comment generation has gained traction as a means to improve code comprehension and maintainability in blockchain systems. However, evaluating the quality of generated comments remains a challenge. Traditional metrics such as BLEU and ROUGE fail to capture domain-specific nuances, while human evaluation is costly and unscalable. In this paper, we present \texttt{evalSmarT}, a modular and extensible framework that leverages large language models (LLMs) as evaluators. The system supports over 400 evaluator configurations by combining approximately 40 LLMs with 10 prompting strategies. We demonstrate its application in benchmarking comment generation tools and selecting the most informative outputs. Our results show that prompt design significantly impacts alignment with human judgment, and that LLM-based evaluation offers a scalable and semantically rich alternative to existing methods.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages

[1]

A massive analysis of ethereum smart contracts empirical study and code metrics,

A. Pinna, S. Ibba, G. Baralla, R. Tonelli, and M. Marchesi, “A massive analysis of ethereum smart contracts empirical study and code metrics,” Ieee Access , vol. 7, pp. 78 194–78 213, 2019

work page 2019
[2]

Ccgir: Information retrieval-based code comment generation method for smart contracts,

G. Yang, K. Liu, X. Chen, Y . Zhou, C. Yu, and H. Lin, “Ccgir: Information retrieval-based code comment generation method for smart contracts,” Knowledge-Based Systems , vol. 237, p. 107858, 2022

work page 2022
[3]

Automatic smart contract comment generation via large language models and in-context learning,

J. Zhao, X. Chen, G. Yang, and Y . Shen, “Automatic smart contract comment generation via large language models and in-context learning,” Information and Software Technology , vol. 168, p. 107405, 2024

work page 2024
[4]

Leveraging llms for the quality assurance of software requirements,

S. Lubos, A. Felfernig, T. N. T. Tran, D. Garber, M. El Mansi, S. P. Erdeniz, and V .-M. Le, “Leveraging llms for the quality assurance of software requirements,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) , 2024, pp. 389–397

work page 2024
[5]

Can large language models serve as evaluators for code summarization?

Y . Wu, Y . Wan, Z. Chu, W. Zhao, Y . Liu, H. Zhang, X. Shi, and P. S. Yu, “Can large language models serve as evaluators for code summarization?” arXiv preprint arXiv:2412.01333 , 2024

arXiv 2024
[6]

Towards interpreting smart contract against contract fraud: A practical and automatic realiza- tion,

M. Li, J. Weng, A. Yang, J. Weng, and Y . Zhang, “Towards interpreting smart contract against contract fraud: A practical and automatic realiza- tion,” Cryptology ePrint Archive , 2020

work page 2020
[7]

Stan: Towards describing bytecodes of smart contract,

X. Li, T. Chen, X. Luo, T. Zhang, L. Yu, and Z. Xu, “Stan: Towards describing bytecodes of smart contract,” in 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS) . IEEE, 2020, pp. 273–284

work page 2020
[8]

A multi-modal transformer-based code summarization approach for smart contracts,

Z. Yang, J. Keung, X. Yu, X. Gu, Z. Wei, X. Ma, and M. Zhang, “A multi-modal transformer-based code summarization approach for smart contracts,” in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) . IEEE, 2021, pp. 1–12

work page 2021

Show all 15 references

[9]

Automating user notice generation for smart contract functions,

X. Hu, Z. Gao, X. Xia, D. Lo, and X. Yang, “Automating user notice generation for smart contract functions,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 5–17

2021
[10]

Machine translation- based fine-grained comments generation for solidity smart contracts,

C. Shi, Y . Xiang, J. Yu, K. Sood, and L. Gao, “Machine translation- based fine-grained comments generation for solidity smart contracts,” Information and Software Technology , vol. 153, p. 107065, 2023

2023
[11]

Scla: Automated smart contract summarization via llms and semantic augmentation,

Y . Mao, X. Li, W. Li, X. Wang, and L. Xie, “Scla: Automated smart contract summarization via llms and semantic augmentation,” arXiv preprint arXiv:2402.04863, 2024

2024 arXiv
[12]

Fmcf: A fusing multiple code features approach based on transformer for solidity smart contracts source code summarization,

G. Lei, D. Zhang, J. Xiao, G. Fan, Y . Cao, and Z. Feng, “Fmcf: A fusing multiple code features approach based on transformer for solidity smart contracts source code summarization,” Applied Soft Computing, vol. 166, p. 112238, 2024

2024
[13]

Automating com- ment generation for smart contract from bytecode,

J. Xiang, Z. Gao, L. Bao, X. Hu, J. Chen, and X. Xia, “Automating com- ment generation for smart contract from bytecode,” ACM Transactions on Software Engineering and Methodology , 2024

2024
[14]

Ccgra: Smart contract code comment generation with retrieval-enhanced approach

Z. Zhang, S. Chen, G. Fan, G. Yang, and Z. Feng, “Ccgra: Smart contract code comment generation with retrieval-enhanced approach.” in SEKE, 2023, pp. 212–217

2023
[15]

Large language models for blockchain security: A systematic literature review,

Z. He, Z. Li, S. Yang, H. Ye, A. Qiao, X. Zhang, X. Luo, and T. Chen, “Large language models for blockchain security: A systematic literature review,” arXiv preprint arXiv:2403.14280 , 2024

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

A massive analysis of ethereum smart contracts empirical study and code metrics,

A. Pinna, S. Ibba, G. Baralla, R. Tonelli, and M. Marchesi, “A massive analysis of ethereum smart contracts empirical study and code metrics,” Ieee Access , vol. 7, pp. 78 194–78 213, 2019

work page 2019

[2] [2]

Ccgir: Information retrieval-based code comment generation method for smart contracts,

G. Yang, K. Liu, X. Chen, Y . Zhou, C. Yu, and H. Lin, “Ccgir: Information retrieval-based code comment generation method for smart contracts,” Knowledge-Based Systems , vol. 237, p. 107858, 2022

work page 2022

[3] [3]

Automatic smart contract comment generation via large language models and in-context learning,

J. Zhao, X. Chen, G. Yang, and Y . Shen, “Automatic smart contract comment generation via large language models and in-context learning,” Information and Software Technology , vol. 168, p. 107405, 2024

work page 2024

[4] [4]

Leveraging llms for the quality assurance of software requirements,

S. Lubos, A. Felfernig, T. N. T. Tran, D. Garber, M. El Mansi, S. P. Erdeniz, and V .-M. Le, “Leveraging llms for the quality assurance of software requirements,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) , 2024, pp. 389–397

work page 2024

[5] [5]

Can large language models serve as evaluators for code summarization?

Y . Wu, Y . Wan, Z. Chu, W. Zhao, Y . Liu, H. Zhang, X. Shi, and P. S. Yu, “Can large language models serve as evaluators for code summarization?” arXiv preprint arXiv:2412.01333 , 2024

arXiv 2024

[6] [6]

Towards interpreting smart contract against contract fraud: A practical and automatic realiza- tion,

M. Li, J. Weng, A. Yang, J. Weng, and Y . Zhang, “Towards interpreting smart contract against contract fraud: A practical and automatic realiza- tion,” Cryptology ePrint Archive , 2020

work page 2020

[7] [7]

Stan: Towards describing bytecodes of smart contract,

X. Li, T. Chen, X. Luo, T. Zhang, L. Yu, and Z. Xu, “Stan: Towards describing bytecodes of smart contract,” in 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS) . IEEE, 2020, pp. 273–284

work page 2020

[8] [8]

A multi-modal transformer-based code summarization approach for smart contracts,

Z. Yang, J. Keung, X. Yu, X. Gu, Z. Wei, X. Ma, and M. Zhang, “A multi-modal transformer-based code summarization approach for smart contracts,” in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) . IEEE, 2021, pp. 1–12

work page 2021