Pith. sign in

REVIEW 5 major objections 4 minor 43 references

iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM agent that reads protocol specs, writes conformance tests, and debugs them lifts first-try test-program success by 4.7 to 10.8 times.

desk verdict Useful pipeline, but the headline Pass@1 gain is inflated by an unfair repair budget; the core idea and the RFC 9177 find survive scrutiny. read the letter →

arxiv 2507.00378 v2 pith:QDKEK2Z4 submitted 2025-07-01 cs.SE cs.AI

Xikai Sun , Fan Dang , Shiqi Jiang , Jingao Xu , Kebin Liu , Xin Miao , Zihao Yang , Weichen Zhang
show 3 more authors
Haimo Lu Yawen Zheng Yunhao Liu
This is my paper
classification cs.SEcs.AI
keywords conformancetestingcommunicationprotocolslargelanguagemodelsretrieval-augmentedgenerationchain-of-thoughttestprogramiterativeoptimizationCoAP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a large language model can drive the entire conformance-testing workflow for communication protocols: reading the specification, generating test cases, writing executable test programs against a given implementation library, running them, and repairing them until they succeed. The proposed agent, iPanda, combines keyword-based extraction of functional points from RFC-style documents, retrieval-augmented generation over the implementation's own source code, a chain-of-thought decomposition of client/server test scenarios, and an execution loop that feeds error logs back into the model up to a retry limit. On test-case sets derived from the CoAP and RSocket specifications, the paper reports that iPanda raises the Pass@1 rate of test-program generation by 4.675x to 10.751x over a single-step pure-LLM baseline, with GPT-4o rising from 17.32% to 80.95% on the CoAP set. A reader should care because, if these numbers hold, conformance testing shifts from weeks of manual script writing to an automated, protocol-agnostic pipeline whose results can be traced to specific specification clauses.

What carries the argument

The load-bearing mechanism is the iterative optimization loop of Section III-C: each failed program, its execution log, and a debugging prompt are appended to the model's context, so the next generated program is conditioned on actual runtime feedback rather than only on the original test case. The loop is bounded by a short-term memory window of 10 interactions and a default retry cap of 6, and it is coupled to an automated testing platform that parses the JSON blueprint, launches subprograms in an isolated virtual environment, and returns execution errors. Two supporting mechanisms carry the input side: keyword-based test-case generation locates functional points by RFC 2119 keywords such as MUST and SHALL, reaching 87.5% section coverage across 32 CoAP-related RFC documents, and code-oriented retrieval-augmented generation retrieves implementation source files by cosine similarity to ground API usage in the generated programs.

What would settle it

Run the published CoAP-set and RSocket-set through an independent oracle—human-authored tests or an existing conformance checker—and compare each program's Pass@1 label against the oracle's verdict; if many LLM-labeled successes fail the oracle, the reported multi-fold improvements do not measure true conformance.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in LLM-based conformance testing is not understanding the specification but grounding that understanding in the concrete API of an implementation library and producing code that actually executes. iPanda closes that loop: keyword-based test-case generation turns MUST and SHALL statements into structured cases; a customized chain-of-thought strategy decomposes each case into per-role subprograms and a startup-order blueprint in JSON; the execution platform runs the program and returns real error logs; and the iterative optimization algorithm uses those logs as context for repair, up to a default maximum of six attempts. The ablation studies attribute most of the gain to the repair loop, and retrieval-augmented generation to API grounding when the library is too new for the model to know it: on the RSocket set, enabling RAG raises Pass@1 from 14.51% to 38.71%. The paper presents iPanda as the first LLM-based framework that integrates specification, implementation, and test execution into one conformance-testing process.

Load-bearing premise

The claimed success rates assume that the model's judgment of whether its own generated tests pass, judged against assertions the same model wrote, is trustworthy, since the only shields are a keyword filter and manual review.

Editorial extensions

If this is right

  • If the Pass@1 gains are real, protocol conformance suites can be built from specification documents alone, without a human test engineer writing each script.
  • The ablation results imply that libraries with little training exposure benefit most from retrieval-augmented generation, while the iterative repair loop helps regardless of model scale.
  • The finding that increasing repair iterations helps more than increasing repeated sampling favors sequential, feedback-driven generation over parallel sampling for this task.
  • Concentrating negative samples in RFC 9177 matches the implementation's documented lack of support, so the pipeline can point testers to specific unimplemented specification areas.
  • The pipeline is protocol-agnostic and model-agnostic in the paper's setup, so the same agent can be pointed at other RFC-based protocols with a new specification document and implementation library.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the evaluator is the same model that wrote the assertions, an independent oracle could change the Pass@1 numbers; the paper's manual review is a filter, not a formal oracle.
  • Editorial inference: the 87.5% section coverage bounds the claim's scope; the generated sets do not exercise the 12.5% of specification text without keywords.
  • Editorial inference: a natural stress test is to run iPanda on a protocol with a mature, hand-written conformance suite and compare verdicts, which would separate genuine specification checking from convergence on the model's own criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. iPanda is an LLM-based agent for protocol conformance testing. It extracts functional points from RFC-style specifications via a keyword-based test-case generator, uses retrieval-augmented generation and a customized chain-of-thought to produce executable test programs for a target implementation library, executes these programs, and iteratively repairs them with execution feedback up to a maximum reasoning step Smax. The framework is evaluated on a CoAP test set (231 cases, aiocoap) and an RSocket test set (62 cases, rsocket-py) with GPT-4o, DeepSeek-V3, and Qwen2.5-Coder-32B, reporting Pass@1 improvements of 4.675x to 10.751x over a pure-LLM baseline and a conformance finding that aiocoap has not implemented RFC 9177.

Significance. If the reported results are valid, iPanda would be a useful contribution to the relatively unexplored area of LLM-based protocol conformance testing. The architecture is sensible, the RFC 9177 finding is specific and externally corroborated by the aiocoap README, and the use of execution feedback for iterative repair is a plausible mechanism for improving test-program generation. However, the headline gains are currently not fairly measured, the correctness assessment relies on the LLM evaluating its own outputs, and at least one ablation claim is directly contradicted by the reported table. These issues are load-bearing for the paper's central quantitative claims, so the manuscript needs substantial revision before the results can be accepted at face value.

major comments (5)
  1. [§IV-C, Table II, Abstract] The headline Pass@1 comparisons do not compare like with like. According to §IV-A4, the baseline receives a single generation with no feedback, while iPanda runs Algorithm 1 with up to Smax=6 iterations. Equation (5) with k=1 measures first-try success for the baseline but final-output success for iPanda. The paper's own Fig. 4(a) provides a budget-matched comparison: at Smax=1,k=6 the number of positive samples is 89, and at Smax=6,k=1 it is 182, a ratio of about 2.0 for GPT-4o rather than the claimed 4.675x. The abstract's improvement factors must be recomputed with equal budgets or explicitly reframed as agent-vs-single-shot rather than Pass@1. There is also an internal inconsistency: Fig. 4(a) reports 31 positive samples at Smax=1,k=1, while Table II's baseline of 17.32% on 231 samples implies 40 positive samples; this discrepancy needs to be explained.
  2. [§IV-E and §III-D] The correctness labels that define Pass@1 and the conformance findings are produced by the LLM evaluating its own generated programs against its own test assertions, supplemented by a keyword filter and manual review. No independent oracle, released execution logs, or inter-rater protocol is described. Because the iterative optimization loop receives feedback from the same evaluator that defines success, the reported gains could partly reflect the model learning to satisfy its own criteria rather than true specification conformance. The paper should at minimum report manual verification statistics for a sample of positive and negative classifications, ideally with independent annotators, and release execution artifacts so the RFC 9177 finding can be reproduced.
  3. [§IV-B and §IV-C] The maximum iteration step is selected on the same CoAP-set that is later used for the headline results. Specifically, §IV-B constructs a histogram of required reasoning iterations on the CoAP-set and then sets Smax=6 for all subsequent experiments, and Table II reports the resulting Pass@1 on the same CoAP-set. This is a data-dependent parameter selection that can inflate the reported improvement. The authors should report Pass@1 for other values of Smax, use a separate validation set for parameter selection, or otherwise quantify the selection effect.
  4. [Table III and §IV-D] The ablation claim that 'on the RSocket-set, even without the iterative optimization algorithm, iPanda still outperforms the baseline' is contradicted by Table III, which reports 3.23% for iPanda without iterative optimization versus 11.29% for the baseline on RSocket-set. This suggests that adding code-oriented RAG without the iterative loop decreases performance on that dataset, which also weakens the stated conclusion about RAG's effectiveness. The table and the text must be reconciled, and the interpretation of the RAG ablation needs to be revisited.
  5. [§IV-A5, Eq. (5)] Equation (5) defines Pass@k using the XOR operator, but the standard metric Pass@k is the probability that at least one of k generated solutions is correct, which requires a logical OR. Using XOR changes the metric: if two generated solutions are both correct, XOR returns 0, undercounting successes. This affects the quantitative interpretation of all the reported improvements and should be corrected.
minor comments (4)
  1. [§I and §VI] There are several language and formatting issues: 'we using iPanda to perform' in the contributions list, 'ierative optimization algorithm' in the conclusion, and 'B Corresponding Authors' in the author block. These should be fixed in a copyediting pass.
  2. [§IV-E] RFC 9177 is mentioned by name but is not included in the reference list; a proper citation for RFC 9177 should be added.
  3. [References] Reference [25], for boofuzz, contains the placeholder text 'THIS REFERENCE STILL NEEDS TO BE FIXED' and must be completed.
  4. [§I] The claim of being 'the first' LLM-based framework for protocol conformance testing is asserted without a systematic survey; given the existence of related tools discussed in §V, such a claim should be made more cautiously or supported by a dedicated prior-work comparison.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported Pass@1 gains rest on self-generated test assertions and a test-set-tuned iteration cap; the RFC 9177 finding gives partial external grounding.

  1. self definitional [Sec. III-A and Sec. III-C; metric in Sec. IV-A-5]
    ""The test cases generated by the LLM comprise four components: ... test assertions ... The test assertions define the criteria for judging test success, which serve as the basis for evaluating the results. ... Once the program runs without errors, the LLM evaluates the success of the test by comparing the test assertions defined in the test case with the executed program and its results.""

    The 'Test Assertion' in each test case is written by the LLM during keyword-based TCG, and the same LLM later labels the generated test program as passing by checking it against that self-written assertion. Pass@1 in Eq. (5) therefore measures whether the LLM's program is consistent with the LLM's own assertion text, not whether the program satisfies an externally fixed conformance criterion. The semantic labels behind Table II are produced by the system itself, with no independent oracle for the pass/fail classification of individual generated programs.

  2. fitted input called prediction [Sec. IV-B and Sec. IV-C / Table II]
    ""when setting the maximum step to 10, using GPT-4o, iPanda successfully tested 195 cases in the CoAP-set ... Resultly, we set the maximum step of iterations to 6 in subsequent experiments to achieve the optimal marginal benefit. ... P ass@1 is improved by 4.675×(GPT-4o), 6×(Deepseek-V3), and 10.751×(Qwen2.5-Coder-32B) respectively.""

    Smax=6 is selected by inspecting the iteration-cost histogram computed on the CoAP-set, and the same CoAP-set is then used to report the final Pass@1 in Table II. The reported 'prediction' is thus measured under a hyperparameter fitted to the evaluation set itself. In addition, the baseline is a single generation while iPanda receives up to six repair rounds (Alg. 1); the paper's own Fig. 4(a) shows Smax=1,k=6 gives 89 positives whereas Smax=6,k=1 gives 182, shrinking the apparent GPT-4o gain under an equal-budget comparison. The headline factors therefore partly reflect test-set tuning and budget asymmetry rather than a pure first-try success rate.

full rationale

The paper is not built on a self-citation chain or an imported uniqueness theorem; references to the authors' prior work (e.g., Autodroid) are background and not load-bearing. However, the central quantitative claim is partially self-referential. The test-case assertions that define 'success' are LLM-generated, and the same LLM is the judge of whether its own program satisfies those assertions; no independent oracle is used for the semantic pass labels in Table II. The iterative optimization loop itself is a legitimate mechanism, and the RFC 9177 conformance finding is externally corroborated by aiocoap's GitHub documentation, which is genuine evidence. Still, the reported 4.675x-10.751x gains are not a pure Pass@1 comparison: Smax=6 is chosen from the CoAP-set histogram and then used to report Pass@1 on the same CoAP-set, and the baseline is single-shot while iPanda is allowed up to six repair rounds. The paper's own Fig. 4(a) provides the equal-budget data point (Smax=1,k=6: 89 positives; Smax=6,k=1: 182 positives), which materially reduces the headline factor. These issues do not make the whole framework circular, but they make the headline metric partly defined by the system's own outputs and tuning choices.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework postulates no physical entities. Its empirical claims rest on hand-chosen hyperparameters and on the validity of self-generated, manually reviewed test sets and LLM-based correctness assessment.

free parameters (3)
  • Maximum reasoning iterations Smax = 6
    Set in Section IV-B from a histogram of successful iterations on the CoAP-set, then used for all headline Pass@1 results on that same set.
  • Short-term memory window size m = 10
    Chosen in Section IV-A without a sensitivity study; affects the amount of error feedback the LLM sees during iterative repair.
  • Sampling temperature and top-p = 0 and 0.1
    Set in Section IV-A; no analysis of sensitivity, and temperature 0 does not guarantee determinism for API models.
assumptions (4)
  • domain assumption RFC 2119 keywords (MUST, SHALL, etc.) reliably locate the functional points that need conformance testing.
    Used to justify keyword-based TCG in Section III-A; the 87.5% section coverage experiment measures keyword presence, not whether each extracted paragraph is actually a testable requirement.
  • domain assumption A generated test program that executes and whose assertions pass under LLM evaluation demonstrates conformance.
    Section IV-E relies on this to classify positive samples; the authors' two-stage filtering and manual review imply the LLM's raw judgment is not sufficient, so the final verdicts depend on human judgment.
  • domain assumption The manually reviewed CoAP-set and RSocket-set are valid and unbiased benchmarks.
    All Pass@1 numbers use these self-created sets; no inter-rater agreement or selection criteria are reported, and the sets are not released.
  • domain assumption The single-step pure-LLM baseline is a fair comparator for measuring iPanda's added value.
    Section IV-A4 uses a single-step generation baseline with the same prompt; it does not compare against other agentic frameworks or iterative-repair baselines, so the gain could be partly due to the extra compute budget rather than the specific mechanisms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols." pith.science (2026). https://pith.science/paper/QDKEK2Z4

@misc{pith2026250700378,
  author       = {Pith},
  title        = {Pith review of: iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDKEK2Z4}},
  note         = {Machine review of arXiv:2507.00378}
}
read the original abstract

Conformance testing is essential for ensuring that protocol implementations comply with their specifications. However, traditional testing approaches involve manually creating numerous test cases and scripts, making the process labor-intensive and inefficient. Recently, Large Language Models (LLMs) have demonstrated impressive text comprehension and code generation abilities, providing promising opportunities for automation. In this paper, we propose iPanda, the first framework that leverages LLMs to automate protocol conformance testing. Given a protocol specification document and its implementation, iPanda first employs a keyword-based method to automatically generate comprehensive test cases. Then, it utilizes retrieval-augmented generation and customized CoT strategy to effectively interpret the implementation and produce executable test programs. To further enhance programs' quality, iPanda incorporates an iterative optimization mechanism to refine generated test scripts interactively. Finally, by executing and analyzing the generated tests, iPanda systematically verifies compliance between implementations and protocol specifications. Comprehensive experiments on various protocols show that iPanda significantly outperforms pure LLM-based approaches, improving the success rate (Pass@1) of test-program generation by factors ranging from 4.675 times to 10.751 times.

Figures

Figures reproduced from arXiv: 2507.00378 by the authors.

Figure 1
Figure 1. The overview of iPanda. executable program or reaches a retry limit, iPanda compiles a final debugging report and evaluates test outcomes, determin￾ing compliance with the protocol’s conformance requirements. To address the challenges mentioned in Sec. I and enhance iPanda’s effectiveness, we have designed several optimization methods. The following sub-sections will provide detailed descriptions of these methods. A… view at source ↗
Figure 2
Figure 2. Example of generating test cases using few-shot in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The number of code successfully generated within the maximum iteration step limit, using (a) GPT-4o, (b) Deepseek [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) The joint experiment of maximum reasoning step [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 28 canonical work pages

  1. [1]

    Llmif: Augmented large language model for fuzzing iot devices,

    J. Wang, L. Yu, and X. Luo, “Llmif: Augmented large language model for fuzzing iot devices,” in 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 881–896

    work page 2024

  2. [2]

    Autodroid: Llm-powered task automation in android,

    H. Wen, Y . Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y . Liu, Y . Zhang, and Y . Liu, “Autodroid: Llm-powered task automation in android,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , 2024, pp. 543–557

    work page 2024

  3. [3]

    Autodroid-v2: Boosting slm-based gui agents via code generation,

    H. Wen, S. Tian, B. Pavlov, W. Du, Y . Li, G. Chang, S. Zhao, J. Liu, Y . Liu, Y .-Q. Zhanget al., “Autodroid-v2: Boosting slm-based gui agents via code generation,” arXiv preprint arXiv:2412.18116 , 2024

    arXiv 2024

  4. [4]

    (2024) Rsocket

    rsocket. (2024) Rsocket. [Online]. Available: https://rsocket.io/about/ protocol/

    work page 2024

  5. [5]

    (2025) rsocket-py

    rsocket py. (2025) rsocket-py. [Online]. Available: https://github.com/ rsocket/rsocket-py

    work page 2025

  6. [6]

    Blind software- assisted conformance and security assessment of fido2/webauthn imple- mentations

    A. V . Grammatopoulos, I. Politis, and C. Xenakis, “Blind software- assisted conformance and security assessment of fido2/webauthn imple- mentations.” J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl., vol. 13, no. 2, pp. 96–127, 2022

    work page 2022

  7. [7]

    An llm agent for functional bug detection in network protocols,

    M. Zheng, C. Wang, X. Liu, J. Guo, S. Feng, and X. Zhang, “An llm agent for functional bug detection in network protocols,” arXiv preprint arXiv:2506.00714, 2025

    arXiv 2025

  8. [8]

    Rfc 7252: The constrained application protocol (coap),

    Z. Shelby, K. Hartke, and C. Bormann, “Rfc 7252: The constrained application protocol (coap),” 2014

    work page 2014

Show all 43 references
  1. [9]

    R. Editor. (2025) Rfc editor. [Online]. Available: https://www.rfc-editor. org/search/rfc search detail.php?page=All&title=coap

    2025

  2. [10]

    Automated software test data generation with generative adversarial networks,

    X. Guo, H. Okamura, and T. Dohi, “Automated software test data generation with generative adversarial networks,” IEEE Access, vol. 10, pp. 20 690–20 700, 2022

    2022

  3. [11]

    (2025) gpt-o3-pro

    OpenAI. (2025) gpt-o3-pro. [Online]. Available: https://platform.openai. com/docs/models/o3-pro

    2025

  4. [12]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  5. [13]

    (2025) deepseek-r1

    Deepseek. (2025) deepseek-r1. [Online]. Available: https://api-docs. deepseek.com/news/news250528

    2025

  6. [14]

    (2025) deepseek-v3

    ——. (2025) deepseek-v3. [Online]. Available: https://api-docs. deepseek.com/news/news250325

    2025

  7. [15]

    (2025) claude-opus-4

    Anthropic. (2025) claude-opus-4. [Online]. Available: https://docs. anthropic.com/en/docs/about-claude/models/overview

    2025

  8. [16]

    (2025) Gemini-2.5-pro

    Google. (2025) Gemini-2.5-pro. [Online]. Available: https://cloud. google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro

    2025

  9. [17]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al. , “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  10. [18]

    (2024) text-embedding-3-large

    OpenAI. (2024) text-embedding-3-large. [Online]. Available: https: //openai.com/index/new-embedding-models-and-api-updates/

    2024

  11. [19]

    Autoiot: Llm-driven automated natural language programming for aiot applications,

    L. Shen, Q. Yang, Y . Zheng, and M. Li, “Autoiot: Llm-driven automated natural language programming for aiot applications,” arXiv preprint arXiv:2503.05346, 2025

    2025 arXiv

  12. [20]

    Critic method,

    A. Alinezhad, J. Khalili, A. Alinezhad, and J. Khalili, “Critic method,” New methods and applications in multiple attribute decision making (MADM), pp. 199–203, 2019

    2019

  13. [21]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  14. [22]

    (2025) aiocoap

    aiocoap. (2025) aiocoap. [Online]. Available: https://github.com/chrysn/ aiocoap

    2025

  15. [23]

    An introduction to the testing and test control notation (ttcn-3),

    J. Grabowski, D. Hogrefe, G. R ´ethy, I. Schieferdecker, A. Wiles, and C. Willcock, “An introduction to the testing and test control notation (ttcn-3),” Computer Networks, vol. 42, no. 3, pp. 375–403, 2003

    2003

  16. [24]

    Scapy-a powerful interactive packet manipulation program,

    R. Rohith, M. Moharir, G. Shobha et al., “Scapy-a powerful interactive packet manipulation program,” in 2018 international conference on networking, embedded and wireless systems (ICNEWS) . IEEE, 2018, pp. 1–5

    2018

  17. [25]

    boofuzz documentation,

    J. Pereyda, “boofuzz documentation,” THIS REFERENCE STILL NEEDS TO BE FIXED , 2019

    2019

  18. [26]

    Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage,

    C. Lemieux and K. Sen, “Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage,” in Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, 2018, pp. 475–485

    2018

  19. [27]

    T-fuzz: fuzzing by program transformation,

    H. Peng, Y . Shoshitaishvili, and M. Payer, “T-fuzz: fuzzing by program transformation,” in 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 2018, pp. 697–710

    2018

  20. [28]

    Online machine learning for intrusion detection in electric vehicle charging systems,

    F. Makhmudov, D. Kilichev, U. Giyosov, and F. Akhmedov, “Online machine learning for intrusion detection in electric vehicle charging systems,” Mathematics, vol. 13, no. 5, p. 712, 2025

    2025

  21. [29]

    Using large language models to generate junit tests: An empirical study,

    M. L. Siddiq, J. C. Da Silva Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V . Carvalho Lopes, “Using large language models to generate junit tests: An empirical study,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering...

    2024

  22. [30]

    An initial investigation of chatgpt unit test generation capability,

    V . Guilherme and A. Vincenzi, “An initial investigation of chatgpt unit test generation capability,” in Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing , 2023, pp. 15–24

    2023

  23. [31]

    Pentestgpt: An llm-empowered automatic penetration testing tool,

    G. Deng, Y . Liu, V . Mayoral-Vilches, P. Liu, Y . Li, Y . Xu, T. Zhang, Y . Liu, M. Pinzger, and S. Rass, “Pentestgpt: An llm-empowered automatic penetration testing tool,” arXiv preprint arXiv:2308.06782 , 2023

    2023 arXiv

  24. [32]

    Llm as dba,

    X. Zhou, G. Li, and Z. Liu, “Llm as dba,” arXiv preprint arXiv:2308.05481, 2023

    2023 arXiv

  25. [33]

    Mind2web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2web: Towards a generalist agent for the web,” Advances in Neural Information Processing Systems , vol. 36, pp. 28 091–28 114, 2023

    2023

  26. [34]

    Webgpt: Browser- assisted question-answering with human feedback,

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders et al., “Webgpt: Browser- assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  27. [35]

    Weinfer: Unleashing the power of we- bgpu on llm inference in web browsers,

    Z. Chen, Y . Ma, M. Liu et al., “Weinfer: Unleashing the power of we- bgpu on llm inference in web browsers,” in THE WEB CONFERENCE 2025, 2025

    2025

  28. [36]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,

    X. Du, G. Zheng, K. Wang, J. Feng, W. Deng, M. Liu, B. Chen, X. Peng, T. Ma, and Y . Lou, “Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,” arXiv preprint arXiv:2406.11147 , 2024

    2024 arXiv

  29. [37]

    Generative ai service implementation using llm application architecture: based on rag model and langchain framework,

    C. Jeong, “Generative ai service implementation using llm application architecture: based on rag model and langchain framework,” Journal of Intelligence and Information Systems, vol. 29, no. 4, pp. 129–164, 2023

    2023

  30. [38]

    Rag in health care: a novel framework for improving communication and decision-making by addressing llm limitations,

    K. K. Y . Ng, I. Matsuba, and P. C. Zhang, “Rag in health care: a novel framework for improving communication and decision-making by addressing llm limitations,” NEJM AI , vol. 2, no. 1, p. AIra2400380, 2025

    2025

  31. [39]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  32. [40]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” Advances in Neural Information Processing Systems , vol. 36, pp. 38 154–38 180, 2023

    2023

  33. [41]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  34. [42]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR) , 2023

    2023

  35. [43]

    Reflex- ion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflex- ion: Language agents with verbal reinforcement learning,” Advances in Neural Information Processing Systems , vol. 36, pp. 8634–8652, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.