Pith. sign in

REVIEW 3 major objections 5 minor 35 references

IntelliAudit: Using Large Language Models to Evaluate Audit Controls

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read IntelliAudit claims that a retrieval-grounded, selectively escalated multi-agent LLM system can support IT audit evidence evaluation, with sufficiency judgments remaining the point where human oversight is required.

desk verdict Honest systems paper with a genuinely new benchmark, but the evaluation misses an easy internal check that would have strengthened RQ2. read the letter →

arxiv 2608.07688 v1 pith:S23EVHMX submitted 2026-08-07 cs.AI cs.CLcs.CRcs.HCcs.MA

Allison Wilson , Sina Moradi Sabet , Diar Shakimov , Panteha Shahrivar , Mohammad Reza Bagheri , Dean Konenkamp , Mohammad A. Tayebi This is my paper · ORCID
classification cs.AIcs.CLcs.CRcs.HCcs.MA
keywords ITauditISO/IEC27001largelanguagemodelsmulti-agentsystemsretrieval-augmentedgenerationevidencesufficiencyhuman-in-the-loopreadiness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IntelliAudit sets out to show that a retrieval-grounded, selectively escalated multi-agent LLM system can assist IT auditors in judging whether organizational evidence satisfies semantic security and compliance controls, using ISO/IEC 27001 as the test case. The paper argues that the task should be framed as an explainable evidence-grounded recommendation, not autonomous certification, and that the system's output should be an auditor-facing package with cited evidence, rationale, missing-evidence analysis, and remediation guidance. Across four simulated organizations and 14 controls, practicing auditors rated the system positively for control interpretation (mean 3.92) and factual evidence evaluation (mean 3.75), while evidence fulfillment scored lower (mean 3.20); audit-preparation users rated usefulness at 3.84. If the central claim is right, AI can take on labor-intensive evidence review and audit preparation, but the final sufficiency judgment must remain a human decision.

What carries the argument

The central mechanism is a selectively escalated multi-agent workflow. The Auditor always produces the first evidence-grounded assessment; the Defender is invoked only for adverse findings (PARTIAL or NON_COMPLIANT) to search for overlooked mitigating evidence; the Judge is invoked only when the agents disagree or when a human auditor supplies feedback, and it is the only agent allowed to return INSUFFICIENT_EVIDENCE, an epistemic abstention distinct from PARTIAL. Retrieval grounding is enforced by a hybrid search over semantic, document-level, and spreadsheet-row indexes, and synthesis is restricted to the retrieved and cited evidence bundle, with every substantive claim required to be supported by a cited artifact. The output is a recommendation package (status, cited evidence, rationale, missing-evidence analysis, follow-up actions) that remains provisional until a human auditor accepts or challenges it.

What would settle it

Run IntelliAudit on a benchmark of real or realistically constructed ISO 27001 evidence bundles whose sufficiency status has been independently determined by completed professional audits, then count, over all control instances, the fraction of COMPLIANT recommendations that later receive an adverse audit finding; if that false-compliance rate approaches the base rate of non-compliance, the system's sufficiency judgments add no information.

Watch

Extended reading notes

Core claim

The paper's central claim is that IT audit evidence evaluation can be operationalized as a retrieval-grounded recommendation task in which an Auditor agent produces a provisional status from cited evidence, a Defender searches for overlooked mitigating evidence when the initial finding is adverse, and a Judge resolves disagreements and may issue INSUFFICIENT_EVIDENCE when the record cannot support a reliable substantive status. The authors report that this workflow supports control interpretation beyond surface text and produces factually accurate evidence evaluation, but that converting evidence into calibrated sufficiency judgments remains harder, with human feedback acting as a conservative calibration layer: across 27 case reviews, 15 re-adjudications changed status, 13 moved away from a more permissive label, and none ended as COMPLIANT. The authors therefore present IntelliAudit as decision support for audit preparation and evidence review rather than as an autonomous certification system.

Load-bearing premise

The load-bearing premise is that Likert ratings from a small panel of practicing auditors (12 to 17 per control) on author-constructed simulated evidence corpora are a reliable measure of audit-evidence sufficiency, because the paper states that no public ground-truth dataset exists for ISO 27001 evidence sufficiency and uses human evaluation as the primary reference.

Editorial extensions

If this is right

  • Multi-agent escalation can be selective: the Judge activated in only 5 of 48 control cases, changing 3 statuses, all in a stricter direction, so adversarial review can be reserved for contested cases without a universal debate tax.
  • Human feedback routed through the Judge functions as a conservative calibration layer: 15 of 27 feedback interactions changed status, 13 moved away from a more permissive label, and no revision ended as COMPLIANT.
  • The system is more reliable at factual evidence interpretation than at sufficiency calibration, so deployment should pair LLM evidence review with deterministic completeness checks, especially over spreadsheet evidence where Control 5.9 showed a population-level reasoning error.
  • Audit-preparation users found the outputs actionable for identifying gaps and next steps (mean 3.84), which supports using such systems to help organizations prepare evidence before a formal audit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to build a public ground-truth benchmark of ISO 27001 evidence sufficiency from completed real audits; the paper explicitly notes that no such dataset exists, so without it the sufficiency results rest on subjective ratings.
  • The selective-escalation pattern likely transfers to other evidence-sufficiency tasks such as regulatory filing review or due-diligence documentation, but the paper evaluates only ISO 27001, so this is an extrapolation.
  • The Control 5.9 spreadsheet failure suggests a concrete improvement the paper does not implement: verify population coverage of structured evidence (for example, all asset categories in an inventory) before allowing LLM synthesis to generalize.
  • If sufficiency calibration improves, the human role in audits may shift from reviewing evidence to adjudicating system-recommended gaps; the paper's traceable human-feedback loop is a step toward that division of labor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents IntelliAudit, a retrieval-grounded multi-agent system for IT audit evidence evaluation. Given an ISO/IEC 27001 control and an organizational evidence corpus, the Auditor produces an initial evidence-grounded recommendation, a Defender searches for mitigating evidence when the finding is adverse, and a Judge adjudicates disagreements or human feedback; the final output is an auditor-facing recommendation with cited evidence, rationale, missing-evidence analysis, and remediation guidance. The system is evaluated on an author-constructed benchmark covering 14 controls across four simulated organizations, using Likert-scale ratings from practicing auditors (Group A) and cybersecurity-informed audit-readiness users (Group B), plus a small blinded forced-choice comparison and an analysis of human-feedback re-adjudications. The paper reports that the system supports control interpretation and artifact-level evidence review, that sufficiency judgments are harder than interpretation, that spreadsheet-heavy cases expose completeness errors, and that human feedback often corrects overly permissive recommendations. The claims are modest in scope, and the paper explicitly frames the system as a decision-support tool rather than an autonomous certification system.

Significance. If the evaluation supported the claims, IntelliAudit would be a useful contribution to an under-studied area: LLM support for heterogeneous-evidence IT audit review. The paper's strengths include a clear task formulation with traceability constraints, a sensible selective-escalation workflow, a substantively realistic simulated benchmark, and unusual honesty about failures such as Control 5.9's population-level spreadsheet error and the calibration problems in Controls 5.24 and 6.4. The authors also ship prompts and configuration files, which aids reproducibility. However, the evidence quality is limited: the panels are small (n=12-17 per control for Group A), the benchmark is author-constructed with no external ground truth, the multi-agent and feedback analyses are based on small, purposively selected subsets, and no statistical tests or inter-rater reliability are reported. The central claims are modest enough that these limitations do not require rejection, but they do require either additional analysis or explicit re-scoping of what the evaluation can establish.

major comments (3)
  1. [Benchmark and Evaluation, Benchmark] The benchmark assigns each control instance a seeded evidence condition ('compliant,' 'semi-compliant,' or 'non-compliant'), which is an internal ground truth for the substantive status, yet the paper never reports agreement between IntelliAudit's final (or Auditor-only) status labels and these seeds. The statement 'Because no public ground-truth dataset exists for ISO 27001 evidence sufficiency, we use human evaluation as the primary reference' addresses external reproducibility, not the internal labels the authors already built. As a result, RQ2 ('the factual accuracy and sufficiency of the cited evidence in fulfilling the assigned status') is supported only by Group A Likert means (Factual 3.75, SD 1.04; Fulfill 3.20, SD 1.32) from small panels, and no precision, recall, or confusion analysis is reported. A simple table of final status labels by seeded condition, including INSUFFICIENT_EVIDENCE abstentions, would directly test the load-bearing claim that final recommendations are evidence-grounded. I request this analysis, or an explicit statement that the seeds are not intended as ground truth, with justification.
  2. [Results, Effect of Multi-Agent Review] The evidence for the multi-agent workflow's benefit is thin and selected by the activation condition. The Judge was activated in 5 of 48 cases; among these, the blinded forced choice (3 reviewers, 15 judgments) yielded 9 preferences for Judge-final output, concentrated in status-changing cases (7/9), while the two status-preserving cases favored the Auditor-only output (4/6). With five purposively selected cases and no statistical test, this cannot support a general claim that multi-agent review improves quality. The paper's own caveat ('not evidence of general multi-agent superiority') is appropriate, but the Introduction still presents the 'selectively escalated multi-agent workflow' as a contribution. Please either report all 48 cases in the ablation (e.g., forcing the Judge to run on all cases) or explicitly re-scope the claim to a feasibility demonstration.
  3. [Results, Human Feedback as Audit Calibration] The claim that human feedback 'corrects overly permissive recommendations' is partly a design property rather than a measured system capability. Because the workflow routes any human feedback h_j to the Judge for re-evaluation (Human Feedback section), it is expected that supplying a correction often changes the label; the observed 15/27 status changes and 9/15 moves away from COMPLIANT do not distinguish between genuine model correction and the mechanical effect of conditioning the Judge on the human's stated conclusion. What is missing is a control condition (e.g., re-running the Judge without feedback, or with feedback phrased as a no-change instruction) or an analysis of how often the Judge resists incorrect feedback. I recommend tempering the calibration claim or adding such a comparison.
minor comments (5)
  1. [Results, RQ4] The sentence reporting the mean usefulness rating contains a doubled closing parenthesis: 'across four organizations))'; please fix.
  2. [Benchmark and Evaluation, Benchmark] The benchmark is described as covering 14 ISO 27001 controls but also as having 12 assessment units (with 5.25–5.27 consolidated); the relationship between controls and assessment units should be stated explicitly in the main text, not only implied by Table 3.
  3. [Benchmark and Evaluation, Human Evaluation] The report says Group A ratings have n=12–17 per control, but it is unclear whether n counts distinct auditors, distinct rating instances, or both; please clarify the sampling unit and report inter-rater agreement (e.g., Krippendorff's alpha) if multiple ratings per control are available.
  4. [Methodology, Implementation] The implementation names 'Claude Sonnet 4.6' and 'LangGraph' without version details or citations; since these are central to reproducibility, please add references or state that they are proprietary/commercial components.
  5. [Results, Effect of Multi-Agent Review] The text says 44 of 48 final statuses (91.7%) matched the Auditor-agent-only label and then notes that one case produced no output; please state the denominator explicitly for the match rate (e.g., 44/47 output-producing cases) so the reader can interpret the 91.7% figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system's claims rest on external human raters and measured feedback outcomes, not on fitted parameters or self-citation chains.

full rationale

IntelliAudit does not derive results from fitted parameters or from prior claims by the same authors. The central evaluation is a human study: Group A practicing auditors and Group B audit-readiness users rated system outputs on Likert scales. The paper explicitly disclaims a public ground truth ('Because no public ground-truth dataset exists for ISO 27001 evidence sufficiency, we use human evaluation as the primary reference'), which is a limitation statement rather than a circular step. The human-feedback calibration result is an observed intervention effect: feedback is routed to the Judge, and the paper reports 15 status changes, 13 stricter, 9 from COMPLIANT; the direction of these changes is measured, not guaranteed by construction. The seeded compliant/semi/non-compliant benchmark conditions are internal, but the paper does not claim to predict them; it claims professional acceptability, which requires the external raters it uses. The absence of a reported confusion matrix against seeds is a completeness or validity concern, not circularity. Citations to prior work (D3, AuditFlow, etc.) are background and are not load-bearing; no author self-citations substitute for evidence. Therefore the derivation chain is self-contained and no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper makes no formal or quantitative derivation; its central claim rests on a working system whose behavior depends on hand-set retrieval hyperparameters and on strong domain assumptions: that simulated corpora represent real audit evidence, that Likert ratings from 12-17 auditors per control are valid measures of sufficiency, and that the dependency-graph encoding of ISO 27001 is faithful. The multi-agent feedback loop means some evaluation results are co-produced by the human reviewers, not the system alone. No new physical or formal entities are introduced.

free parameters (4)
  • retrieval top-k = 15
    Set in Implementation; each search backend returns up to 15 candidates; chosen by hand, not optimized.
  • similarity threshold = 0.35
    Set in Implementation; candidates below 0.35 are filtered; hand-set.
  • iteration limits = Auditor 4, Defender/Judge 3
    Set in Implementation; maximum adaptive query/iteration counts per agent; chosen to bound cost.
  • adaptive queries per search call = 3
    Set in Implementation; each SearchAgent call uses an initial plan plus at most three adaptive queries.
assumptions (4)
  • domain assumption Simulated evidence corpora approximate real audit engagements.
    Benchmark uses four simulated organizations with expert-crafted and synthetic artifacts because real audit evidence is sensitive, so realism is asserted rather than demonstrated.
  • domain assumption Likert ratings by a small panel of auditors are a valid measure of audit-evidence sufficiency.
    No public ground truth exists; the paper states human evaluation is the primary reference, and ratings are averaged without inter-rater reliability tests.
  • domain assumption ISO 27001 requirements can be encoded as a dependency graph with evidence profiles alpha(r).
    Task Formulation defines R, P, and alpha; this is a modeling simplification of the standard, not proven equivalent to audit practice.
  • ad hoc to paper Hand-set retrieval parameters surface all relevant evidence.
    k=15, threshold 0.35, and iteration caps are selected without sensitivity analysis, so completeness of retrieval is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IntelliAudit: Using Large Language Models to Evaluate Audit Controls." pith.science (2026). https://pith.science/paper/S23EVHMX

@misc{pith2026260807688,
  author       = {Pith},
  title        = {Pith review of: IntelliAudit: Using Large Language Models to Evaluate Audit Controls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S23EVHMX}},
  note         = {Machine review of arXiv:2608.07688}
}
read the original abstract

IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls. This judgment is difficult to automate because relevant evidence is distributed across policies, records, spreadsheets, and operational artifacts, and because audit conclusions depend on evidentiary sufficiency rather than keyword matching. We present IntelliAudit, a retrieval-grounded multi-agent system for IT audit evidence evaluation. Given a control and an evidence corpus, IntelliAudit retrieves relevant artifacts, generates an evidence-grounded assessment, challenges adverse findings, adjudicates disagreements, and produces an auditor-facing recommendation with cited evidence, rationale, missing-evidence analysis, and remediation guidance. We instantiate IntelliAudit on ISO/IEC 27001 and evaluate it across multiple simulated organizations using expert auditor review and audit-readiness user feedback. The evaluation shows that IntelliAudit can support control interpretation, evidence-grounded reasoning, and audit-preparation workflows, while also revealing the importance of human oversight for calibrating sufficiency judgments and correcting overly permissive recommendations. These results suggest that retrieval-grounded multi-agent systems can assist audit evidence review, but should remain decision-support tools rather than autonomous certification systems.

Figures

Figures reproduced from arXiv: 2608.07688 by the authors.

Figure 1
Figure 1. IntelliAudit workflow for assessment, counterev￾idence review, adjudication, and human feedback. Multi-agent reasoning and document understanding. Multi-agent debate elicits and adjudicates competing inter￾pretations (Du et al. 2023), while surveys emphasize role spe￾cialization and structured coordination (Tran et al. 2025; Guo et al. 2024). D3 implements advocate, judge, and jury roles (Harrasse, Bandi, and Bandi … view at source ↗
Figure 2
Figure 2. IntelliAudit evidence-indexing and adaptive-search [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. IntelliAudit evaluation design. We report mean Likert scores, standard deviations (SD), and response counts per question (n). For RQ1 and RQ2, Group A participants assess control understanding (Un￾derst.), factuality of the evidence evaluation (Factual.), and the evidence’s fulfillment of the control (Fulfill.), reported per assessment unit in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ratings by primary evidence type (RQ3), reported [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 18 canonical work pages

  1. [1]

    M.; Zobeiri, A.; Dehghani, M.; Mohammadkhani, M.; Mohammadi, B.; Ghahroodi, O.; Baghshah, M

    Abootorabi, M. M.; Zobeiri, A.; Dehghani, M.; Mohammadkhani, M.; Mohammadi, B.; Ghahroodi, O.; Baghshah, M. S.; and Asgari, E. 2025. Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation. arXiv:2502.08826

    arXiv 2025

  2. [2]

    S.; Kaplunov, S.; Krolick, J.; and Rojkova, V

    Agarwal, B.; Jomraj, H. S.; Kaplunov, S.; Krolick, J.; and Rojkova, V. 2025. RAGulating Compliance: A Multi-Agent Knowledge Graph for Regulatory QA. arXiv:2508.09893

    arXiv 2025

  3. [3]

    AICPA . 2022. TSP Section 100: 2017 Trust Services Criteria for Security, Availability, Processing Integrity, Confidentiality, and Privacy (With Revised Points of Focus --- 2022) . American Institute of Certified Public Accountants, New York, NY. Control criteria for SOC 2 examinations

    work page 2022

  4. [4]

    I.; Abualhaija, S.; and Briand, L

    Amaral Cejas, O.; Azeem, M. I.; Abualhaija, S.; and Briand, L. C. 2023. NLP -Based Automated Compliance Checking of Data Processing Agreements Against GDPR . IEEE Transactions on Software Engineering, 49(9): 4282--4303

    work page 2023

  5. [5]

    S.; Chadha, A.; Roosta, T.; and Shah, C

    Amirizaniani, M.; Yao, J.; Lavergne, A.; Okada, E. S.; Chadha, A.; Roosta, T.; and Shah, C. 2024. LLMAuditor : A Framework for Auditing Large Language Models Using Human-in-the-Loop. arXiv:2402.09346

    arXiv 2024

  6. [6]

    Automating Security Audit Using Large Language Model based Agent: An Exploration Experiment

    Chin, J. H.; Zhang, P.; Cheong, Y. X.; and Pan, J. 2025. Automating Security Audit Using Large Language Model based Agent: An Exploration Experiment. arXiv:2505.10732

    work page Pith review arXiv 2025

  7. [7]

    Chung, J.; Ko, R.; Yoo, W.; Onizuka, M.; Kim, S.; Kim, T.-W.; and Shin, W.-Y. 2025. GraphCompliance: Aligning Policy and Context Graphs for LLM-Based Regulatory Compliance. arXiv:2510.26309

    arXiv 2025

  8. [8]

    CyberSeek . 2024. CyberSeek : Cybersecurity Supply/Demand Heat Map. https://www.cyberseek.org/. Produced by CompTIA and Lightcast in partnership with NICE. October 2024 data release (457,398 cybersecurity job postings, Sept.\ 2023--Aug.\ 2024). Accessed: 2026-07-24

    work page 2024

Show all 35 references
  1. [9]

    B.; and Mordatch, I

    Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325

    2023 arXiv

  2. [10]

    Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai, Y.; Sun, J.; Wang, M.; and Wang, H. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997

    2024 arXiv

  3. [11]

    V.; Wiest, O.; and Zhang, X

    Guo, T.; Chen, X.; Wang, Y.; Chang, R.; Pei, S.; Chawla, N. V.; Wiest, O.; and Zhang, X. 2024. Large Language Model based Multi-Agents: A Survey of Progress and Challenges. arXiv:2402.01680

    2024 arXiv

  4. [12]

    Harrasse, A.; Bandi, C.; and Bandi, H. 2026. Debate, Deliberate, Decide ( D3 ): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Vol...

    2026

  5. [13]

    Hillebrand, L.; Berger, A.; Uedelhoven, D.; Berghaus, D.; Warning, U.; Dilmaghani, T.; Kliem, B.; Schmid, T.; Loitz, R.; and Sifa, R. 2024. Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance. In 2024 IEEE International Conference on Big Data...

    2024

  6. [14]

    Huang, Y.; Lv, T.; Cui, L.; Lu, Y.; and Wei, F. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. arXiv:2204.08387

    2022 arXiv

  7. [15]

    ISC2 . 2024. ISC2 Cybersecurity Workforce Study 2024 . https://www.isc2.org/Insights/2024/10/ISC2-2024-Cybersecurity-Workforce-Study. Accessed: 2026-07-24

    2024

  8. [16]

    ISC2 . 2025. ISC2 Cybersecurity Workforce Study 2025 . https://www.isc2.org/Insights/2025/12/2025-ISC2-Cybersecurity-Workforce-Study. Accessed: 2026-07-24

    2025

  9. [17]

    ISO/IEC . 2022. ISO/IEC 27001:2022, Information Security, Cybersecurity and Privacy Protection --- Information Security Management Systems --- Requirements . International Organization for Standardization, Geneva, Switzerland. Third edition

    2022

  10. [18]

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; tau Yih, W.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Published in Advances in Neural Information Processi...

    2020 arXiv

  11. [19]

    R.; Susnjak, T.; Liu, T.; Watters, P.; Nowrozy, R.; and Halgamuge, M

    McIntosh, T. R.; Susnjak, T.; Liu, T.; Watters, P.; Nowrozy, R.; and Halgamuge, M. N. 2024. From COBIT to ISO 42001: Evaluating Cybersecurity Frameworks for Opportunities, Risks, and Regulatory Compliance in Commercializing Large Language Models. Computers & Security, 144: 103...

    2024 arXiv

  12. [20]

    National Institute of Standards and Technology . 2024. The NIST Cybersecurity Framework (CSF) 2.0 . NIST Cybersecurity White Paper (CSWP) 29, Gaithersburg, MD. DOI: 10.6028/NIST.CSWP.29

    2024 doi

  13. [21]

    OpenAI . 2024. New Embedding Models and API Updates . https://openai.com/index/new-embedding-models-and-api-updates/

    2024

  14. [22]

    PCI Security Standards Council . 2024. Payment Card Industry Data Security Standard: Requirements and Testing Procedures, Version 4.0.1 . PCI Security Standards Council, Wakefield, MA

    2024

  15. [23]

    T.; King, N.; Nori, H.; and Amershi, S

    Rastogi, C.; Ribeiro, M. T.; King, N.; Nori, H.; and Amershi, S. 2023. Supporting Human- AI Collaboration in Auditing LLM s with LLM s. arXiv:2304.09991

    2023 arXiv

  16. [24]

    I.; and Brachten, C

    Riasat, S. I.; and Brachten, C. E. W. 2024. Automating ISO 27001 Consultancy and Auditing using AI : A Feasibility Study Based on Expert Interviews. Preprint, Utrecht University, supervised by R. L. Jansen

    2024

  17. [25]

    T.; and Vasilakos, A

    Singh, A.; Ehtesham, A.; Kumar, S.; Khoei, T. T.; and Vasilakos, A. V. 2025. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. arXiv:2501.09136

    2025 arXiv

  18. [26]

    Tran, K.-T.; Dao, D.; Nguyen, M.-D.; Pham, Q.-V.; O'Sullivan, B.; and Nguyen, H. D. 2025. Multi-Agent Collaboration Mechanisms: A Survey of LLMs. arXiv:2501.06322

    2025 arXiv

  19. [27]

    G.; and Ramanathan, M

    Waikar, S.; Bhat, A. G.; and Ramanathan, M. 2026. Retrieval Augmented Generation ( RAG ) for Evaluating Regulatory Compliance of Drug Information and Clinical Trial Protocols. CPT: Pharmacometrics & Systems Pharmacology, 15(3): e70201

    2026

  20. [28]

    Wang, D.; Raman, N.; Sibue, M.; Ma, Z.; Babkin, P.; Kaur, S.; Pei, Y.; Nourbakhsh, A.; and Liu, X. 2024. DocLLM : A Layout-Aware Generative Language Model for Multimodal Document Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...

    2024

  21. [29]

    Wang, R.; Liu, J.; Zhao, W.; Li, S.; and Zhang, D. 2025 a . Automating Financial Statement Audits with Large Language Models. arXiv:2506.17282

    2025 arXiv

  22. [30]

    Wang, Y.; Ai, X.; Patel, J.; Peng, X.; Mo, F.; Cao, Y.; Li, H.; Cao, M.; Qian, L.; and Guti\'errez-Basulto, V. 2026. AuditFlow : Executable Symbolic Environments for Structured Financial Reporting Verification. arXiv:2606.03031

    2026 arXiv

  23. [31]

    Wang, Y.; Wang, K.; Yang, S.; Patel, J.; Zhao, J.; Mo, F.; Peng, X.; Qian, L.; Chen, Y.; Guti\'errez-Basulto, V.; Huang, J.; Xiong, G.; Liu, X.-Y.; Liu, X.; and Nie, J.-Y. 2025 b . FinAuditing : A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLM s. Acc...

    2025 arXiv

  24. [32]

    Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020. LayoutLM: Pre-training of Text and Layout for Document Image Understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, 1192–1200. ACM

    2020

  25. [33]

    Xu, Y.; Xu, Y.; Lv, T.; Cui, L.; Wei, F.; Wang, G.; Lu, Y.; Florencio, D.; Zhang, C.; Che, W.; Zhang, M.; and Zhou, L. 2021. LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding. Published at ACL-IJCNLP 2021, arXiv:2012.14740

    2021 arXiv

  26. [34]

    Yao, X.; Wu, X.; Li, X.; Xu, H.; Li, C.; Huang, P.; Li, S.; Ma, X.; and Shan, J. 2024. Smart Audit System Empowered by LLM . arXiv:2410.07677

    2024 arXiv

  27. [35]

    Zou, H. P.; Huang, W.-C.; Wu, Y.; Guo, J.; Chen, Y.; Miao, C.; Nguyen, H.; Zhou, Y.; Zhang, W.; Fang, L.; Zhang, H.; Wang, F.; Zhang, P.; Wang, H.; He, L.; Li, Y.; Li, D.; Jiang, R.; Liu, X.; and Yu, P. S. 2025. LLM-Based Human-Agent Collaboration and Interaction Systems: A Su...

    2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.