Pith. sign in

REVIEW 5 major objections 4 minor 36 references

DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark measures LLMs in digital forensics and finds strong theory, weak practice.

desk verdict Useful three-module DFIR benchmark with real artifacts, but the headline numbers need a revision pass before they can be trusted. read the letter →

arxiv 2505.19973 v1 pith:U2RQ6T5Z submitted 2025-05-26 cs.CR cs.AI

Bilel Cherif , Tamas Bisztray , Richard A. Dubniczky , Aaesha Aldahmani , Saeed Alshehhi , Norbert Tihanyi This is my paper · ORCID
classification cs.CRcs.AI
keywords digitalforensicsincidentresponseLLMbenchmarkingTaskUnderstandingScoreNISTCFTTCTFchallengesmultiple-choiceknowledgeassessmentreliabilitymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DFIR-Metric is a three-part benchmark for testing large language models on digital forensics and incident response: 700 certification-style multiple-choice questions for knowledge, 150 CTF-style challenges for multi-step reasoning, and 500 disk-image string-search tasks from NIST's tool-testing program for hands-on analysis. The authors evaluated 14 models and found that the best ones score around 90 percent on the knowledge questions while almost never completing a full practical task. To make sense of that near-zero performance, they introduce the Task Understanding Score (TUS), which awards partial credit for completing steps of a multi-step pipeline, such as writing runnable code or identifying the right file system offset. The benchmark is meant to give practitioners a reproducible way to ask which DFIR tasks LLMs can support and where expert validation is still required.

What carries the argument

The load-bearing object is the benchmark itself, composed of three modules: Module I's 700 MCQs, Module II's 150 hand-crafted, parameter-randomized CTF templates, and Module III's 500 NIST CFTT string-search prompts that ask the model to generate a Python script and return results in a rigid JSON-like format. The new mechanism is the Task Understanding Score (TUS@m), which splits a solution into m binary criteria — for Module III, whether the model identifies the right file system offset, image path, search string, and file extension — and averages the fraction satisfied across all questions, so models that fail the whole task can still be ranked by partial progress. The evaluation also reuses reliability metrics such as Reliability Score and Confidence Index to track consistency across repeated trials.

What would settle it

Re-score the 500 NIST string-search tasks against independently hand-verified baselines; if the automated pipeline's ground truth disagrees with human experts on a substantial share of tasks, the TUS rankings collapse. Alternatively, have a panel of certified DFIR practitioners independently label the 700 MCQs; low inter-rater agreement would invalidate the knowledge module.

Watch

Extended reading notes

Core claim

The central claim is that DFIR-Metric is the first benchmark that jointly measures theoretical knowledge and practical proficiency of LLMs across the first four phases of the NIST 800-86 forensic workflow. Concretely, the authors report that GPT-4.1 reaches a Confidence Index of 89.34 percent and mean accuracy of 92.75 percent on the 700 MCQs, while on the 500 NIST string-search prompts fully correct extractions are rare (at most 5 of 500 for any model) and the best TUS@4 score is 38.52 percent partial credit; in the CTF module the best model solves only 28 percent of tasks consistently. They argue that this knowledge-practice gap, quantified through repeated trials and the new TUS metric, shows that end-to-end forensic workflows remain out of reach for current models while certification-level recall is largely mastered.

Load-bearing premise

The benchmark's ground-truth labels and scoring criteria are correct: if the expert review of the 700 MCQs or the automated pipeline that builds the 500 NIST baselines is unreliable, the model rankings and the conclusion that practical DFIR is out of reach do not follow.

Editorial extensions

If this is right

  • Current LLMs can be trusted for certification-level DFIR knowledge retrieval but not for unattended forensic analysis: a practitioner should keep a human in the loop for evidence extraction and chain-of-custody steps.
  • TUS gives a way to rank models on tasks where accuracy is near zero, so benchmark designers can track incremental improvement instead of waiting for full task success.
  • The gap between MCQ performance and practical performance suggests that knowledge-test scores alone overstate readiness for real forensic work.
  • Open-weight models trail proprietary leaders on practical tasks, but Qwen-2.5 and DeepSeek V3 are within a few points on knowledge, so the proprietary edge is not uniform.
  • The authors expect reasoning models such as o4-mini and DeepSeek R1 to score slightly better, a prediction that can be checked by running the same pipeline on them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the MCQ near-ceiling could partly reflect memorization of public certification material; a paraphrase-based retest would separate recall from understanding.
  • TUS-style partial credit transfers naturally to other code-generation benchmarks with rigid output schemas, where binary pass/fail hides incremental progress.
  • Because models in the CTF module were not allowed to run code, their low scores may understate what tool-augmented agents with a sandbox could achieve.
  • A testable extension is to build dynamic multi-evidence cases that require correlating disk, memory, and logs, the exact area where the paper's results show the largest shortfall.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces DFIR-Metric, a three-module benchmark for evaluating LLMs in digital forensics and incident response: 700 expert-reviewed MCQs, 150 CTF-style forensic challenges, and 500 NIST CFTT string-search prompts. The authors evaluate 14 LLMs, reporting high MCQ accuracy but near-zero full success on practical tasks, and introduce the Task Understanding Score (TUS) to grant partial credit in near-zero-accuracy settings. The paper concludes that current LLMs can recall DFIR knowledge but that end-to-end practical forensic workflows remain out of reach. All datasets, scripts, and results are promised on GitHub.

Significance. If the benchmark construction and scoring are sound, this is a potentially valuable contribution: it is a public, extensible DFIR benchmark covering knowledge, reasoning, and practical string-search; it evaluates 14 models; and TUS is a reasonable attempt to measure partial competence where binary accuracy is uninformative. The metric is not circular: the TUS criteria are stated in advance and are not fitted to model outputs, and the reuse of some CTF templates from the authors' earlier work does not by itself undermine the empirical claims. The significance of the central claim, however, depends on the reliability of the Module III ground-truth baselines and the TUS rubric, both of which are under-specified in the current manuscript.

major comments (5)
  1. [Table 4 and Section 3.4] Every row of Table 4 sums to 510, not the declared 500 prompts: for example, GPT-4.1 has 1+217+292+0=510 and Claude 3.7 has 5+309+195+1=510. If the dataset has 510 prompts, Eq. (4) uses the wrong denominator |Q| and the text's repeated "500" statements are wrong; if the outcome categories are not mutually exclusive, the Correct/Syntax/Wrong/T-O columns cannot be interpreted as frequency counts. This must be corrected because the TUS@4 percentages and the conclusion "no model produced meaningful results across the 500 prompts" depend directly on this table.
  2. [Section 3.4 and Section 4.3] The TUS rubric for Module III is under-specified. Eq. (4) defines TUS abstractly, and Section 4.3 lists four criteria (file-system offset, image path, search string/regex identification, artifact extension), but the manuscript never states how each r_ij is computed from an LLM-generated script, how the script's execution outcome (Correct/Syntax/Wrong/Timeout) maps onto the four criteria, or whether partial credit is awarded through code inspection, output parsing, or both. Without this operationalization, an independent reader cannot determine whether TUS measures forensic competence or the evaluator's pattern-matching, and the results are not reproducible as reported.
  3. [Section 3.3] The ground-truth baselines for Module III are not documented sufficiently. The text states that an automated pipeline parsed disk images, extracted memory blocks, parsed file systems, and recovered active and deleted files, and that the baselines were "rigorously validated by human experts," but it gives no parser version, no validation protocol, no inter-rater or spot-check statistics, and no explanation of how the required <inode>:<filename> and LIVE/DELETED output maps to the official NIST CFTT ground truth. Since every Correct and TUS determination in Table 4 is compared against these baselines, this missing documentation is load-bearing for the paper's central empirical claim.
  4. [Section 5, RQ2, and Table 4] The conclusion "no model produced meaningful results across the 500 prompts" is overstated relative to the paper's own data. Table 4 records five Correct outputs for Claude 3.7 Sonnet and one each for GPT-4.1 and Gemini 2.5 Flash. A more precise statement would be that full success was extremely rare (at most 5 of 510 or 500 prompts) and that partial-credit scores dominated the rankings. The current wording contradicts the numerical evidence and should be revised.
  5. [Tables 2-4 and Section 4.1] The evaluation lacks any measure of variance or uncertainty. Module I repeats each question three times and Module II repeats templates three times, but the reported percentages (e.g., Confidence Index and Mean Accuracy in Table 2) are point estimates with no confidence intervals, standard deviations, or per-template breakdowns, and Module III appears to be a single pass per prompt. Several top-model differences in Table 2 (e.g., GPT-4.1 vs. GPT-4o vs. Grok 3) are within a few percentage points, so without error bars or trial-level data, the ranking claims and the conclusion that proprietary models "consistently lead" are not statistically supported.
minor comments (4)
  1. [Throughout] There are several typographical errors that should be fixed: "calcualte" in Section 4.3, "preformed" in Section 4.2, "Reproducability" in the contributions list, "sting search" in Section 5, and "Cybersewcurity" in Table 1's DIA-Bench row.
  2. [Table 3] Table 3 lists only 13 models even though the text and Table 2 report 14 evaluated models; WizardLM 2 is missing. Please clarify whether it was excluded from the CTF module or whether this is a tabulation error.
  3. [Section 4.2 after Table 3] The sentence "The frequency of such errors drives every model's reliability score (range −300 to 150) into negative territory" is inconsistent with Definition 1: with 150 questions and k=3, the RS@3 formula yields a range of −100 to +50. The stated range appears to be the undivided sum rather than the RS value reported in Table 3.
  4. [Section 3.1] The claim of a "200-hour expert review" for the 700 MCQs would be more convincing with a brief protocol description: how many reviewers, whether each question was independently reviewed, and how disagreements were resolved. This is a documentation request rather than a substantive objection, but it would strengthen the benchmark's credibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark data and TUS rubric are defined a priori from external sources; self-citations are contextual, not load-bearing.

full rationale

The paper's central deliverables are a curated MCQ set (700 items from certifications and official documentation), CTF templates (150 tasks, some modified from the authors' prior DIA-Bench), and NIST CFTT string-search prompts (500) evaluated by a manually specified partial-credit rubric (TUS, Eq. 4). None of these outputs is obtained by fitting a parameter to the quantity it later 'predicts'. TUS@m is defined in Eq. (4) as an average over externally declared binary criteria (correct offset, image path, search string, extension), with no term that encodes a model's score or the headline conclusion; it is a measurement convention, not a result derived from itself. The citations to [30] provide the earlier RS/TSR/Conf metrics and the provenance of some CTF templates, but the empirical finding that LLMs score near zero on practical DFIR follows from executing generated scripts against NIST CFTT reference data, not from any theorem or fitted value imported from [30]. The only forward-looking self-citation (the expectation that reasoning models would perform slightly better based on [30]) is explicitly a speculation and does not support any headline result. The under-specified automated ground-truth pipeline and the Table 4 row sums (510 per row against a declared 500 prompts) are correctness and reproducibility defects, not circularity: they do not make an output equivalent to an input by construction. No circular step satisfying the quoted-reduction test was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central evaluation depends on several unverified assumptions about the data sources and grading, and on hand-chosen constants in the scoring metrics. There are no fitted physical parameters or invented entities; the benchmark is a measurement artifact rather than a model of a physical system.

free parameters (3)
  • TUS criterion weights (four equal weights of 1/4) = 1/4 each
    Section 3.4 manually chooses four criteria for Module III and weights them equally; rankings are sensitive to this choice and no sensitivity analysis is given.
  • Reliability Score wrong-answer penalty = -2
    Definition 1 assigns -2 for wrong answers, inherited from the authors' prior DIA-Bench work; it is a hand-selected constant that turns all CTF scores negative and affects model ordering.
  • Knowledge module trial count k = 3
    Three repetitions per question/template are used to define Confidence Index; the paper does not test robustness to k.
assumptions (4)
  • domain assumption NIST CFTT string-search data and its derived baselines are correct ground truth.
    Section 3.3 constructs Module III and the reference outputs from an automated parser over NIST CFTT images; any parser error propagates into TUS and all Module III scores.
  • ad hoc to paper The 200-hour human expert review actually happened and preserved correctness while paraphrasing.
    Section 3.1 asserts 'a 200-hour expert review' but provides no protocol, reviewer list, agreement statistics, or audit trail; all 700 MCQ labels depend on this unverifiable step.
  • domain assumption MCQ sources map to the skills 'theoretical DFIR' is meant to cover.
    Section 3.1 states questions come from certifications and official documentation; if these materials are not representative of modern DFIR practice, the knowledge benchmark's construct validity fails.
  • domain assumption The four NIST 800-86 phases selected map onto the three modules.
    The Introduction says the benchmark covers the first four stages of NIST SP 800-86; this mapping justifies comprehensiveness, but no task-level mapping table is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response." pith.science (2026). https://pith.science/paper/U2RQ6T5Z

@misc{pith2026250519973,
  author       = {Pith},
  title        = {Pith review of: DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2RQ6T5Z}},
  note         = {Machine review of arXiv:2505.19973}
}
read the original abstract

Digital Forensics and Incident Response (DFIR) involves analyzing digital evidence to support legal investigations. Large Language Models (LLMs) offer new opportunities in DFIR tasks such as log analysis and memory forensics, but their susceptibility to errors and hallucinations raises concerns in high-stakes contexts. Despite growing interest, there is no comprehensive benchmark to evaluate LLMs across both theoretical and practical DFIR domains. To address this gap, we present DFIR-Metric, a benchmark with three components: (1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice questions sourced from industry-standard certifications and official documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500 disk and memory forensics cases from the NIST Computer Forensics Tool Testing Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their accuracy and consistency across trials. We also introduce a new metric, the Task Understanding Score (TUS), designed to more effectively evaluate models in scenarios where they achieve near-zero accuracy. This benchmark offers a rigorous, reproducible foundation for advancing AI in digital forensics. All scripts, artifacts, and results are available on the project website at https://github.com/DFIR-Metric.

Figures

Figures reproduced from arXiv: 2505.19973 by the authors.

Figure 1
Figure 1. DFIR-Metric evaluation framework, consisting of three modules. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    In: Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track (2024)

    Alam, M.T., Bhusal, D., Nguyen, L., Rastogi, N.: Ctibench: A benchmark for evaluating llms in cyber threat intelligence. In: Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track (2024)

    work page 2024

  2. [2]

    Barrington,S.,Bohacek,M.,Farid,H.:TheDeepSpeakDataset(Apr2025).https: //doi.org/10.48550/arXiv.2408.05366, arXiv:2408.05366 [cs] version: 3

    arXiv doi:10.48550/arxiv.2408.05366

  3. [3]

    In: Proceedings of the 8th International Con- ference on Information Systems Security and Privacy

    Carrier, T., Victor, P., Tekeoglu, A., Lashkari, A.: Detecting obfuscated malware using memory feature engineering. In: Proceedings of the 8th International Con- ference on Information Systems Security and Privacy. SCITEPRESS - Science and Technology Publications (2022) DFIR-Metric: A Benchmark Dataset for Evaluating LLMs in DFIR 13

    work page 2022

  4. [4]

    Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M.B., Ai, X., Lai, P.T., Wang, Z., Keloth, V.K., Raja, K., Huang, J., He, H., Lin, F., Du, J., Zhang, R., Zheng, W.J., Adelman, R.A., Lu, Z., Xu, H.: Benchmarking large language models for biomedical natural language processing applications and recommenda- tions. Nat. Commun.16(1), 3280 (Apr 2025)

    work page 2025

  5. [5]

    Dang-Nguyen, D.T., Pasquini, C., Conotter, V., Boato, G.: RAISE: a raw images datasetfordigitalimageforensics.In:Proceedingsofthe6thACMMultimediaSys- tems Conference. pp. 219–224. MMSys ’15, Association for Computing Machinery, New York, NY, USA (Mar 2015).https://doi.org/10.1145/2713168.2713194

    work page arXiv 2015

  6. [6]

    In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

    Fei, Z., Shen, X., Zhu, D., Zhou, F., Han, Z., Huang, A., Zhang, S., Chen, K., Yin, Z., Shen, Z., Ge, J., Ng, V.: LawBench: Benchmarking legal knowledge of large language models. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 7933–7962. Association for Computati...

    doi:10.18653/v1/2024.emnlp-main.452 2024

  7. [7]

    Internet of Things and Cyber-Physical Systems5, 1–46 (2025).https://doi.org/10.1016/j.iotcps.2025.01.001

    Ferrag, M.A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisz- tray, T., Debbah, M.: Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities. Internet of Things and Cyber-Physical Systems5, 1–46 (2025).https://doi.org/10.1016/j.iotcps.2025.01.001

    doi:10.1016/j.iotcps.2025.01.001 2025

  8. [8]

    IEEE Access12, 23733–23750 (2024).https://doi.org/10.1109/ACCESS

    Ferrag, M.A., Ndhlovu, M., Tihanyi, N., Cordeiro, L.C., Debbah, M., Lestable, T., Thandi, N.S.: Revolutionizing Cyber Threat Detection With Large Language Models: A Privacy-Preserving BERT-Based Lightweight Model for IoT/IIoT De- vices. IEEE Access12, 23733–23750 (2024).https://doi.org/10.1109/ACCESS. 2024.3363469

    arXiv 2024

Show all 36 references
  1. [9]

    Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Ols- son, C.F., Denain, J.S., Ho, A., Santos, E.d.O., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., Grechuk, ...

    2024 doi arXiv

  2. [10]

    Forensic Science International: Digital Investigation38, 301264 (Sep 2021).https: //doi.org/10.1016/j.fsidi.2021.301264

    Horsman, G., Lyle, J.R.: Dataset construction challenges for digital forensics. Forensic Science International: Digital Investigation38, 301264 (Sep 2021).https: //doi.org/10.1016/j.fsidi.2021.301264

    2021

  3. [11]

    Packt Publishing, Birm- ingham, England, 2 edn

    Johansen, G.: Digital Forensics and Incident Response. Packt Publishing, Birm- ingham, England, 2 edn. (Jan 2020)

    2020

  4. [12]

    Joyce, R.J., Patel, T., Nicholas, C., Raff, E.: AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora. In: Proceedings of the 16thACMWorkshoponArtificialIntelligenceandSecurity.pp.185–196.AISec’23, Association for Computing Machinery, New Yor...

    2023

  5. [13]

    Kent, K., Chevalier, S., Grance, T., Dang, H.: Guide to Integrating Forensic Tech- niques into Incident Response. Tech. Rep. NIST Special Publication (SP) 800-86, National Institute of Standards and Technology (Sep 2006).https://doi.org/10. 6028/NIST.SP.800-86

    2006

  6. [14]

    IEEE Networking Letters4(3), 162–166 (Sep 2022).https://doi.org/10.1109/ LNET.2022.3185553 14 B

    Liu, J., Shen, Y., Simsek, M., Kantarci, B., Mouftah, H.T., Bagheri, M., Djukic, P.: A New Realistic Benchmark for Advanced Persistent Threats in Network Traffic. IEEE Networking Letters4(3), 162–166 (Sep 2022).https://doi.org/10.1109/ LNET.2022.3185553 14 B. Cherif et al

    2022

  7. [15]

    In: GLOBECOM 2022 - 2022 IEEE Global Communications Conference

    Liu, J., Simsek, M., Kantarci, B., Bagheri, M., Djukic, P.: Collaborative Feature Maps of Networks and Hosts for AI-driven Intrusion Detection. In: GLOBECOM 2022 - 2022 IEEE Global Communications Conference. pp. 2662–2667 (Dec 2022). https://doi.org/10.1109/GLOBECOM48099.2022....

    2022

  8. [16]

    Computers14(2), 67(Feb2025).https://doi.org/10.3390/computers14020067, number: 2 Publisher: Multidisciplinary Digital Publishing Institute

    Loumachi, F.Y., Ghanem, M.C., Ferrag, M.A.: Advancing Cyber Incident Timeline Analysis Through Retrieval-Augmented Generation and Large Language Models. Computers14(2), 67(Feb2025).https://doi.org/10.3390/computers14020067, number: 2 Publisher: Multidisciplinary Digital Publis...

    doi

  9. [18]

    Forensic Science International: Digital Investigation48, 301683 (Mar 2024)

    Michelet, G., Breitinger, F.: ChatGPT, Llama, can you write my report? An exper- iment on assisted digital forensics reports written using (local) large language mod- els. Forensic Science International: Digital Investigation48, 301683 (Mar 2024). https://doi.org/10.1016/j.fsi...

    2024

  10. [19]

    Ad Hoc Networks174, 103840 (Jul 2025)

    Mohamed, H., Koroniotis, N., Schiliro, F., Moustafa, N.: IoT-CAD: A compre- hensive Digital Forensics dataset for AI-based Cyberattack Attribution Detec- tion methods in IoT environments. Ad Hoc Networks174, 103840 (Jul 2025). https://doi.org/10.1016/j.adhoc.2025.103840

    2025

  11. [20]

    Computer Networks227, 109688 (May 2023).https://doi.org/10.1016/j.comnet.2023

    Myneni, S., Jha, K., Sabur, A., Agrawal, G., Deng, Y., Chowdhary, A., Huang, D.: Unraveled — A semi-synthetic dataset for Advanced Persistent Threats. Computer Networks227, 109688 (May 2023).https://doi.org/10.1016/j.comnet.2023. 109688

    2023 doi

  12. [21]

    In: 2024 5th International Conference in Electronic Engineering, Information Technology & Education (EEITE)

    Nikolakopoulos, A., Evangelatos, S., Veroni, E., Chasapas, K., Gousetis, N., Apos- tolaras, A., Nikolopoulos, C.D., Korakis, T.: Large language models in modern forensic investigations: Harnessing the power of generative artificial intelligence in crime resolution and suspect ...

    2024

  13. [22]

    IEEE Software40(3), 4–8 (2023).https: //doi.org/10.1109/MS.2023.3248401

    Ozkaya, I.: Application of large language models to software engineering tasks: Opportunities, risks, and implications. IEEE Software40(3), 4–8 (2023).https: //doi.org/10.1109/MS.2023.3248401

    2023

  14. [23]

    In: Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Processing

    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Processing. Association for Computational Linguistics, Stroudsburg, PA, USA (2016)

    2016

  15. [24]

    Forensic Science International: Digital Investigation46, 301609 (Oct 2023).https://doi

    Scanlon, M., Breitinger, F., Hargreaves, C., Hilgert, J.N., Sheppard, J.: ChatGPT for digital forensic investigation: The good, the bad, and the unknown. Forensic Science International: Digital Investigation46, 301609 (Oct 2023).https://doi. org/10.1016/j.fsidi.2023.301609

    2023

  16. [26]

    Forensic Science International: DFIR-Metric: A Benchmark Dataset for Evaluating LLMs in DFIR 15 Digital Investigation52, 301872 (Mar 2025).https://doi.org/10.1016/j.fsidi

    Sharma, B., Ghawaly, J., McCleary, K., Webb, A.M., Baggili, I.: ForensicLLM: A local large language model for digital forensics. Forensic Science International: DFIR-Metric: A Benchmark Dataset for Evaluating LLMs in DFIR 15 Digital Investigation52, 301872 (Mar 2025).https://d...

    2025

  17. [27]

    EURASIP Journal on Information Security 2017(1), 15 (Oct 2017).https://doi.org/10.1186/s13635-017-0067-2

    Shullani, D., Fontani, M., Iuliani, M., Shaya, O.A., Piva, A.: VISION: a video and image dataset for source identification. EURASIP Journal on Information Security 2017(1), 15 (Oct 2017).https://doi.org/10.1186/s13635-017-0067-2

    2017 doi

  18. [28]

    Computers and Electrical Engineering124, 110307 (2025).https://doi

    Sood, A.K., Zeadally, S., Hong, E.: The paradigm of hallucinations in ai-driven cybersecurity systems: Understanding taxonomy, classification outcomes, and mit- igations. Computers and Electrical Engineering124, 110307 (2025).https://doi. org/https://doi.org/10.1016/j.compelec...

    2025

  19. [29]

    https://doi.org/10.48550/arXiv.2505.03100

    Studiawan, H., Breitinger, F., Scanlon, M.: Towards a standardized methodology anddatasetforevaluatingLLM-baseddigitalforensictimelineanalysis(May2025). https://doi.org/10.48550/arXiv.2505.03100

    doi

  20. [30]

    In: 2024 IEEE Interna- tional Conference on Big Data (BigData)

    Tihanyi, N., Bisztray, T., Dubniczky, R.A., Toth, R., Borsos, B., Cherif, B., Jain, R., Muzsai, L., Ferrag, M.A., Marinelli, R., Cordeiro, L.C., Debbah, M., Mavroei- dis, V., Jøsang, A.: Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Mode...

    2024

  21. [31]

    In: 2024 IEEE International Conference on Cyber Security and Resilience (CSR)

    Tihanyi, N., Ferrag, M.A., Jain, R., Bisztray, T., Debbah, M.: CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge. In: 2024 IEEE International Conference on Cyber Security and Resilience (CSR). pp. 296–302 (S...

    2024

  22. [32]

    In: Ideas That Cre- ated the Future, pp

    Turing, A.M.: Computing machinery and intelligence (1950). In: Ideas That Cre- ated the Future, pp. 147–164. The MIT Press (Feb 2021)

    1950

  23. [33]

    In: Proceedings of the 31st International Conference on Neural Information Processing Systems

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6000–6010. NIPS’17, Curran Associates Inc., Red...

    2017

  24. [34]

    In: Linzen, T., Chrupała, G., Alishahi, A

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: A multi- task benchmark and analysis platform for natural language understanding. In: Linzen, T., Chrupała, G., Alishahi, A. (eds.) Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpr...

    2018 doi

  25. [35]

    In: Proceedings of the 31st Inter- national Conference on Computational Linguistics

    Wang, S., Long, Z., Fan, Z., Huang, X., Wei, Z.: Benchmark self-evolving: A multi- agent framework for dynamic LLM evaluation. In: Proceedings of the 31st Inter- national Conference on Computational Linguistics. pp. 3310–3328. Association for Computational Linguistics, Abu Dha...

    2025

  26. [36]

    In: Proceedings of the Digital Forensics Doctoral Sym- posium

    Wickramasekara, A., Densmore, A., Breitinger, F., Studiawan, H., Scanlon, M.: AutoDFBench: A Framework for AI Generated Digital Forensic Code and Tool Testing and Evaluation. In: Proceedings of the Digital Forensics Doctoral Sym- posium. pp. 1–7. ACM, Brno Czech Republic (Apr ...

    2025

  27. [37]

    In: 2024 12th International Sympo- sium on Digital Forensics and Security (ISDFS)

    Wickramasekara, A., Scanlon, M.: A Framework for Integrated Digital Forensic Investigation Employing AutoGen AI Agents. In: 2024 12th International Sympo- sium on Digital Forensics and Security (ISDFS). pp. 01–06. IEEE, San Antonio, TX, USA (Apr 2024).https://doi.org/10.1109/I...

    2024

  28. [38]

    Yin, Z., Wang, Z., Xu, W., Zhuang, J., Mozumder, P., Smith, A., Zhang, W.: Digital forensics in the age of large language models (2025),https://arxiv.org/ abs/2504.02963

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.