Pith. sign in

REVIEW 3 major objections 5 minor 26 references

System Calls for Malware Detection and Classification: Methodologies and Applications

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This chapter argues that system calls and their arguments form a behavioral fingerprint that machine-learning models can use to detect and classify malware, with reported accuracies around 99.5% for CNN classification and 99.7% for GNN…

desk verdict Useful survey, but the headline 99.5% CNN accuracy is a training number—needs correction before it can be trusted. read the letter →

arxiv 2506.01412 v1 pith:X7K5KITD submitted 2025-06-02 cs.CR cs.AIcs.LG

Bishwajit Prasad Gond , Durga Prasad Mohapatra This is my paper
classification cs.CRcs.AIcs.LG
keywords malwaredetectionsystemcallsAPIcallsequencesbehavior-basedanalysisdynamicsandboxingmachinelearningclassifiersdeepforsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Malware must ask the operating system to act, and this chapter argues that the resulting system call and API call traces are one of the most revealing behavioral signals available to defenders. It surveys how those traces are collected—statically from binaries, dynamically during execution, and inside sandboxes—and then shows four use cases in which the calls, paired with their arguments, are turned into features for machine-learning classifiers. The reported results are very high: a CNN reaches 99.5% classification accuracy, a GNN reaches 99.7% detection accuracy with a 99.59% Matthews correlation coefficient, and an image-based CNN averages 98.36%. The sympathetic reader should take away that behavior-based detection built on system calls is credible and worth pursuing, because even packed or obfuscated malware still has to invoke the kernel to do harm.

What carries the argument

The central object is the argument-aware system call or API call trace: each observed call is paired with its arguments (e.g., the DLL name loaded by LdrLoadDll, the file path passed to NtCreateFile) and converted into one of four feature representations. The first is n-grams—a sliding window of $n$ consecutive tokens—vectorized with TF-IDF and compared by Jaccard similarity, $J(A,B)=|A\cap B|/|A\cup B|$; the second is the same n-gram representation reduced by term-frequency feature selection and fed to deep networks; the third is a graph built from Levenshtein-distance embeddings of API calls and analyzed by graph convolution layers; the fourth is a grayscale image rendered from normalized API-call arguments. These representations do the work of turning raw behavioral logs into inputs that classifiers can separate into benign and malicious classes.

What would settle it

Re-run one of the four pipelines on newly collected malware with a time-based or family-aware train/test split and compare the resulting accuracy with the reported 99.5% and 99.7%; a material drop, or the discovery of near-duplicate samples shared between the original train and test sets, would undercut the claim.

Watch

Extended reading notes

Core claim

The central claim is that system calls provide a low-level view of a program's interactions with the operating system kernel, revealing behaviors that are critical for detecting malicious activity. Because system calls are universal across Windows, Linux, and Android and are hard to hide, the chapter treats them as a defensible foundation for malware analysis. The four use cases all combine API names with their arguments: n-grams scored by TF-IDF and compared with Jaccard similarity; n-gram features fed to ANN, CNN, and RNN classifiers; Levenshtein-distance embeddings assembled into graphs and classified by a GNN; and API-call arguments rendered as grayscale images for a CNN. The best reported numbers are CNN classification accuracy of 99.5%, GNN detection accuracy of 99.70% with an F1-score of 99.83%, and image-CNN average accuracy of 98.36%.

Load-bearing premise

The chapter's case rests on accepting the reported accuracies (99.5% CNN, 99.7% GNN) as fair, generalizable measurements; it reports no independent replication or leakage check to confirm those numbers would hold on fresh data.

Editorial extensions

If this is right

  • Behavioral detectors built on system call traces should continue to catch packed or obfuscated malware that signature scanners miss, because malicious code still has to call the kernel to act.
  • Including API arguments in the feature representation should add discriminating power over using API names alone.
  • Graph representations of API call embeddings can plausibly support near-real-time scanning, since the reported GNN runs in about 20 ms per sample.
  • Rendering API call arguments as grayscale images is a viable route to high-accuracy classification, with the reported average accuracy of 98.36%.
  • The next steps the chapter proposes—hybrid static-dynamic models and LLM-based classifiers tuned on API call sequences—build directly on these use cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the headline metrics come from the authors' own earlier papers using samples from a single malware repository; an independent replication on fresh, temporally separated samples is needed before treating them as established performance levels.
  • Editorial inference: the chapter asserts that appending API arguments to API names improves discrimination but reports no ablation that removes arguments; such an ablation on any of the four use cases would directly test that design choice.
  • Editorial inference: the GNN's reported advantage is tied to Levenshtein-distance embedding; comparing that embedding with a frequency-based embedding on the same graph construction would isolate what the distance metric adds.
  • Editorial inference: the chapter's proposed LLM-based classifiers can be benchmarked directly against the reported CNN and GNN numbers on the same datasets, which would show whether larger models move the accuracy ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This chapter surveys the use of system calls and API calls for malware detection and classification. It covers background on malware types, OS-specific system call interfaces (Windows, Linux, Android), static and dynamic analysis, sandboxing tools, and four application use cases: n-gram feature vectors with Jaccard similarity, deep learning with ANN/CNN/RNN on n-gram features, graph neural networks on API call embeddings, and CNN-based image conversion of API call arguments. The chapter claims that system call analysis is vital for malware detection and that modern machine learning models achieve very high accuracy, specifically 99.5% CNN classification accuracy and 99.70% GNN detection accuracy, and it concludes with future directions such as hybrid models and LLM-based classifiers.

Significance. The survey provides a readable overview of system-call-based malware detection, organizing tools and techniques by platform and analysis type. It also assembles several concrete use cases with performance tables, which could be useful for practitioners entering the field. However, the headline performance claims are not presented in a way that supports the chapter's central assertion that these methods generalize well. The most important reported numbers are drawn from the authors' own prior work, and one of them (the 99.5% CNN accuracy) is demonstrably the training accuracy rather than held-out validation accuracy. If these numbers are corrected and properly contextualized, the chapter would be a useful survey; as written, the evidence base for its central quantitative claims is internally inconsistent and partially unverifiable.

major comments (3)
  1. [Section 4.2, Table 3] The text above Table 3 states that the CNN 'achieves a validation accuracy of 98.53% and a loss of 0.0161 at 100 epochs,' but Table 3 lists at epoch 100: Accuracy = 0.9950, Val Accuracy = 0.9853, Loss = 0.0161, Val Loss = 0.1846. Therefore, the value 0.0161 is the training loss, not the validation loss, and the value 99.5% is the training accuracy, not the held-out accuracy. The sentence 'achieving a classification accuracy of 99.5% for the CNN model as shown in Table 3' misattributes a training-set number to the model's classification performance. Section 5 repeats 'CNNs have reached classification accuracies as high as 99.5%' as if this were a held-out result. Since the chapter's positive conclusion about deep learning rests on this number, this is a load-bearing error that should be corrected by reporting validation/test accuracy and by explicitly stating whether any independent test set was held out after model selection.
  2. [Section 4.3, Table 4] Table 4 reports a GNN detection accuracy of 99.70%, an F1-score of 99.83%, and an MCC of 99.59%, but the text gives no information about how the 41,483-sample dataset was split (e.g., train/validation/test proportions or cross-validation) or whether these metrics were computed on a held-out test set. The sole source cited is the authors' prior paper [3]. Without split details, these numbers cannot be evaluated for generalization, and the chapter's claim that GNNs 'have pushed detection rates to around 99.7%' is not supported by the evidence presented. Please add the dataset split protocol and clarify whether the reported numbers are test-set metrics or training/validation metrics.
  3. [Sections 4.1-4.4 and References] The chapter's four use cases rely heavily on the authors' own prior publications: references [1], [3], [21], [25], and [26] are all by the authors, and the dataset in reference [20] was published by the first author on Kaggle. While self-citation is not inherently inappropriate, the survey does not critically examine these results or compare them with independent studies, and it presents them as representative evidence for the field. This self-referential chain weakens the survey's impartiality. The authors should either add independent corroborating studies or explicitly frame Sections 4.1-4.4 as a summary of their own prior work, with a discussion of how much independent replication exists.
minor comments (5)
  1. [Abstract and Introduction] There are numerous grammatical errors and nonstandard phrasings, such as 'an useful tool,' 'a deep down look,' and 'while also looking at the ways sophisticated malware tries to evade detection.' These should be corrected throughout.
  2. [Section 2.3.1] The list of 'Hooked APIs and Categories' includes a category labeled 'Notification' whose content reads 'missing, exception , Process , Anomaly etc,' which is incomplete or garbled. This should be fixed.
  3. [Section 3.1] The description of the .idata section as 'the sixth part, placed above the .text section and below the .esrc section' is confusing and likely incorrect; the .idata section is typically part of the optional header's data directories, and the ordering of PE sections is not fixed as implied.
  4. [Section 4.4] Table 5 reports per-class accuracies that appear to be class-wise classification rates rather than overall accuracy; the text also says the model achieves 'up to 100%' and 'an average classification accuracy of 98.36%,' which are not clearly reconciled. Please clarify the metric definitions.
  5. [Section 5] The conclusion contains typos such as 'explainbility,' 'classifers,' and 'APi Call,' and the future-work sentence about LLM-based classifiers is incomplete. These should be edited.

Circularity Check

1 steps flagged · score 6.0 of 10

The chapter's flagship 99.5% CNN accuracy is the training Accuracy column of Table 3, not a held-out result; Section 5 presents it as an established predictive performance.

  1. fitted input called prediction [Section 4.2, paragraph after Table 3; Table 3 CNN row; echoed in Section 5]
    "The use of API arguments in n-grams enhances the model’s ability to differentiate malware types, achieving a classification accuracy of 99.5% for the CNN model as shown in Table 3."

    Table 3's own columns list the CNN at epoch 100 as Loss=0.0161, Val Loss=0.1846, Accuracy=0.9950, Val Accuracy=0.9853. The text first attributes 98.53% and 0.0161 to validation, then calls 99.5% the model's 'classification accuracy.' Because 99.5% sits in the Accuracy (training) column and no test-set accuracy is reported, the claimed predictive accuracy reduces by construction to the model's fit on its training data. Section 5 repeats the figure as an established result, so the paper's headline performance claim is an in-sample metric presented as a generalizable prediction.

full rationale

This is a survey chapter rather than a formal derivation, and its core thesis that system calls provide a valuable low-level view of program behavior is standard, widely shared, and independently grounded in the broader malware-analysis literature. That part is not circular. The circularity is concentrated in the quantitative performance claims. In Section 4.2, Table 3 separates Accuracy from Val Accuracy; the CNN row at epoch 100 shows Accuracy 0.9950 and Val Accuracy 0.9853. The surrounding text nevertheless reports a 'classification accuracy of 99.5%' and even conflates the training loss 0.0161 with validation loss. Section 5 then restates 'CNNs have reached classification accuracies as high as 99.5%' as if it were a held-out result. This is a fitted-input-called-prediction issue: the headline number is the training accuracy by the table's own column labels, so the conclusion partially reduces to the training fit. Separately, the 99.70% GNN detection figure in Sections 4.3 and 5 is imported from the authors' own prior paper [3]. That is a self-citation and a real evidence-quality weakness, but it is a published, externally falsifiable result on public data rather than a by-construction reduction, so I do not count it as circular by itself. Overall, the central qualitative claim has independent content, but one of the two flagship quantitative predictions reduces to an in-sample metric, giving a partial circularity score of 6.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The chapter is a review and introduces no free parameters, new entities, or original derivations. Its load-bearing assumptions are the reliability of the reported results and the representativeness of sandbox-derived system call traces.

assumptions (2)
  • domain assumption The experimental results cited from prior papers (e.g., CNN 99.5%, GNN 99.7%) are accurate and were obtained under the stated conditions.
    The chapter presents no new experiments; its conclusions about method effectiveness rest on these numbers.
  • domain assumption System call and API call traces captured in sandboxed environments are representative of real-world malware behavior.
    Section 3.3 assumes sandbox monitoring faithfully records behavior without substantial evasion or environment-specific artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of System Calls for Malware Detection and Classification: Methodologies and Applications." pith.science (2026). https://pith.science/paper/X7K5KITD

@misc{pith2026250601412,
  author       = {Pith},
  title        = {Pith review of: System Calls for Malware Detection and Classification: Methodologies and Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7K5KITD}},
  note         = {Machine review of arXiv:2506.01412}
}
read the original abstract

As malware continues to become more complex and harder to detect, Malware Analysis needs to continue to evolve to stay one step ahead. One promising key area approach focuses on using system calls and API Calls, the core communication between user applications and the operating system and their kernels. These calls provide valuable insight into how software or programs behaves, making them an useful tool for spotting suspicious or harmful activity of programs and software. This chapter takes a deep down look at how system calls are used in malware detection and classification, covering techniques like static and dynamic analysis, as well as sandboxing. By combining these methods with advanced techniques like machine learning, statistical analysis, and anomaly detection, researchers can analyze system call patterns to tell the difference between normal and malicious behavior. The chapter also explores how these techniques are applied across different systems, including Windows, Linux, and Android, while also looking at the ways sophisticated malware tries to evade detection.

Figures

Figures reproduced from arXiv: 2506.01412 by the authors.

Figure 1
Figure 1. PE and APK File Architecture DOS Header The DOS header is the first part of the PE file. It’s like an old piece from the MS-DOS days. It helps the file work on older systems and points to the newer PE header. PE Header The PE header comes next. It’s the main header that tells the computer how to read the file. It has important details like the file type and where other sections start. Optional Header The optional he… view at source ↗
Figure 2
Figure 2. Cuckoo Sandbox Analysis and feature engineering [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. JSON Report Feature Selection after Cuckoo Analysis [ [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Feature creation from API Sequence [21] Measuring Text Similarity with Jaccard Similarity and TF-IDF Preprocess the Data We start by preprocessing the input data and reference data, which are in the form of data frames. We convert each data frame into a single text str…
Figure 5
Figure 5. Figure 5: Similarity Based Approach [1] 11 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Malware Classification using NLP and Jaccard Similarity [ [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Confusion matrices for malware classification [ [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Malware Classification using Deep Learning Techniques [ [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Malware Classification using Graph Neural Network [ [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Malware Classification using Convolutional Neural Network [ [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Detailed Convolutional Neural Network [26] 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages

  1. [3]

    Malware Detec- tor and Classifier Using API Call Embedding and Graph Neural Networks

    Rajneekant, Pushkar Kishore, Bishwajit Prasad Gond, and Durga Prasad Mohapatra. 2024. “Malware Detec- tor and Classifier Using API Call Embedding and Graph Neural Networks.” InInternational Conference on Innovations and Advances in Cognitive Systems, 515–531. Springer

    work page 2024

  2. [1]

    NLP-Driven Malware Classification: A Jaccard Similarity Approach

    Gond, Bishwajit Prasad, Md Shahnawaz, Rajneekant, and Durga Prasad Mohapatra. 2024. “NLP-Driven Malware Classification: A Jaccard Similarity Approach.” In2024 IEEE International Conference on Information Technology, Electronics and Intelligent Communication Systems (ICITEICS), 1–8. IEEE. doi:10.1109/ICITEICS61368.2024.10624953

    work page arXiv 2024

  3. [21]

    A Deep Learn- ing Framework for Malware Classification using NLP Techniques

    Gond, Bishwajit Prasad, Atul Kumar Singh, and Durga Prasad Mohapatra. 2024. “A Deep Learn- ing Framework for Malware Classification using NLP Techniques.” In2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), 1–8. IEEE. doi:10.1109/ICCCNT61001.2024.10725427

    arXiv 2024

  4. [25]

    Deep Learning-Driven Malware Classification with API Call Sequence Analysis and Concept Drift Handling

    Gond, Bishwajit Prasad, and Durga Prasad Mohapatra. 2025. “Deep Learning-Driven Malware Classification with API Call Sequence Analysis and Concept Drift Handling.” arXiv preprint arXiv:2502.08679

    arXiv 2025

  5. [26]

    Dynamic Malware Classification of Windows PE Files using CNNs and Greyscale Images Derived from Runtime API Call Argument Conversion

    Md Shahnawaz, Bishwajit Prasad Gond, and Durga Prasad Mohapatra. 2025.Dynamic Malware Classi- fication of Windows PE Files using CNNs and Greyscale Images Derived from Runtime API Call Argu- ment Conversion. Accepted at the 16th International Conference on Computing, Communication and Net- working Technologies (ICCCNT), Indian Institute of Technology Indo...

    work page Pith review arXiv 2025

  6. [20]

    Malware Benign API Call Argument Feature Vector

    Gond, Bishwajit Prasad. 2025. “Malware Benign API Call Argument Feature Vector.” Kaggle.https: //www.kaggle.com/dsv/11600829. doi:10.34740/KAGGLE/DSV/11600829. 18

    work page arXiv 2025

  7. [2]

    VirusShare.com - Malware Repository

    VirusShare. 2025. “VirusShare.com - Malware Repository.”https://virusshare.com/. Accessed May 22, 2025

    work page 2025

  8. [4]

    Process Monitor

    Microsoft Sysinternals. 2006. “Process Monitor.”https://docs.microsoft.com/en-us/ sysinternals/downloads/procmon. A Windows tool for real-time monitoring of file system, registry, and process/thread activity, used for capturing system call sequences in malware analysis

    work page 2006

Show all 26 references
  1. [5]

    strace Team. 1991. “strace.”https://strace.io/. A Linux tool for tracing system calls and signals, used to monitor calls like open and fork for malware detection

    1991

  2. [6]

    Frida Team. 2012. “Frida.”https://frida.re/. A dynamic instrumentation toolkit for hooking and monitoring system calls and API calls on Android for malware analysis

    2012

  3. [7]

    IDA Pro

    Hex-Rays. 1991. “IDA Pro.”https://hex-rays.com/ida-pro/. A disassembler and debugger for static analysis of binaries to identify system calls and API calls in malware

    1991

  4. [8]

    Cuckoo Sandbox

    Cuckoo Sandbox Team. 2010. “Cuckoo Sandbox.”https://github.com/cuckoosandbox. An open-source sandbox for analyzing malware behavior across Windows, Linux, and Android, generating re- ports on system calls

    2010

  5. [9]

    FireEye Malware Analysis

    FireEye, Inc. 2004. “FireEye Malware Analysis.”https://www.fireeye.com/. An enterprise-grade malware analysis platform for advanced system call tracing and threat intelligence

    2004

  6. [10]

    Linux Kernel Developers. 1989. “ptrace.”https://man7.org/linux/man-pages/man2/ ptrace.2.html. A Linux system call used by tools like strace to trace and control process execution for malware detection

    1989

  7. [11]

    Xposed Framework

    Xposed Team. 2012. “Xposed Framework.”https://repo.xposed.info/. A framework for hooking and modifying Android app behavior to monitor API and system calls for malware detection

    2012

  8. [12]

    objdump

    GNU Project. 1988. “objdump.”https://www.gnu.org/software/binutils/. A Linux tool for disassembling binaries to extract system call references in static malware analysis

    1988

  9. [13]

    PE Explorer

    HeavenTools. 2001. “PE Explorer.”http://www.heaventools.com/. A tool for inspecting and edit- ing Portable Executable (PE) files to identify API calls and system call dependencies

    2001

  10. [14]

    National Security Agency. 2019. “Ghidra.”https://ghidra-sre.org/. An open-source reverse engi- neering framework for static analysis of binaries to extract system calls and API calls

    2019

  11. [15]

    APKTool

    APKTool Team. 2010. “APKTool.”https://apktool.org/. A tool for reverse engineering Android APK files to extract and analyze Dalvik bytecode containing API calls

    2010

  12. [16]

    dex2jar

    pxb1988. 2010. “dex2jar.”https://github.com/pxb1988/dex2jar. A tool for converting Android classes.dex files into JAR files for decompilation to reveal API calls

    2010

  13. [17]

    JD-GUI Team. 2008. “JD-GUI.”https://jd-gui.software.informer.com/. A graphical Java decompiler for inspecting decompiled code from dex2jar output to identify API calls in Android apps

    2008

  14. [18]

    Monnappa K A. 2016. “Limon.”https://github.com/monnappa22/Limon. A Linux-specific open-source sandbox built on strace and ptrace for lightweight analysis of Linux malware system calls

    2016

  15. [19]

    Joe Sandbox

    Joe Security LLC. 2010. “Joe Sandbox.”https://www.joesecurity.org/. A paid sandbox for Win- dows, Linux, and Android, offering detailed system call and behavioral analysis with cloud-based reporting

    2010

  16. [22]

    Malware Detection with Deep Neural Network Using Process Behavior

    Tobiyama, Shun, Yukiko Yamaguchi, Hajime Shimada, Tomonori Ikuse, and Takeshi Yagi. 2016. “Malware Detection with Deep Neural Network Using Process Behavior.” In2016 IEEE 40th Annual Computer Soft- ware and Applications Conference (COMPSAC), 2:577–582. IEEE. doi:10.1109/COMPSA...

    2016 doi

  17. [23]

    Malware Images: Visualization and Automatic Classification

    Nataraj, Lakshmanan, Sreejith Karthikeyan, Gregoire Jacob, and Bangalore S. Manjunath. 2011. “Malware Images: Visualization and Automatic Classification.” InProceedings of the 8th International Symposium on Visualization for Cyber Security, 1–7

    2011

  18. [24]

    Ko, Seok Min, JaeHyeok Yang, TaeGuen Kim, and Ilsun You. 2025. ”N-gram opcode frequency based mal- ware detection using CNN algorithm.” Soft Computing: 1-9

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.