Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read A general-purpose LLM, prompted but not fine-tuned, can summarize Android apps and flag malware with 77% accuracy, then trace its verdict down to the specific malicious function.

desk verdict Useful hierarchical summarization pipeline for Android malware triage, but the headline 77% accuracy is unsupported by the paper's own confusion matrices, which do not add up. read the letter →

arxiv 2501.04848 v1 pith:VQIQL2NS submitted 2025-01-08 cs.CR cs.AI

classification cs.CRcs.AI
keywords Androidmalwarelargelanguagemodelscodesummarizationpromptengineeringcategorizationstaticanalysisreversehierarchical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a general large language model, with no fine-tuning on malware, can serve as the core of an Android malware analysis pipeline by summarizing an app's code at function, class, and package levels and then labeling the app malicious or benign. With prompts that teach the model what malware behavior looks like, the system correctly classified 154 of 200 apps (77%), while neutral prompts managed only 49.5%. The layered summaries also allow backtracking, so an analyst can move from a package-level verdict to the exact code snippet responsible for the malicious action. If this holds, LLMs could offer fast first-pass triage and explanation for reverse engineers facing unknown or zero-day Android samples.

What carries the argument

The load-bearing mechanism is the hierarchical-tiered summarization chain, a bottom-up pipeline that compresses the large codebase of an APK into compact, security-relevant summaries at function, class, and package granularity, allowing a model with a finite context window to process an entire application. It is driven by three prompt scopes (Vanilla, API-Scoped, and Malware-Scoped) that inject progressively more security knowledge into the model; the Malware-Scoped prompts, which list suspicious behaviors such as dynamic class loading, rooting, privilege escalation, and data exfiltration, are what push accuracy from 49.5% to 77%. The chain also provides explainability, because each summary level is derived from the level below, so tracing a package-level claim back to the code recovers the root cause.

What would settle it

Take a set of APKs whose malicious code lives in classes that Dex2Jar or CFR demonstrably fail to recover (confirmed by comparing the decompiled class list against the actual dex bytecode), run MalParse's Malware-Scoped pipeline on them, and check whether the classification accuracy drops measurably below 77%; if it stays near 77%, the decompilation-preservation premise is wrong, and if it drops, the premise is confirmed as the load-bearing limitation.

Watch

Extended reading notes

Core claim

MalParse, built on GPT-4o-mini, decompiles an Android APK into Java source and runs a bottom-up hierarchical summarization chain: each function is summarized, function summaries are merged into class summaries, and class summaries are merged into a package summary. By injecting malware-specific context through what the authors call Malware-Scoped prompting, the pipeline reaches a balanced classification rate of 77%, correctly identifying 76% of benign apps and 78% of malware samples, all without any training on malware or benign examples and without revealing APK names to the model. Backtracking from the package summary through class and function summaries lets the system pinpoint the precise code snippet responsible, demonstrated on a rooting malware sample where the function using a DexClassLoader is identified. The paper presents this as evidence that prompt engineering alone can give a general LLM meaningful semantic understanding of malicious Android code.

Load-bearing premise

The whole chain assumes that turning an APK into Java source with Dex2Jar and CFR preserves enough of the app's behavior that the LLM's summaries reflect what the malware actually does.

Editorial extensions

If this is right

  • Without fine-tuning, a general LLM can achieve 77% balanced accuracy on a 100-malware/100-benign Android set, suggesting that zero-day triage may not require retraining for every new malware family.
  • Prompt context matters more than model specialization: moving from generic to malware-scoped prompts raised accuracy from 49.5% to 77%, making prompt design an effective control knob in LLM-based security analysis.
  • Analysts can use the hierarchical summaries as a navigation map, reading the package-level verdict and then drilling into classes and functions to see the specific code that triggered it, which shortens manual reverse engineering.
  • Because only the prompt carries the malware knowledge, expanding the Malware-Scoped prompt's list of suspicious behaviors could directly reduce the 22% false-negative rate, as the paper itself hypothesizes.
  • The same summarization chain could be applied to any LLM, not just GPT-4o-mini, providing a baseline to evaluate cheaper or open-source models on the same malware analysis task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 77% figure is dataset-specific and should be read as an upper bound; a deployment would need validation on a much larger and more diverse APK corpus before relying on that rate.
  • Although the paper hides APK names to prevent the model from relying on memory, the decompiled Java itself may still contain package or class identifiers that leak app identity to the LLM's pretraining, making the 'no prior knowledge' claim less clean than stated.
  • A natural extension is to run the same pipeline on obfuscated and packed malware, where static decompilation is known to degrade; the paper's own limitation section predicts accuracy will drop, turning this weakness into a testable prediction.
  • The backtracking capability could be formalized as an evidence requirement: if every package-level verdict had to cite at least one function-level snippet, then an analyst's trust in a verdict could be quantified by checking whether that snippet is actually malicious.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents MalParse, a pipeline for Android malware analysis that decompiles APKs to Java source using Dex2Jar and CFR, generates hierarchical function-, class-, and package-level summaries with GPT-4o-mini under three prompt scopes (Vanilla, API-Scoped, Malware-Scoped), and uses the final package summary to classify each app as benign or malicious. On a balanced set of 200 apps (100 VirusTotal malware, 100 top Similarweb apps), the authors report 49.5% accuracy with Vanilla prompts, 56% with API-Scoped prompts, and 77% with Malware-Scoped prompts, and illustrate that backward tracing from package-level summaries can locate the code responsible for malicious behavior (e.g., a rooting payload in the RTAccessHandler class). The paper also discusses limitations, chiefly decompilation quality, obfuscation, and prompt bias.

Significance. If the reported figures are correct after revision, the paper makes a modest but useful empirical contribution: it shows that a general-purpose LLM with carefully engineered prompts can produce human-readable, hierarchical code summaries for Android apps and perform coarse benign/malicious triage without fine-tuning, which could help accelerate triage of novel samples. The hierarchical summarization design is a pragmatic answer to LLM context-window limits, and the three-way prompt comparison is a reasonable internal ablation. The main weaknesses are the small and not fully documented dataset, the absence of conventional static-analysis baselines, and the fact that the central accuracy numbers are not internally consistent as printed.

major comments (5)
  1. [Section 4.2.1, Tables 1-3] All three confusion matrices are arithmetically impossible as printed. In Table 1, the 'Benign Actual' row sums to 185% (TP=92%, FN=93%) and the 'Malware Actual' row sums to 15% (FP=8%, TN=7%); the stated 49.5% accuracy (99/200) corresponds to TP=92 and TN=7, which forces FN=8% and FP=93%, i.e., the off-diagonal values are interchanged. In Table 2, the row sums are 168% and 32%; the stated 56% (112/200) forces FN=10% and FP=78% rather than the printed 78% and 10%. In Table 3, the row sums are 98% and 102%; the stated 77% (154/200) forces FN=24% and FP=22% rather than the printed 22% and 24%. Please provide corrected raw contingency counts for all three tables, since the headline accuracy depends directly on these numbers.
  2. [Section 4.2.1 and Introduction] The Vanilla prompt accuracy is reported as 49.5% (Introduction and §4.2.1.1), as 45% (§4.2.1.2), and as 45.5% (§4.2.1.3). These three values cannot all be correct, and the text does not explain the discrepancy. Please verify the underlying results and report a single consistent value, with raw counts.
  3. [Section 4.3.1] The statement that the system achieved 'a balanced accuracy at 77% with a precision of 76% and a recall rate of 77%' is not derivable from the corrected Table 3 counts under any standard class convention: with malware as the positive class, precision is approximately 76% but recall is 78%; with benign as positive, recall is 76% and precision is approximately 78%. Please specify the positive class and formulas used, and recompute all derived metrics so that they are mutually consistent.
  4. [Section 4.2.1, Section 3.3] The three prompt scopes are evaluated on the same 200-sample set, and the best-performing configuration (Malware-Scoped) is then reported as the system's headline accuracy. Because Section 3.3 describes iterative prompt crafting and Section 4.3.1 states that accuracy improves as prompting is tuned, the 77% figure is an in-sample optimum; no hold-out validation, cross-validation, or repeated sampling (e.g., multiple GPT-4o-mini runs with nonzero temperature) is reported. Please add a validation protocol, report variance across runs, or provide raw per-sample decisions so that the generality of the 77% number can be assessed.
  5. [Section 4.2.2] The claim that backtracking through the summaries 'allowed us to pinpoint the precise code snippets responsible for malicious behavior' (Abstract) is supported only by a single hand-picked example, the rooting sample in Table 5. Please evaluate the traceability systematically over the correctly classified malware samples (for example, how often the package-level summary leads to the correct malicious class and function), or explicitly restrict the claim to an illustrative case study.
minor comments (5)
  1. [Section 3.1] 'Davik Bytecode' should read 'Dalvik Bytecode', and 'decomples' should read 'decompiles'.
  2. [Section 4.1] The text states that 'the names of the APKs were not provided' to the model; since decompiled Java source retains package and class names, the model may still be able to infer the application identity from the code content. Please clarify whether package names were stripped or otherwise anonymized, and how the file names were excluded from the inputs.
  3. [Section 4.3.1] The acknowledged decompilation failures of Dex2Jar/CFR are not quantified; please report how many of the 200 APKs failed to decompile fully, and, if possible, separate those cases from LLM misclassifications so that the impact on the 77% figure can be understood.
  4. [Section 2] The literature review would benefit from a discussion of prior Android malware detection efforts that use machine learning or LLMs (beyond code summarization and npm malware detection), so that the novelty of the categorization and hierarchical summarization approach can be positioned more precisely.
  5. [Section 4.1 and Abstract] The paper repeatedly states that the model was 'not pre-trained on previous malware or benign samples'; this claim about GPT-4o-mini's pretraining data is not verifiable. Please rephrase to 'not fine-tuned on the evaluation dataset' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the classification claim is an empirical measurement against external ground-truth labels, not a quantity derived from the input definitions by construction.

full rationale

MalParse's central claim is an empirical classification rate obtained by comparing GPT-4o-mini's package-level summaries to externally sourced ground truth: malware samples from VirusTotal and benign samples from Similarweb's top-500 list. The three prompt variants are engineering choices, and the 77% figure is the reported result of one of the three runs (Malware-Scoped), not a parameter fitted to the labels or a quantity forced by the prompt text. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known empirical result. The paper's own limitation statement (Section 4.3.1) about Dex2Jar/CFR decompilation failures weakens external validity but is not a circular step. The apparent arithmetic inconsistencies in the confusion matrices and the post-hoc selection of the best prompt variant as 'optimized' are correctness/reproducibility and test-set-selection concerns, not circularity, because the reported numbers are empirical observations rather than reductions to input definitions.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numeric parameters appear in the design; the prompt scopes are discrete design choices, not fitted values. The four axioms listed are the key unproved premises the evaluation rests on. No new physical or conceptual entities are introduced.

assumptions (4)
  • domain assumption Dex2Jar and CFR decompilation preserves the semantic content needed for malware classification.
    Section 3.1 relies on these tools; Section 4.3.1 admits they sometimes fail to decompile complex classes, which the authors say caused malware to be labeled benign.
  • domain assumption GPT-4o-mini's generated summaries accurately reflect the code content and are not hallucinated.
    All categorization and traceability results are based on model outputs; no systematic human verification of summary correctness was performed, only one example in Section 4.2.2.
  • domain assumption VirusTotal labels for malicious apps and Similarweb top-500 selection for benign apps correctly define ground truth.
    Section 4.1 uses these sources without independent verification or discussion of label noise.
  • domain assumption The malware behavioral categories injected into the Malware-Scoped prompts are sufficient to cover the behaviors in the test set.
    The authors speculate in Section 4.2.1.3 that misclassifications may be due to suspicious activities not explicitly covered in the prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware." pith.science (2026). https://pith.science/paper/VQIQL2NS

@misc{pith2026250104848,
  author       = {Pith},
  title        = {Pith review of: Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQIQL2NS}},
  note         = {Machine review of arXiv:2501.04848}
}
read the original abstract

Malware analysis is a complex process of examining and evaluating malicious software's functionality, origin, and potential impact. This arduous process typically involves dissecting the software to understand its components, infection vector, propagation mechanism, and payload. Over the years, deep reverse engineering of malware has become increasingly tedious, mainly due to modern malicious codebases' fast evolution and sophistication. Essentially, analysts are tasked with identifying the elusive needle in the haystack within the complexities of zero-day malware, all while under tight time constraints. Thus, in this paper, we explore leveraging Large Language Models (LLMs) for semantic malware analysis to expedite the analysis of known and novel samples. Built on GPT-4o-mini model, \msp is designed to augment malware analysis for Android through a hierarchical-tiered summarization chain and strategic prompt engineering. Additionally, \msp performs malware categorization, distinguishing potential malware from benign applications, thereby saving time during the malware reverse engineering process. Despite not being fine-tuned for Android malware analysis, we demonstrate that through optimized and advanced prompt engineering \msp can achieve up to 77% classification accuracy while providing highly robust summaries at functional, class, and package levels. In addition, leveraging the backward tracing of the summaries from package to function levels allowed us to pinpoint the precise code snippets responsible for malicious behavior.

Figures

Figures reproduced from arXiv: 2501.04848 by the authors.

Figure 1
Figure 1. Diagram of the MalParse workflow outlining the hierarchical-tiered Code summarization process. It starts with function-level summarization for each method of each class in the package, followed by class-level summarization, and finishing with package-level summarization. P refers to a prompt, R refers to an LLM response, and f refers to a class method. technique for these apps. In our decompilation workflow, we crea… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

    cs.CR 2025-09 conditional novelty 6.0 of 10

    A new evaluation framework shows current LLMs are not yet reliable at fine-grained Android malware behavior auditing, with the best model scoring 50.67% on the composite workload metric.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Codebert: A pre-trained model for pro- gramming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for pro- gramming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

  2. [2]

    Graphcodebert: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366 , 2020

  3. [3]

    Uni- fied pre-training for program understanding and generation,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Uni- fied pre-training for program understanding and generation,” arXiv preprint arXiv:2103.06333 , 2021

  4. [4]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information pro- cessing systems , vol. 33, pp. 1877–1901, 2020

  5. [5]

    An analysis of the automatic bug fixing performance of chatgpt,

    D. Sobania, M. Briesch, C. Hanna, and J. Petke, “An analysis of the automatic bug fixing performance of chatgpt,” in 2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE, 2023, pp. 23–30

  6. [6]

    The potential use of chatgpt for debugging and bug fixing,

    M. A. Haque and S. Li, “The potential use of chatgpt for debugging and bug fixing,” 2023

  7. [7]

    An evaluation of the effectiveness of openai’s chatgpt for automated python program bug fixing using quixbugs,

    M. C. Wuisang, M. Kurniawan, K. A. W. Santosa, A. A. S. Gunawan, and K. E. Saputra, “An evaluation of the effectiveness of openai’s chatgpt for automated python program bug fixing using quixbugs,” in 2023 International Seminar on Application for Technology of Information and Communication (iSemantic) . IEEE, 2023, pp. 295– 300

  8. [8]

    A critical review of large language model on software engineering: An example from chatgpt and automated program repair,

    Q. Zhang, T. Zhang, J. Zhai, C. Fang, B. Y u, W. Sun, and Z. Chen, “A critical review of large language model on software engineering: An example from chatgpt and automated program repair,” arXiv preprint arXiv:2310.08879, 2023

Show all 18 references
  1. [9]

    Refining chatgpt-generated code: Charac- terizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Charac- terizing and mitigating code quality issues,” ACM Transactions on Software Engineering and Methodology , 2023

  2. [10]

    Shifting the lens: Detecting malware in npm ecosystem with large language models,

    N. Zahan, P . Burckhardt, M. Lysenko, F. Aboukhadijeh, and L. Williams, “Shifting the lens: Detecting malware in npm ecosystem with large language models,” arXiv preprint arXiv:2403.12196 , 2024

  3. [11]

    Malware analysis and detection using reverse engineering technique,

    S. Megira, A. Pangesti, and F. Wibowo, “Malware analysis and detection using reverse engineering technique,” in Journal of Physics: Conference Series , vol. 1140, no. 1. IOP Publishing, 2018, p. 012042

  4. [12]

    dex2jar,

    Panxiaobo, “dex2jar,” http://code.google.com/p/dex2jar/, available on- line

  5. [13]

    Cfr decompiler,

    L. Benfield, “Cfr decompiler,” 2020, available online. [Online]. Available: https://www.benf.org/other/cfr/

  6. [14]

    Summit: Iterative text summarization via chatGPT,

    H. Zhang, X. Liu, and J. Zhang, “Summit: Iterative text summarization via chatGPT,” in The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. [Online]. Available: https://openreview.net/forum?id=LCEbV5nsb8

  7. [15]

    Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,

    Y . Wang, Z. Zhang, and R. Wang, “Element-aware summarization with large language models: Expert-aligned evaluation and chain- of-thought method,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) . Toronto, Can...

  8. [16]

    Langchain,

    C. Harrison, “Langchain,” ver. 2, 2024. [Online]. Available: https://github.com/langchain-ai/langchain

  9. [17]

    Top 500 applications in the united states,

    Similarweb, “Top 500 applications in the united states,”

  10. [2023]

    Available: https://www.similarweb.com/top-websites/ united-states/category/application

    [Online]. Available: https://www.similarweb.com/top-websites/ united-states/category/application

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.