REVIEW 3 major objections 4 minor 31 references
The paper claims that an LLM, guided by malware family behavior labels, can localize malicious code to individual Smali methods and explain each method's role, with perfect recall on a demo app and all predictions verified on one real-world
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
MalLoc uses a two-phase LLM pipeline to localize malicious Smali methods and generate role explanations, with promising but very small-scale evidence.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A sensible two-phase LLM pipeline for method-level Android payload localization, but the evaluation leaks the answer key and rests on two apps, so the reported numbers are upper bounds. the 3 major comments →
MalLoc: Toward Fine-grained Android Malicious Payload Localization via LLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
MalLoc's discovery is that the localization problem becomes tractable for LLMs when it is decomposed along behavior and structure rather than asked in one shot. The pipeline uses a family behavior lookup table built from MalRadar's annotations to tell the model which of 12 behaviors to look for in a given sample. Phase 1 presents each Smali class together with one behavior description and asks whether the class implements it; explained malicious classes go to Phase 2, which asks the model to list every method involved and assign a role. The paper reports that this two-phase structure lifts GPT-4.1 from 0.64 method F1 in the baseline to 0.76, with recall going from 0.58 to 1.00 on MalApp, and
What carries the argument
The central object is the MalLoc two-phase prompting pipeline with its family behavior lookup table. The lookup table maps each malware family (from MalRadar) to the subset of 12 malicious behaviors that family is known to exhibit; this targeted behavior description is what Phase 1 uses to interrogate each decompiled Smali class. Smali is a low-level, human-readable decompiled form of Android's DEX bytecode, produced with ApkTool. Phase 2 resends the class plus the Phase 1 explanation and demands METHOD/ROLE/CONFIDENCE lines for each involved method. The work this machinery does is to split the triple task of detection, behavior identification, and method pinpointing into two chained decisio
Load-bearing premise
The evaluation assumes that giving the LLM the exact list of malicious behaviors to look for—derived from family labels in MalRadar or from the authors' own knowledge of their demo app—does not inflate the measured localization accuracy, and that the authors' manual verification of the real-world sample is an adequate substitute for independent ground truth.
What would settle it
Run MalLoc on malware samples whose true malicious methods have been labeled by at least two independent analysts without access to the LLM output, and withhold the family behavior lookup so the model must infer behaviors from the code alone. If method-level precision and recall in that setup fall to the unguided-baseline level, the central claim fails. For the real-world result specifically, a blind manual marking of the RuMMs sample's malicious classes and methods, done before seeing MalLoc's output, would test whether the reported six classes and seventeen methods are the complete and corre
If this is right
- Analyst workload drops sharply: on MalApp, MalLoc flagged 22 of 165 methods as positive (about 87% fewer code units to review) while still catching every malicious method.
- Behavior role descriptions turn localization output into evidence an analyst can read and verify, supporting explainable malware analysis rather than black-box scores.
- Family knowledge can be reused as a guiding signal: the same LLM that fails on an undirected prompt performs much better when told which behaviors to look for.
- The approach supplies the missing 'Stage 3' input for downstream tasks the paper names: dynamic behavior modeling of localized payloads, robustness testing, and targeted countermeasures.
- Method-level localization is a new dimension beyond class-level approaches, and the paper's results suggest it is feasible with current LLMs despite obfuscation and the synthetic-method explosion from Java lambda compilation.
Where Pith is reading between the lines
- Editorial inference: the reported accuracy likely depends heavily on the family behavior lookup table; for a sample whose family is unknown, misclassified, or new, the LLM would have to enumerate behaviors itself, and precision may drop to something closer to the baseline.
- Editorial inference: MalApp's ground truth is author-known and its code is simple, so the perfect method recall is a best-case estimate, not evidence about large commercial apps where context windows and prompt size become limiting (the paper itself lists scalability as future work).
- Editorial inference: a natural testable extension would be to remove the lookup table and let Phase 1 propose behaviors, then compare localization accuracy; this would measure the approach under real analyst-facing conditions.
- Editorial inference: the role descriptions could serve as seed labels for training smaller, cheaper local classifiers on Smali methods, a use the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MalLoc, a two-phase LLM-based pipeline for fine-grained localization of malicious payloads in Android Smali code. Phase 1 determines whether each Smali class implements a specific malicious behavior drawn from a MalRadar-derived family behavior lookup table; Phase 2 identifies the individual methods responsible and provides human-readable role explanations. The authors evaluate on a self-built demo app (MalApp, 165 methods, 12 malicious methods, 3 known behaviors) with Phi-4 and GPT-4.1, and on one real-world RuMMs sample (21 classes, 66 methods) with manual verification. They report that GPT-4.1 with MalLoc achieves class-level precision/recall 0.83/1.00 and method-level 0.65/1.00 on MalApp, and 100% precision on the RuMMs sample. The paper claims this is the first method-level malicious payload localization approach with behavioral explanations.
Significance. If the result holds, MalLoc would be a useful step toward bridging malware detection/classification and actionable defense: it provides method-level localization and interpretable role descriptions, which are genuinely missing from most prior work. The two-phase decomposition is sensible, the use of family behavior knowledge is an interesting guiding signal, and the authors release the dataset and source code, which supports reproducibility. However, the significance is prospective rather than established: the empirical evidence is limited to one synthetic app and one small real-world sample, and the evaluation protocol has important confounds that the current manuscript does not fully address.
major comments (3)
- [Section III, Figure 3; Section IV-A; Table I; Section IV-B-2] The Phase 1 prompt (Figure 3, middle) explicitly includes 'Input – Malicious Behaviors to Look For: {behavior_description}', and Section III states that the family behavior lookup table is built from MalRadar family annotations. For MalApp, the behavior list is known because the authors built the app; for RuMMs, the five behaviors come from MalRadar's family labels (Section IV-A-2). Thus the reported metrics measure the LLM's ability to confirm and locate a supplied behavior hypothesis, not its ability to discover which behaviors are present in unknown code. The central claim in the Abstract and Introduction that MalLoc 'localize[s] malicious payloads at a fine-grained level' is only established conditional on a correct behavior list and family label. The authors should either add an experiment in which the behavior list is withheld, mis-specified, or produced by an upstream family/behav
- [Section IV-B-2] The real-world RuMMs evaluation reports '100% precision' based on the authors' own manual verification ('we manually verified all LLM-generated outputs'), with no independent ground truth, no blind protocol, and no inter-rater agreement. Because the authors designed the prompts and the pipeline and also selected the sample, this is a self-verified, unblinded upper bound. The manuscript should either provide an independent verification process (e.g., a second annotator with a reported agreement statistic, or comparison to an external ground-truth source) or present the result as an exploratory case study with this limitation explicitly stated. As it stands, the 100% precision figure is not a reliable measure of correctness.
- [Section IV-A and IV-B] The empirical basis consists of one synthetic app (51 classes/165 methods) and one real sample deliberately chosen because it is small ('We chose a small app in the family to minimize analysis complexity', Section IV-A-2; 21 classes/66 methods). With only two samples, no sampling procedure from MalRadar, and no confidence intervals or significance testing, the evaluation cannot support generalizable conclusions about MalLoc's effectiveness on real-world Android malware. The authors should either analyze a small random sample of apps from multiple families or explicitly label this as a proof-of-concept and temper statements about 'demonstrating the potential' of MalLoc in real-world, obfuscated environments.
minor comments (4)
- [Throughout] There are numerous typographical spacing errors in the extracted text (e.g., 'T oward', 'W e', 'T able' in the abstract, headers, and body). These should be cleaned up in the final version.
- [Figure 3] The baseline prompt (left panel) lists 12 behaviors inside the prompt, while the MalLoc Phase 1 prompt (middle panel) uses a placeholder '{behavior_description}'. It would be helpful to state explicitly where the detailed behavior descriptions are defined and to include the full set of 12 descriptions in the text or an appendix, since they are central to the method's inputs.
- [Section IV-B-1] The sentence 'it predicts 4 positive classes and 22 positive methods... reducing the manual analysis workload by approximately 87%' is slightly ambiguous: 22/165 is about 13.3% of methods flagged, so the 87% reduction refers to methods not requiring manual review. Clarify that the 22 predictions include false positives and that the workload reduction is not the same as accuracy.
- [References] A few references have formatting issues (e.g., [28] has an oddly spaced author name 'T . F . d. A. BISSY ANDE', and [12] appears to be a chapter title rather than a standalone paper). Please check the reference list against the publisher's style.
Circularity Check
No circularity: MalLoc's method-level localization is an open LLM reasoning task; supplying family-level behavior descriptions narrows the behavior space but does not determine method locations.
full rationale
The central claim is that an LLM, guided by per-sample behavior descriptions derived from the MalRadar family labels, can localize malicious Smali methods and describe their roles. The derivation chain is: (1) decompile APK to Smali; (2) obtain behavior descriptions from the family behavior lookup table; (3) Phase 1 asks whether a class implements a specified behavior; (4) Phase 2 asks which methods implement that behavior. The outputs—class labels, method signatures, and role descriptions—are not contained in the inputs. The behavior descriptions name behaviors but do not encode which methods or classes implement them, so the localization task remains an open reasoning problem rather than a construction from the input. The evaluation uses the same behavior labels as both input and ground-truth behavior, which is a benchmark-design limitation (the LLM is not asked to discover which behaviors are present), but this is not circular in the logical sense: the predicted method locations are not determined by the supplied behavior list, and the paper explicitly notes that MalRadar provides no payload locations. The RuMMs manual verification is unblinded and self-verified, which is a threat to external validity, not a circular step. The self-citations in the paper are to prior detection/localization work and to LLM-on-Smali motivation; none is load-bearing for the central derivation, and MalRadar is an external dataset. No fitted parameters are renamed as predictions, and no uniqueness theorem or ansatz is imported from the authors' own prior work to force the conclusion. Therefore no significant circularity is present.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption LLMs can effectively interpret and reason about Smali code
- domain assumption MalRadar family labels and the inferred family-to-behavior mapping are correct and complete
- domain assumption The 12 predefined malicious behavior categories and their descriptions cover all relevant malicious behaviors
- domain assumption The authors' manual verification of the RuMMs predictions is a valid ground truth
Cite this review
Pith. "Pith review of MalLoc: Toward Fine-grained Android Malicious Payload Localization via LLMs." pith.science (2026). https://pith.science/paper/QACKNAKE
@misc{pith2026250817856,
author = {Pith},
title = {Pith review of: MalLoc: Toward Fine-grained Android Malicious Payload Localization via LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/QACKNAKE}},
note = {Machine review of arXiv:2508.17856}
}
read the original abstract
The rapid evolution of Android malware poses significant challenges to the maintenance and security of mobile applications (apps). Traditional detection techniques often struggle to keep pace with emerging malware variants that employ advanced tactics such as code obfuscation and dynamic behavior triggering. One major limitation of these approaches is their inability to localize malicious payloads at a fine-grained level, hindering precise understanding of malicious behavior. This gap in understanding makes the design of effective and targeted mitigation strategies difficult, leaving mobile apps vulnerable to continuously evolving threats. To address this gap, we propose MalLoc, a novel approach that leverages the code understanding capabilities of large language models (LLMs) to localize malicious payloads at a fine-grained level within Android malware. Our experimental results demonstrate the feasibility and effectiveness of using LLMs for this task, highlighting the potential of MalLoc to enhance precision and interpretability in malware analysis. This work advances beyond traditional detection and classification by enabling deeper insights into behavior-level malicious logic and opens new directions for research, including dynamic modeling of localized threats and targeted countermeasure development.
Figures
Reference graph
Works this paper leans on
-
[1]
How many android users are there? global and us statistics (2025),
A. Turner, “How many android users are there? global and us statistics (2025),” https://www.bankmycell.com/blog/ how-many-android-users-are-there, 2025, accessed: 2025-06-02
work page 2025
-
[2]
Understanding android obfuscation techniques: A large-scale investigation in the wild,
S. Dong, M. Li, W . Diao, X. Liu, J. Liu, Z. Li, F . Xu, K. Chen, X. W ang, and K. Zhang, “Understanding android obfuscation techniques: A large-scale investigation in the wild,” in Security and privacy in communication networks: 14th international conference, secureComm 2018, Singapore, Singapore, August 8-10, 2018, proceedings, part i. Springer, 2018, p...
work page 2018
-
[3]
Dynamic trigger-based attacks against next-generation iot malware family classifiers,
Y . Zhang, S. T orabi, J. Y an, and C. Assi, “Dynamic trigger-based attacks against next-generation iot malware family classifiers,” Computers & Security, vol. 149, p. 104187, 2025
work page 2025
-
[4]
Detection of repackaged android malware with code-heterogeneity features,
K. Tian, D. Y ao, B. G. Ryder, G. T an, and G. Peng, “Detection of repackaged android malware with code-heterogeneity features, ”IEEE Transactions on Dependable and Secure Computing, vol. 17, no. 1, pp. 64–77, 2017
work page 2017
-
[5]
A survey and evaluation of android-based malware evasion techniques and detection frameworks,
P . Faruki, R. Bhan, V . Jain, S. Bhatia, N. El Madhoun, and R. Pamula, “ A survey and evaluation of android-based malware evasion techniques and detection frameworks, ”Information, vol. 14, no. 7, p. 374, 2023
work page 2023
-
[6]
Unmasking the veiled: A comprehensive analysis of android evasive malware,
A. Ruggia, D. Nisi, S. Dambra, A. Merlo, D. Balzarotti, and S. Aonzo, “Unmasking the veiled: A comprehensive analysis of android evasive malware,” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, 2024, pp. 383–398
work page 2024
-
[7]
Drebin: Effective and explainable detection of android malware in your pocket
D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, K. Rieck, and C. Siemens, “Drebin: Effective and explainable detection of android malware in your pocket. ” inNdss, vol. 14, no. 1, 2014, pp. 23–26
work page 2014
-
[8]
Malscan: Fast market-wide mobile malware scanning by social-network centrality analysis,
Y . Wu, X. Li, D. Zou, W . Y ang, X. Zhang, and H. Jin, “Malscan: Fast market-wide mobile malware scanning by social-network centrality analysis,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 139–150
work page 2019
-
[9]
N. Daoudi, J. Samhi, A. K. Kabore, K. Allix, T . F . Bissyand´e, and J. Klein, “Dexray: a simple, yet effective deep learning approach to android malware detection based on image representation of bytecode, ” in Deployable Machine Learning for Security Defense: Second International W orkshop, MLHat 2021, V irtual Event, August 15, 2021, Proceedings 2. Spr...
work page 2021
-
[10]
Android malware detection: looking beyond dalvik bytecode,
T . Sun, N. Daoudi, K. Allix, and T . F . Bissyand´e, “ Android malware detection: looking beyond dalvik bytecode, ” in2021 36th IEEE/ACM International Conference on Automated Software Engineering W orkshops (ASEW). IEEE, 2021, pp. 34–39
work page 2021
-
[11]
Detectbert: T owards full app-level representation learning to detect android malware,
T . Sun, N. Daoudi, K. Kim, K. Allix, T . F . Bissyand´e, and J. Klein, “Detectbert: T owards full app-level representation learning to detect android malware, ” inProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2024, pp. 420–426
work page 2024
-
[12]
Android malware detection based on novel representations of apps,
T . Sun, N. Daoudi, K. Allix, J. Samhi, K. Kim, X. Zhou, A. K. Kabore, D. Kim, D. Lo, T . F . Bissyand´e et al., “ Android malware detection based on novel representations of apps,” inMalware: Handbook of Prevention and Detection. Springer, 2024, pp. 197–212
work page 2024
-
[13]
Android malware family classification and analysis: Current status and future directions,
F . Alswaina and K. Elleithy, “ Android malware family classification and analysis: Current status and future directions,”Electronics, vol. 9, no. 6, p. 942, 2020
work page 2020
-
[14]
A hybrid analysis-based approach to android malware family classification,
C. Ding, N. Luktarhan, B. Lu, and W . Zhang, “ A hybrid analysis-based approach to android malware family classification,”Entropy, vol. 23, no. 8, p. 1009, 2021
work page 2021
-
[15]
Efficient deep learning network with multi-streams for android malware family classification,
H.-I. Kim, M. Kang, S.-J. Cho, and S.-I. Choi, “Efficient deep learning network with multi-streams for android malware family classification, ” IEEE Access, vol. 10, pp. 5518–5532, 2021
work page 2021
-
[16]
Malnet: A large-scale image database of malicious software,
S. Freitas, R. Duggal, and D. H. Chau, “Malnet: A large-scale image database of malicious software,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 3948–3952
work page 2022
-
[17]
Temporal-incremental learning for android malware detection,
T . Sun, N. Daoudi, W . Pian, K. Kim, K. Allix, T . F . Bissyand ´e, and J. Klein, “Temporal-incremental learning for android malware detection,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 4, pp. 1–30, 2025
work page 2025
-
[18]
A multi-view context-aware approach to android malware detection and malicious code localization,
A. Narayanan, M. Chandramohan, L. Chen, and Y . Liu, “ A multi-view context-aware approach to android malware detection and malicious code localization,” Empirical Software Engineering, vol. 23, pp. 1222–1274, 2018
work page 2018
-
[19]
Dexbert: Effective, task-agnostic and fine-grained representation learning of android bytecode,
T . Sun, K. Allix, K. Kim, X. Zhou, D. Kim, D. Lo, T . F . Bissyand´e, and J. Klein, “Dexbert: Effective, task-agnostic and fine-grained representation learning of android bytecode,”IEEE Transactions on Software Engineering, vol. 49, no. 10, pp. 4691–4706, 2023
work page 2023
-
[20]
Malradar: Demystifying android malware in the new era,
L. W ang, H. W ang, R. He, R. T ao, G. Meng, X. Luo, and X. Liu, “Malradar: Demystifying android malware in the new era, ”Proceedings of the ACM on Measurement and Analysis of Computing Systems , vol. 6, no. 2, pp. 1–27, 2022
work page 2022
-
[21]
MaMaDroid: Detecting Android Malware by Building Markov Chains of Behavioral Models
E. Mariconti, L. Onwuzurike, P . Andriotis, E. De Cristofaro, G. Ross, and G. Stringhini, “Mamadroid: Detecting android malware by building markov chains of behavioral models,” arXiv preprint arXiv:1612.04433, 2016
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[22]
Unraveling the Key of Machine Learning-based Android Malware Detection
J. Liu, J. Zeng, F . Pierazzi, L. Cavallaro, and Z. Liang, “Unraveling the key of machine learning solutions for android malware detection,” arXiv preprint arXiv:2402.02953, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[23]
Difuzer: Uncovering suspicious hidden sensitive operations in android apps,
J. Samhi, L. Li, T . F . Bissyande, and J. Klein, “Difuzer: Uncovering suspicious hidden sensitive operations in android apps,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, May 2022, pp. 723–735. [Online]. A vailable: https://doi.ieeecomputersociety.org/10.1145/3510003.3510135
arXiv 2022
-
[24]
Improving Logic Bomb Identification in Android Apps via Context-A ware Anomaly Detection ,
M. Alecci, J. Samhi, L. Li, T . F . Bissyande, and J. Klein, “ Improving Logic Bomb Identification in Android Apps via Context-A ware Anomaly Detection ,”IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 05, pp. 4735–4753, Sep. 2024. [Online]. A vail- able: https://doi.ieeecomputersociety.org/10.1109/TDSC.2024.3358979
-
[25]
Lamd: Context-driven android malware detection and classification with llms,
X. Qian, X. Zheng, Y . He, S. Y ang, and L. Cavallaro, “Lamd: Context-driven android malware detection and classification with llms, ” arXiv preprint arXiv:2502.13055, 2025
Pith/arXiv arXiv 2025
-
[26]
“Dalvik executable format,” https://source.android.com/docs/core/ runtime/dex-format, accessed: 2025-06-02
work page 2025
- [27]
-
[28]
T oward llm-driven gdpr compliance checking for android apps,
M. Alecci, N. Sannier, M. Ceci, S. Abualhaija, J. Samhi, D. Bianculli, T . F . d. A. BISSY ANDE, and J. Klein, “T oward llm-driven gdpr compliance checking for android apps,” in33rd ACM International Conference on the F oundations of Software Engineering (FSE Companion’25), 2025
work page 2025
-
[29]
Adapt: As-needed decomposition and planning with language models,
A. Prasad, A. Koller, M. Hartmann, P . Clark, A. Sabharwal, M. Bansal, and T . Khot, “ Adapt: As-needed decomposition and planning with language models,” inFindings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 4226–4252
work page 2024
-
[30]
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P . Kauffmannet al., “Phi-4 technical report, ”arXiv preprint arXiv:2412.08905, 2024
Pith/arXiv arXiv 2024
- [31]
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.