SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents

Aishan Liu; Ee-Chien Chang; Jinyuan He; Siyuan Liang; Tianmeng Fang; Xiaochun Cao; Yan Xiao; Zhe Liu

REVIEW 4 major objections 5 minor 1 cited by

SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A chain-level guard that reviews each action in context cuts mobile-agent jailbreaks from 86.1% to 8.4% while preserving task completion.

desk verdict Plausible modular mobile-agent defense whose headline numbers are internally inconsistent; the method is worth a major revision, not acceptance as-is. read the letter →

arxiv 2507.00841 v1 pith:SI5KQM6W submitted 2025-07-01 cs.AI cs.CR

Siyuan Liang , Tianmeng Fang , Zhe Liu , Aishan Liu , Yan Xiao , Jinyuan He , Ee-Chien Chang , Xiaochun Cao This is my paper

classification cs.AIcs.CR

keywords mobileagentjailbreakdefensemultimodalsecurityactiontrajectorysafetypreferenceoptimizationLLM-as-judgeSafeBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that jailbreak attacks on multimodal mobile agents should be defended at the level of the action chain, not the prompt. It proposes SafeMobile, a plug-in defense plus automated evaluator: SafeTrajGuard reviews each candidate action against the current screen, the task, and the previous two actions, and blocks actions it labels unsafe; GPTJudge scores complete trajectories from 0 to 100 and flags any trajectory scoring below 50 as a jailbreak. On 330 high-risk tasks across six categories, SafeMobile reports cutting the GPT-measured attack success rate from 86.1% to 8.4%, and the human-annotated rate from 72.2% to 16.7%, while normal-task completion stays at 77.8%. If these numbers transfer, agents that control phones and other devices can gain a meaningful safety layer without giving up utility, and can measure residual risk cheaply and consistently.

What carries the argument

The central object is the risk-sensitive behavior chain: a task trajectory $\tau = \{a_1, \dots, a_T\}$ with a subset of key actions $K_\tau$ that are turning points or high-impact decisions. The load-bearing mechanism is the risk-weighted preference objective $L = -\log \frac{\exp(\beta \Psi(\tau^+, x))}{\exp(\beta \Psi(\tau^+, x)) + \exp(\beta \Psi(\tau^-, x))}$, with $\Psi(\tau,x) = \sum_t w_t \log \pi(a_t \mid x, a_{<t})$, where $w_t > 1$ for key actions and 1 otherwise; this makes the guard focus its safety discrimination on the steps that decide whether a chain becomes harmful. Inference adds a two-step history window, so the guard judges the current action in the context of the two preceding screenshot-action pairs, catching patterns like navigation to a sensitive page followed by a configuration change. GPTJudge is the evaluation counterpart: a GPT-4 prompt that outputs a risk score $s \in [0,100]$ for the whole trajectory, with the binary jailbreak label $y = 1$ if $s < 50$.

What would settle it

Run SafeMobile on a held-out set of jailbreak attempts collected from real attackers or red teams on apps not present in the training or benchmark pool, with independent human raters; the claim fails if the attack success rate no longer drops to single-digit percentages, if task completion falls by more than about 5%, or if GPTJudge and human raters disagree on more than roughly 15% of trajectories.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a trajectory is a risk-sensitive behavior chain, and that preference-optimizing a small LLM to weight key risky actions makes chain-level jailbreak detection work in practice. SafeTrajDPO builds paired safe and unsafe trajectories from the same GUI page, marks the key action in each, and trains with a DPO-style objective in which the scoring function $\Psi(\tau,x) = \sum_t w_t \log \pi(a_t \mid x, a_{<t})$ up-weights key actions; subsequences around key actions are oversampled. At inference, SafeTrajGuard receives $\{P_{\mathrm{init}}, x_{\mathrm{text}}, x_{\mathrm{layout}}, h_{t-2}, h_{t-1}, a_t\}$ and outputs SAFE or UNSAFE before the action executes, with a system prompt that defaults to allowing when uncertain. GPTJudge then turns each full trajectory into a 0-100 harm score and labels jailbreaks by threshold $\theta=50$, and the paper reports RSS at least 0.85 against human raters. Table I gives post-defense G-ASR as 8.4% and H-ASR as 16.7%; Table II reports 16.7% and 8.4% for the same two metrics, and task completion remains 77.8% in both cases.

Load-bearing premise

The load-bearing premise is that the 330 test tasks, mostly generated by LLMs from 80 expert-curated seed tasks, are representative of real jailbreak attempts, and that GPTJudge's hand-set threshold of 50 correctly separates harmful from harmless behavior.

Editorial extensions

If this is right

Deploying SafeTrajGuard in front of a mobile agent can cut GPT-measured jailbreak success from 86.1% to single digits (8.4% in Table I) while leaving normal-task completion unchanged at 77.8%.
The defense generalizes across four vision-language backbones and across single-agent and multi-agent frameworks, with post-defense G-ASR at or below 19.4% in every condition the paper tests.
The ablation shows the historical context window is what suppresses chained attacks: without it G-ASR rises from 16.7% to 22.2%, and the base guard without training is too conservative, dropping TCR to 25.1%.
GPTJudge's automatically computed attack success rate matches human annotation with RSS at least 0.85 over more than 300 tasks, so it can replace manual post-hoc review.
Text-level defenses such as Scot, Self-Reminder, and SmoothLLM leave G-ASR between 52.7% and 80.5%, which the paper attributes to their lack of trajectory structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because SafeTrajGuard is itself an LLM with a default-to-safe bias, an attacker could try to jailbreak the guard rather than the agent; the paper does not test attacks targeted at the defense module itself.
The same chain-review recipe could in principle transfer to desktop or web automation agents, since their trajectories share the key-action structure, but the reported evidence covers only Android GUI environments.
GPTJudge's single threshold $\theta = 50$ treats every score below 50 as a full jailbreak; category-specific thresholds or severity labels would likely change measured ASR and could rank defenses differently.
A stronger test would be to collect real-world jailbreak attempts from actual users or red teams rather than LLM-generated expansions of the 80-task benchmark; the paper's 330-task set is closest to simulated attacks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

The stress-test note is correct, and it captures the main problem: the paper's quantitative claims don't survive contact with its own tables. The idea behind SafeMobile is a reasonable one for the mobile-agent security niche: a history-conditioned guard trained with a key-action-weighted DPO objective, plus an LLM judge for trajectory-level harm scoring. That's a sensible modular design, and the paper deserves credit for making the defense non-invasive (pluggable, no retraining of the agent) and for evaluating across two agent frameworks and four VLMs. The ablation in Table V is the most convincing part: adding history conditioning improves G-ASR from 22.2% to 16.7% while keeping TCR at 77.8%, and the base LLaMA-only row (TCR 25.1%) shows why naive refusal is not an answer.

But the numbers in the paper are not internally consistent, and that is a load-bearing flaw. Section IV.B.2 says the 80-task MobileSafetyBench was extended fivefold, which gives 400 tasks, yet the abstract says 330 jailbreak tasks. Table I's percentages imply 36 tasks. Table IV says "expand the number of task instances for each of the three high-risk task categories by five times," which is a different claim again. More directly, the SafeMobile row in Table I reports G-ASR=8.4% and H-ASR=16.7%, while the same row in Table II reports G-ASR=16.7% and H-ASR=8.4%. The abstract's "78.4%" G-ASR reduction matches neither absolute subtraction (77.7 or 69.4) nor relative reduction (90.2% or 80.6%). None of these can be brushed off as minor typos, because the headline metric is the one the whole paper rests on.

There is also a real circularity concern. G-Score and G-ASR come from the paper's own GPTJudge with a hand-set threshold (theta=50). The preference training data are generated by GPT-4o and Claude. So LLM judgment is baked into training, defense, and evaluation. The human H-ASR provides some anchor, and the RSS values in Table VI claim high agreement, but RSS is never defined and the cited reference [36] is Dubois's fuzzy sets book, which has nothing to do with it. No confidence intervals, no seeds, no released code or data. So the qualitative direction is believable—a guard can block dangerous actions without wrecking task completion—but the specific magnitudes are not recoverable from the text.

Who is this for? Someone working on mobile-agent safety or LLM-agent defenses will find the design worth reading about, but only as a proof-of-concept, not as a measurement. The paper needs a major revision: reconcile the task counts, define RSS, correct the swapped table entries, release artifacts, and report variance. I'd send it to reviewers—there is enough substance in the method and the preliminary results to merit referee time—but I would not accept it in this form. The next version might be a solid contribution.

Recommendation: engage with it as a revise-and-resubmit, with specific requests for consistency checks and a calibrated threshold analysis.

Referee Report

4 major / 5 minor

Summary. The paper proposes SafeMobile, a pluggable defense framework for multimodal mobile agents, consisting of SafeTrajGuard (an LLM-based trajectory-level safety reviewer trained with a risk-weighted preference objective) and GPTJudge (an LLM-as-a-judge automated jailbreak evaluator). The authors report experiments on AppAgent and Mobile-Agent-v2 over high-risk task categories, claiming an average G-Score increase of 52.9 points, a G-ASR reduction of 78.4% across 330 jailbreak tasks, unchanged task completion rate, and high consistency between automated and human jailbreak assessment.

Significance. If the reported results were reproducible, the contribution would be valuable: a non-invasive, modular defense that operates at the behavior-chain level rather than the prompt level, together with an automated evaluation protocol, addresses a real gap in mobile-agent security. The paper also ships a concrete training recipe (SafeTrajDPO) and an evaluation protocol that could be reused by other researchers. However, the empirical claim as currently written cannot be verified because the central numbers are internally inconsistent, and the evaluation is heavily dependent on a hand-set threshold and the authors' own LLM judge without reported variance or calibration.

major comments (4)

[Section V.A, Tables I and II] The SafeMobile row reports swapped G-ASR and H-ASR values across the two tables: Table I gives G-ASR=8.4% and H-ASR=16.7%, while Table II gives G-ASR=16.7% and H-ASR=8.4% for the same condition. The abstract's claim of a 78.4% G-ASR reduction cannot be recovered from either table: 86.1%−8.4%=77.7% and 86.1%−16.7%=69.4% are absolute differences, and the relative reductions are 90.2% and 80.6%, respectively. The authors must provide one consistent set of definitions, raw scores, and computed metrics that yields the headline number.
[Section IV.B.2 and Contributions] The dataset size is not internally consistent. Section IV.B.2 states that MobileSafetyBench's 80 expert-curated tasks are extended fivefold, which should yield 400 tasks, yet the contributions section claims 330 jailbreak tasks. Table I's percentages (e.g., 83.3%, 16.7%, 0.0%) imply 6 tasks per category, or 36 total jailbreak tasks, while Table IV's percentages (e.g., 63.3%, 93.6%) imply 30 tasks per category. The manuscript never states N for each table or explains why these counts differ. Every reported G-ASR and H-ASR percentage must be accompanied by the corresponding task count.
[Section III.C.2, Eq. (12)] The jailbreak determination is made by thresholding GPTJudge's risk score at a hand-set value θ=50, and the headline G-ASR and G-Score metrics are computed entirely by GPTJudge (Section IV.B.4). No sensitivity analysis for θ, no confidence intervals, and no repeated-seed or repeated-run variance are reported. Because the training preference data are also generated by GPT-4o and Claude (Section IV.B.2-3), the evaluation is not independent of the LLM family used to build the defense. The authors should report the distribution of G-Scores, a threshold sweep, and at least a stability analysis over judge runs, ideally with external validation beyond H-ASR.
[Section III.A.3, Eq. (8)] SafeTrajDPO is presented as a DPO variant, but Eq. (8) is not standard DPO: it lacks a reference policy and any KL regularization term, while the text claims the method works 'without relying on reward models or KL penalties.' As written, the objective is a binary preference classification loss over exp(β·Ψ), which may be reasonable but is not DPO and requires its own derivation and justification for why it yields calibrated safety judgments. Additionally, the key-action identifier is_key(a_t) in Eq. (7) is never defined operationally; the paper should specify how key actions are obtained in practice, since the weighting in Eq. (9) depends on it.

minor comments (5)

[Throughout] The system name is inconsistent: the abstract and most sections use SafeMobile, while Fig. 2, Fig. 5, and Section VII use TAME-MA, and Section VI uses Team-MA. Please unify the terminology.
[Section IV.B.4] GPTJudge is said to operate via GPT-4, while Section III.C and Fig. 4 mention GPT-4o. Please clarify which model was used for scoring and whether results vary across judge models.
[Section VI] MobileSafetyBench is cited by name throughout the paper but is not included in the reference list; please add the corresponding citation.
[Fig. 6] The axis labels in Fig. 6 are garbled (e.g., 'Jailbreak TaskS ize', 'Clean Task8 6', 'Ratio'), making the plots hard to interpret. Please regenerate the figure with clear labels.
[Section V.D] The RSS metric is attributed to reference [36], which is Dubois's 'Fuzzy Sets and Systems'; this does not appear to be the correct source for a relative similarity score between automated and human jailbreak annotations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; internal numeric inconsistencies exist but the core defense claim has independent human and external-benchmark grounding.

full rationale

SafeMobile's headline G-ASR numbers are computed by the paper's own GPTJudge with a hand-set threshold θ=50, and the preference data are LLM-generated; however, these facts do not make the derivation circular. The defense is evaluated on MobileSafetyBench, an external benchmark, and H-ASR is human-annotated, providing an independent anchor. GPTJudge is validated against H-ASR via RSS rather than being fit to H-ASR, and Eq. (12) is a stated decision rule, not a fitted calibration. Eq. (8) is a DPO-style training loss, and no equation or fitted parameter reduces a predicted quantity to its own input. The self-citations in the related-work section are not load-bearing: no uniqueness theorem or prior result by these authors is invoked to forbid alternatives. The paper does contain serious internal inconsistencies—Table I and Table II swap SafeMobile's G-ASR and H-ASR values, the 80-task benchmark 'extended fivefold' does not clearly yield the claimed 330 tasks, and the abstract's '78.4%' reduction cannot be recovered from any pair of reported percentages—but these are correctness and reporting defects, not circularity. No circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of modeling choices: the benchmark tasks, the LLM judge, the key-action annotation, the training objective, and the thresholds. All are either taken from prior work or set by the authors, and none are validated against an independent ground truth except H-ASR, whose reliability is also unspecified.

free parameters (6)

GPTJudge risk threshold theta = 50
Eq. (12) classifies any trajectory with score below 50 as a successful jailbreak; no calibration or sensitivity analysis is reported, so the headline G-ASR numbers depend on this hand-chosen value.
SafeTrajDPO temperature beta = not reported
Eq. (8) uses beta as a temperature but the paper never states its value or how it was chosen.
Key-action weight w_t = not reported
Section III.A.3 says a larger weight is applied when a_t is a key action, but no concrete value is given; this weight directly shapes the training objective.
History window size = 2
Section III.B.3 selects the most recent two historical traces to balance efficiency and context; this choice affects the guard's ability to detect chain risks.
Preference dataset size and ratio = 3000 pairs, 5:1 positive:negative
Section V.C.2 and V.C.3 select these via ablation; they are tuned to the test set and not independently justified.
Fivefold extension of MobileSafetyBench = 80% LLM-generated, 20% manually crafted
Section IV.B.2 extends the 80-task benchmark with LLM-generated tasks; the mix and verification process are not fully specified, and the total task count is inconsistent with the tables.

assumptions (5)

domain assumption MobileSafetyBench's 80 high-risk tasks are representative of real mobile-agent jailbreak attacks.
Section IV.B.2 uses this benchmark as the primary evaluation; if real attacks differ (for example adversarial prompt optimization or screen manipulation), the measured defense effectiveness may not transfer.
domain assumption GPTJudge's score s in [0,100] and threshold theta=50 is a valid measure of trajectory harmfulness.
Section III.C defines the entire evaluation around this score; no human calibration is shown before the threshold is applied, and the RSS validation is presented without a definition.
ad hoc to paper Key actions can be identified by is_key(a_t).
Eq. (7) defines key actions as page navigations, setting confirmations, account submissions, but no operational rule or evaluation of the identifier is given; the DPO weights depend on it.
domain assumption The 12 risk categories from Wu et al. are sufficient for negative sample construction.
Section III.A.2 draws all negative actions from these categories; any jailbreak behavior outside these categories is not represented in training.
ad hoc to paper The DPO-style objective in Eq. (8) is valid without a reference policy.
Standard DPO includes a reference model to prevent reward hacking; Eq. (8) omits it without justification, so training may drift from the base model's behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents." pith.science (2026). https://pith.science/paper/SI5KQM6W

@misc{pith2026250700841,
  author       = {Pith},
  title        = {Pith review of: SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SI5KQM6W}},
  note         = {Machine review of arXiv:2507.00841}
}

read the original abstract

With the wide application of multimodal foundation models in intelligent agent systems, scenarios such as mobile device control, intelligent assistant interaction, and multimodal task execution are gradually relying on such large model-driven agents. However, the related systems are also increasingly exposed to potential jailbreak risks. Attackers may induce the agents to bypass the original behavioral constraints through specific inputs, and then trigger certain risky and sensitive operations, such as modifying settings, executing unauthorized commands, or impersonating user identities, which brings new challenges to system security. Existing security measures for intelligent agents still have limitations when facing complex interactions, especially in detecting potentially risky behaviors across multiple rounds of conversations or sequences of tasks. In addition, an efficient and consistent automated methodology to assist in assessing and determining the impact of such risks is currently lacking. This work explores the security issues surrounding mobile multimodal agents, attempts to construct a risk discrimination mechanism by incorporating behavioral sequence information, and designs an automated assisted assessment scheme based on a large language model. Through preliminary validation in several representative high-risk tasks, the results show that the method can improve the recognition of risky behaviors to some extent and assist in reducing the probability of agents being jailbroken. We hope that this study can provide some valuable references for the security risk modeling and protection of multimodal intelligent agent systems.

Figures

Figures reproduced from arXiv: 2507.00841 by the authors.

**Figure 1.** Overview of the mobile agent jailbreak defense and evaluation framework. The system consists of a mobile agent for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Overview of the overall framework of the defense system,. The system first receives user task commands and interface [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** System-level instruction used by the SafeTrajGuard [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: System prompt used by GPTJudge to evaluate the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: App-to-Task risk mapping and G-ASR comparison [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Robustness analysis under varying data size, ratio, and LLMs. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Memory Enhanced Fractional-Order Dung Beetle Optimization for Photovoltaic Parameter Identification
cs.NE 2025-08 reject novelty 3.0 of 10

The claimed MFO-DBO algorithm and its CEC2017/PV results are absent from the manuscript, which instead contains an unrelated prompt-stealing attack paper.

Reference graph

Works this paper leans on

54 extracted references · 12 canonical work pages · cited by 1 Pith paper

[36]

D. J. Dubois,Fuzzy sets and systems: theory and applications. Aca- demic press, 1980, vol. 144

work page 1980
[1]

Multimodal foundation models: From specialists to general-purpose assistants,

C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, J. Gaoet al., “Multimodal foundation models: From specialists to general-purpose assistants,”Foundations and Trends® in Computer Graphics and Vision, vol. 16, no. 1-2, pp. 1–214, 2024

work page 2024
[2]

Towards artificial general intelligence via a multimodal foundation model,

N. Fei, Z. Lu, Y . Gao, G. Yang, Y . Huo, J. Wen, H. Lu, R. Song, X. Gao, T. Xianget al., “Towards artificial general intelligence via a multimodal foundation model,”Nature Communications, vol. 13, no. 1, p. 3094, 2022

work page 2022
[3]

Hemm: Holistic evaluation of multimodal foundation models,

P. P. Liang, A. Goindani, T. Chafekar, L. Mathur, H. Yu, R. Salakhut- dinov, and L.-P. Morency, “Hemm: Holistic evaluation of multimodal foundation models,”arXiv preprint arXiv:2407.03418, 2024

arXiv 2024
[4]

Multi-agent architecture search via agentic supernet,

G. Zhang, L. Niu, J. Fang, K. Wang, L. Bai, and X. Wang, “Multi-agent architecture search via agentic supernet,”arXiv preprint arXiv:2502.04180, 2025

arXiv 2025
[5]

Cut the crap: An economical communication pipeline for llm-based multi-agent systems,

G. Zhang, Y . Yue, Z. Li, S. Yun, G. Wan, K. Wang, D. Cheng, J. X. Yu, and T. Chen, “Cut the crap: An economical communication pipeline for llm-based multi-agent systems,”arXiv preprint arXiv:2410.02506, 2024

arXiv 2024
[6]

G-designer: Architecting multi-agent communication topologies via graph neural networks,

G. Zhang, Y . Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng, “G-designer: Architecting multi-agent communication topologies via graph neural networks,”arXiv preprint arXiv:2410.11782, 2024

arXiv 2024
[7]

A comprehensive survey in llm (- agent) full stack safety: Data, training and deployment,

K. Wang, G. Zhang, Z. Zhou, J. Wu, M. Yu, S. Zhao, C. Yin, J. Fu, Y . Yan, H. Luoet al., “A comprehensive survey in llm (- agent) full stack safety: Data, training and deployment,”arXiv preprint arXiv:2504.15585, 2025

arXiv 2025

Show all 54 references

[8]

Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration,

J. Wang, H. Xu, H. Jia, X. Zhang, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration,”arXiv preprint arXiv:2406.01014, 2024

2024 arXiv
[9]

Mobilegpt: Augmenting llm with human-like app memory for mobile task automation,

S. Lee, J. Choi, J. Lee, M. H. Wasi, H. Choi, S. Ko, S. Oh, and I. Shin, “Mobilegpt: Augmenting llm with human-like app memory for mobile task automation,” inProceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024, pp. 1119– 1133

2024
[10]

Mobileexperts: A dynamic tool-enabled agent team in mobile devices,

J. Zhang, C. Zhao, Y . Zhao, Z. Yu, M. He, and J. Fan, “Mobileexperts: A dynamic tool-enabled agent team in mobile devices,”arXiv preprint arXiv:2407.03913, 2024

2024 arXiv
[11]

A survey on evaluation of large language models,

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wanget al., “A survey on evaluation of large language models,”ACM transactions on intelligent systems and technology, vol. 15, no. 3, pp. 1–45, 2024

2024
[12]

A comprehensive overview of large language models,

H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,”arXiv preprint arXiv:2307.06435, 2023

2023 arXiv
[13]

Jailbreak vision language models via bi-modal adversarial prompt,

Z. Ying, A. Liu, T. Zhang, Z. Yu, S. Liang, X. Liu, and D. Tao, “Jailbreak vision language models via bi-modal adversarial prompt,”arXiv preprint arXiv:2406.04031, 2024

2024 arXiv
[14]

Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms,

X. Li, S. Liang, J. Zhang, H. Fang, A. Liu, and E.-C. Chang, “Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms,”arXiv preprint arXiv:2402.14872, 2024

2024 arXiv
[15]

Cogmorph: Cognitive morphing attacks for text-to-image models,

Z. Jing, Z. Ying, L. Wang, S. Liang, A. Liu, X. Liu, and D. Tao, “Cogmorph: Cognitive morphing attacks for text-to-image models,” arXiv preprint arXiv:2501.11815, 2025

2025 arXiv
[16]

Safemlrm: Demystifying safety in multi-modal large reasoning models,

J. Fang, Y . Wang, R. Wang, Z. Yao, K. Wang, A. Zhang, X. Wang, and T.-S. Chua, “Safemlrm: Demystifying safety in multi-modal large reasoning models,”arXiv preprint arXiv:2504.08813, 2025

2025 arXiv
[17]

Alphaedit: Null-space constrained knowledge editing for language models,

J. Fang, H. Jiang, K. Wang, Y . Ma, S. Jie, X. Wang, X. He, and T.-S. Chua, “Alphaedit: Null-space constrained knowledge editing for language models,”arXiv preprint arXiv:2410.02355, 2024

2024 arXiv
[18]

On the role of attention heads in large language model safety,

Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, K. Wang, Y . Liu, J. Fang, and Y . Li, “On the role of attention heads in large language model safety,”arXiv preprint arXiv:2410.13708, 2024

2024 arXiv
[19]

Safety in large reasoning models: A survey,

C. Wang, Y . Liu, B. Li, D. Zhang, Z. Li, and J. Fang, “Safety in large reasoning models: A survey,”arXiv preprint arXiv:2504.17704, 2025

2025 arXiv
[20]

G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems,

S. Wang, G. Zhang, M. Yu, G. Wan, F. Meng, C. Guo, K. Wang, and Y . Wang, “G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems,” 2025. [Online]. Available: https://arxiv.org/abs/2502.11127

2025 arXiv
[21]

A survey on trustworthy llm agents: Threats and countermeasures,

M. Yu, F. Meng, X. Zhou, S. Wang, J. Mao, L. Pang, T. Chen, K. Wang, X. Li, Y . Zhanget al., “A survey on trustworthy llm agents: Threats and countermeasures,”arXiv preprint arXiv:2503.09648, 2025

2025 arXiv
[22]

Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning,

S. Liang, M. Zhu, A. Liu, B. Wu, X. Cao, and E.-C. Chang, “Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 645–24 654

2024
[23]

Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models,

Z. Ying, D. Zhang, Z. Jing, Y . Xiao, Q. Zou, A. Liu, S. Liang, X. Zhang, X. Liu, and D. Tao, “Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models,”arXiv preprint arXiv:2502.11054, 2025

2025 arXiv
[24]

Vl-trojan: Multimodal instruc- tion backdoor attacks against autoregressive visual language models,

J. Liang, S. Liang, A. Liu, and X. Cao, “Vl-trojan: Multimodal instruc- tion backdoor attacks against autoregressive visual language models,” International Journal of Computer Vision, pp. 1–20, 2025

2025
[25]

From assistants to adversaries: Exploring the security risks of mobile llm agents,

L. Wu, C. Wang, T. Liu, Y . Zhao, and H. Wang, “From assistants to adversaries: Exploring the security risks of mobile llm agents,”arXiv preprint arXiv:2505.12981, 2025

2025 arXiv
[26]

Rico: A mobile app dataset for building data-driven design applications,

B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y . Li, J. Nichols, and R. Kumar, “Rico: A mobile app dataset for building data-driven design applications,” inProceedings of the 30th annual ACM symposium on user interface software and technology, 2017, pp. 845– 854

2017
[27]

Theoretical analysis of kl-regularized rlhf with multiple reference models,

G. Aminian, A. R. Asadi, I. Shenfeld, and Y . Mroueh, “Theoretical analysis of kl-regularized rlhf with multiple reference models,”arXiv preprint arXiv:2502.01203, 2025

2025
[28]

Rlaif vs. rlhf: Scaling reinforce- ment learning from human feedback with ai feedback,

H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogiet al., “Rlaif vs. rlhf: Scaling reinforce- ment learning from human feedback with ai feedback,”arXiv preprint arXiv:2309.00267, 2023

2023 arXiv
[29]

Towards analyzing and understanding the limitations of dpo: A theoretical perspective,

D. Feng, B. Qin, C. Huang, Z. Zhang, and W. Lei, “Towards analyzing and understanding the limitations of dpo: A theoretical perspective,” arXiv preprint arXiv:2404.04626, 2024

2024 arXiv
[30]

Is dpo superior to ppo for llm alignment? a comprehensive study,

S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y . Wu, “Is dpo superior to ppo for llm alignment? a comprehensive study,”arXiv preprint arXiv:2404.10719, 2024

2024 arXiv
[31]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

2023 arXiv
[32]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[33]

Mis- tral: Dynamically managing power, performance, and adaptation cost in cloud infrastructures,

G. Jung, M. A. Hiltunen, K. R. Joshi, R. D. Schlichting, and C. Pu, “Mis- tral: Dynamically managing power, performance, and adaptation cost in cloud infrastructures,” in2010 IEEE 30th International Conference on Distributed Computing Systems. IEEE, 2010, pp. 62–73

2010
[34]

Qwen technical report,

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

2023 arXiv
[35]

Gemma: Open models based on gemini research and technology,

G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

2024 arXiv
[37]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millicanet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[38]

Deepseek-v3 technical report,

A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

2024 arXiv
[39]

Appagent: Multimodal agents as smartphone users,

C. Zhang, Z. Yang, J. Liu, Y . Li, Y . Han, X. Chen, Z. Huang, B. Fu, and G. Yu, “Appagent: Multimodal agents as smartphone users,” inProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025, pp. 1–20

2025
[40]

Autodroid: Llm-powered task automation in android,

H. Wen, Y . Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y . Liu, Y . Zhang, and Y . Liu, “Autodroid: Llm-powered task automation in android,” inProceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024, pp. 543–557

2024
[41]

Droidbot-gpt: Gpt-powered ui automation for android,

H. Wen, H. Wang, J. Liu, and Y . Li, “Droidbot-gpt: Gpt-powered ui automation for android,”arXiv preprint arXiv:2304.07061, 2023

2023 arXiv
[42]

Coco-agent: A comprehensive cog- nitive mllm agent for smartphone gui automation,

X. Ma, Z. Zhang, and H. Zhao, “Coco-agent: A comprehensive cog- nitive mllm agent for smartphone gui automation,”arXiv preprint arXiv:2402.11941, 2024

2024 arXiv
[43]

Learnact: Few-shot mobile gui agent with a unified demonstration benchmark,

G. Liu, P. Zhao, L. Liu, Z. Chen, Y . Chai, S. Ren, H. Wang, S. He, and W. Meng, “Learnact: Few-shot mobile gui agent with a unified demonstration benchmark,”arXiv preprint arXiv:2504.13805, 2025

2025 arXiv
[44]

Fedmo- bileagent: Training mobile agents using decentralized self-sourced data from diverse users,

W. Wang, Z. Yu, W. Liu, R. Ye, T. Jin, S. Chen, and Y . Wang, “Fedmo- bileagent: Training mobile agents using decentralized self-sourced data from diverse users,”arXiv preprint arXiv:2502.02982, 2025

2025 arXiv
[45]

Advancing mobile gui agents: A verifier-driven approach to practical deployment,

G. Dai, S. Jiang, T. Cao, Y . Li, Y . Yang, R. Tan, M. Li, and L. Qiu, “Advancing mobile gui agents: A verifier-driven approach to practical deployment,”arXiv preprint arXiv:2503.15937, 2025

2025
[46]

Adversarial training for multimodal large language models against jailbreak attacks,

L. Lu, S. Pang, S. Liang, H. Zhu, X. Zeng, A. Liu, Y . Liu, and Y . Zhou, “Adversarial training for multimodal large language models against jailbreak attacks,”arXiv preprint arXiv:2503.04833, 2025

2025
[47]

T2vshield: Model-agnostic jailbreak defense for text-to-video models,

S. Liang, J. Liu, J. Zhai, T. Fang, R. Tu, A. Liu, X. Cao, and D. Tao, “T2vshield: Model-agnostic jailbreak defense for text-to-video models,” arXiv preprint arXiv:2504.15512, 2025

2025 arXiv
[48]

Security matrix for multimodal agents on mobile devices: A systematic and proof of concept study,

Y . Yang, X. Yang, S. Li, C. Lin, Z. Zhao, C. Shen, and T. Zhang, “Security matrix for multimodal agents on mobile devices: A systematic and proof of concept study,”arXiv preprint arXiv:2407.09295, 2024

2024 arXiv
[49]

Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Leiet al., “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,”Advances in Neural Information Processing Systems, vol. 37, pp. 52 040–52 094, 2024

2024
[50]

Webshop: Towards scalable real-world web interaction with grounded language agents,

S. Yao, H. Chen, J. Yang, and K. Narasimhan, “Webshop: Towards scalable real-world web interaction with grounded language agents,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 20 744– 20 757, 2022

2022
[51]

Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,

D. Zhang, Z. Shen, R. Xie, S. Zhang, T. Xie, Z. Zhao, S. Chen, L. Chen, H. Xu, R. Caoet al., “Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,”arXiv preprint arXiv:2305.08144, 2023

2023 arXiv
[52]

Mobile-bench: An evaluation benchmark for llm-based mobile agents,

S. Deng, W. Xu, H. Sun, W. Liu, T. Tan, J. Liu, A. Li, J. Luan, B. Wang, R. Yanet al., “Mobile-bench: An evaluation benchmark for llm-based mobile agents,”arXiv preprint arXiv:2407.00993, 2024

2024 arXiv
[53]

Spa-bench: A comprehensive benchmark for smartphone agent evaluation,

J. Chen, D. Yuen, B. Xie, Y . Yang, G. Chen, Z. Wu, L. Yixing, X. Zhou, W. Liu, S. Wanget al., “Spa-bench: A comprehensive benchmark for smartphone agent evaluation,” inNeurIPS 2024 Workshop on Open- World Agents, 2024

2024
[54]

Webwalker: Benchmarking llms in web traver- sal,

J. Wu, W. Yin, Y . Jiang, Z. Wang, Z. Xi, R. Fang, L. Zhang, Y . He, D. Zhou, P. Xieet al., “Webwalker: Benchmarking llms in web traver- sal,”arXiv preprint arXiv:2501.07572, 2025

2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [36]

D. J. Dubois,Fuzzy sets and systems: theory and applications. Aca- demic press, 1980, vol. 144

work page 1980

[2] [1]

Multimodal foundation models: From specialists to general-purpose assistants,

C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, J. Gaoet al., “Multimodal foundation models: From specialists to general-purpose assistants,”Foundations and Trends® in Computer Graphics and Vision, vol. 16, no. 1-2, pp. 1–214, 2024

work page 2024

[3] [2]

Towards artificial general intelligence via a multimodal foundation model,

N. Fei, Z. Lu, Y . Gao, G. Yang, Y . Huo, J. Wen, H. Lu, R. Song, X. Gao, T. Xianget al., “Towards artificial general intelligence via a multimodal foundation model,”Nature Communications, vol. 13, no. 1, p. 3094, 2022

work page 2022

[4] [3]

Hemm: Holistic evaluation of multimodal foundation models,

P. P. Liang, A. Goindani, T. Chafekar, L. Mathur, H. Yu, R. Salakhut- dinov, and L.-P. Morency, “Hemm: Holistic evaluation of multimodal foundation models,”arXiv preprint arXiv:2407.03418, 2024

arXiv 2024

[5] [4]

Multi-agent architecture search via agentic supernet,

G. Zhang, L. Niu, J. Fang, K. Wang, L. Bai, and X. Wang, “Multi-agent architecture search via agentic supernet,”arXiv preprint arXiv:2502.04180, 2025

arXiv 2025

[6] [5]

Cut the crap: An economical communication pipeline for llm-based multi-agent systems,

G. Zhang, Y . Yue, Z. Li, S. Yun, G. Wan, K. Wang, D. Cheng, J. X. Yu, and T. Chen, “Cut the crap: An economical communication pipeline for llm-based multi-agent systems,”arXiv preprint arXiv:2410.02506, 2024

arXiv 2024

[7] [6]

G-designer: Architecting multi-agent communication topologies via graph neural networks,

G. Zhang, Y . Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng, “G-designer: Architecting multi-agent communication topologies via graph neural networks,”arXiv preprint arXiv:2410.11782, 2024

arXiv 2024

[8] [7]

A comprehensive survey in llm (- agent) full stack safety: Data, training and deployment,

K. Wang, G. Zhang, Z. Zhou, J. Wu, M. Yu, S. Zhao, C. Yin, J. Fu, Y . Yan, H. Luoet al., “A comprehensive survey in llm (- agent) full stack safety: Data, training and deployment,”arXiv preprint arXiv:2504.15585, 2025

arXiv 2025