FENCE: A Financial and Multimodal Jailbreak Detection Dataset

Mirae Kim; Seonghun Jeong; Youngjun Kwak

REVIEW 4 major objections 6 minor 13 references

FENCE: A Financial and Multimodal Jailbreak Detection Dataset

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FENCE, a 10,000-sample bilingual, image-based jailbreak dataset for finance, exposes hidden vulnerabilities in 15 VLMs and powers a compact 3B guardrail to 99% defense success.

desk verdict A useful new finance-specific multimodal jailbreak dataset whose core validation numbers need more transparency before I'd trust the 99%/95% headline claims. read the letter →

arxiv 2602.18154 v2 pith:Z7U2NLOL submitted 2026-02-20 cs.CL cs.AIcs.DB

Mirae Kim , Seonghun Jeong , Youngjun Kwak This is my paper

classification cs.CLcs.AIcs.DB

keywords multimodaljailbreakvision-languagemodelsafetyfinancialAIdetectiondatasetimage-basedattacksguardrailtrainingbilingualKorean-Englishattacksuccessrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces FENCE, the first benchmark explicitly designed for multimodal jailbreak detection in the financial domain. FENCE contains 10,000 balanced, bilingual (Korean–English) image–text pairs built from real FAQ queries, transformed into harmful variants and paired with real-world photos. The authors show that FENCE consistently triggers higher attack success rates than existing benchmarks across 15 vision-language models, revealing weaknesses that general-purpose datasets miss. They further demonstrate that a 3B-parameter model fine-tuned on FENCE reaches near-perfect defense rates on both FENCE and four external benchmarks, outperforming much larger dedicated guardrail systems.

What carries the argument

The central object is FENCE itself, a dataset built from a three-stage pipeline: (1) a two-step GPT-4o prompting process (role-playing rephrasing plus an evaluation prompt) that converts 2,500 real FAQ queries into semantically paired harmful variants while avoiding safety-triggering phrases; (2) collection of real-world photographs from a copyright-free platform, chosen to match the harmful or benign content; (3) text–image fusion, with 40% of samples using FigStep-style typographic templates. The balanced 50:50 benign/harmful design, the native Korean-first construction with English translation, and the explicit labeling of harm location (text vs image) together make FENCE a training resou

What would settle it

Have independent, diverse human annotators (including financial-domain experts) re-label a random subset of FENCE using a detailed, pre-registered rubric, and compare their labels to GPT-4o's. If agreement is substantially below 95%, or if a guardrail retrained on the human-labeled subset shows a meaningful drop in defense success rate on external benchmarks, the paper's central claim that FENCE is a robust training resource would be undermined.

Watch

Extended reading notes

Core claim

The paper's central discovery is that finance-specific, image-grounded jailbreak samples form a distinct and under-explored attack surface. FENCE's construction — benign-to-harmful transformations that preserve semantic alignment, real-world images, balanced training pairs, and multiple image-based attack strategies — produces a benchmark that is both harder to pass and more effective for training guardrails. Empirically, FENCE yields a mean attack success rate of 17.76% across 15 VLMs, higher than four existing benchmarks (8.77–16.31%), and even GPT-4o shows measurable vulnerability (3.2%) on FENCE. Fine-tuning a 3B Qwen2.5-VL on FENCE raises its defense success rate from 66.29% to 99.34% a

Load-bearing premise

The reliability of the dataset labels depends on GPT-4o's judgments of what is harmful, backed by a claimed 95% agreement with human annotations that the paper does not detail; if GPT-4o's notion of harm is systematically off, then the benchmark's attack success rates, the detector's training signal, and the reported defense improvements all inherit that bias.

Editorial extensions

If this is right

A compact, domain-specific multimodal guardrail can match or beat general-purpose safety models that are several times larger, suggesting that focused training data matters more than raw model scale for safety classification.
Existing safety benchmarks may systematically understate VLM vulnerability in regulated domains; FENCE-style construction may be needed to properly diagnose deployability of AI in finance and similar high-stakes sectors.
Strong image–text recognition ability is a prerequisite for accurate harmful-query classification, implying that improving a VLM's OCR and visual-semantic grounding could directly improve its jailbreak detection.
Balanced training with semantically aligned benign–harmful pairs yields strong out-of-distribution generalization, as the FENCE-trained model reaches 98–100% defense success on unseen general-purpose benchmarks.
The bilingual Korean–English design introduces a template for culturally grounded financial safety data, potentially transferable to other languages and regions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The unusually high ASR on FENCE relative to existing benchmarks suggests that finance-domain queries with image-grounded threats exploit a blind spot in general-purpose safety alignment; this may extend to other regulated verticals like healthcare and legal advice, where domain-specific adversarial datasets could similarly expose hidden vulnerabilities.
If the claimed 95% human-consistency rate for GPT-4o label judgments is not carefully replicated with a detailed annotation protocol, the entire label set — and thus the benchmark ASRs and the trained guardrail's behavior — may be biased by GPT-4o's own safety preferences rather than by an objective notion of financial harm.
The correlation between ITR performance and classification accuracy implies a testable extension: fine-tuning a VLM's visual-encoder or adding a dedicated OCR branch might raise the ceiling for jailbreak detection on typographic attacks, an avenue the paper does not explicitly explore.
FENCE's restricted release (access on request) means the headline numbers may not be immediately reproducible; an open, anonymized, or formally versioned release would allow the community to verify the generalization claims under different training budgets and base models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — here's my read. FENCE is a genuinely new resource: the first finance-focused, Korean-English multimodal jailbreak detection dataset, balanced 50/50 benign/harmful, built around image-grounded attacks. That fills a real gap. The construction is sensible: start with real FAQ queries from Korean financial institutions, transform to harmful via GPT-4o with role-playing and evaluation prompts, pair with Pixabay images, apply FigStep-style layouts. The t-SNE analysis gives some evidence that the harmful queries are semantically close to benign ones, which should make detection harder. The OOD results—Qwen2.5-VL 3B fine-tuned on FENCE doing well on JailBreakV-28K, FigStep, HADES, and MM-SafetyBench—are the strongest evidence that the dataset teaches something transferable, not just memorized patterns. That part holds up.

Now the soft spots. The 95% human-validation figure for GPT-4o's harmfulness judgments is load-bearing and under-specified: no sample size, no protocol, no disagreement breakdown. Since GPT-4o was used to generate the harmful queries and to score ASR, this is a real circularity risk. I'd want the annotation details released before treating the labels as ground truth. The dataset itself is also not public—only 'upon legitimate research request'—and there's no code. That limits verification. The headline numbers also need error bars; Tables 4, 6, and 7 report single ASR/accuracy values with no repeated-seed variance or confidence intervals. Minor: Table 7's mean DSR differs between the table (66.69%) and text (66.29%), and LlamaGuard4's size is given as 12B in text but 11B in the table. These are fixable. The 'state-of-the-art' claim in Table 6 is based on a narrow comparison set; soften it.

Bottom line: the resource is useful and the central claim—that FENCE can train a compact, effective finance-domain guardrail—is plausible and independently supported by OOD generalization. The missing validation details and release artifacts keep me from signing off on it as-is. For a reader working on multimodal safety or financial AI, worth a look. I'd send it to peer review with requests for the annotation protocol, error bars, and access details.

Referee Report

4 major / 6 minor

Summary. The paper introduces FENCE, a 10k-sample bilingual (Korean-English) multimodal dataset for jailbreak detection in finance. The dataset is balanced between benign and harmful samples, covers 15+ financial topics, and focuses on image-based attacks (BaseImg, TextImg, FigStep). Harmful queries are generated from benign FAQ queries via a two-step GPT-4o role-play/evaluation pipeline, with images sourced from Pixabay. The authors evaluate 15 VLMs on FENCE and four existing benchmarks, reporting higher attack success rates on FENCE. They then fine-tune a Qwen2.5-VL 3B model on FENCE and report 99% defense success across FENCE and four external benchmarks, outperforming larger LlamaGuard guard models. The central claim is that FENCE is a useful training and evaluation resource for multimodal jailbreak detection in finance.

Significance. If the label-generation and evaluation pipeline is trustworthy, FENCE fills a genuine gap: existing jailbreak benchmarks are predominantly English, text-based, and not finance-specific. The balanced benign/harmful design, the use of real-world images, and the broad coverage of financial topics are clear strengths. The OOD evaluation across four external benchmarks and the comparison with LlamaGuard models provide practically relevant evidence that fine-tuning on FENCE can produce a compact, effective guardrail. The paper also includes a useful diagnostic analysis linking image-text recognition quality to classification performance. However, the dataset's correctness and the headline numbers rest heavily on GPT-4o as generator and judge, and the only independent check—a 95% human-validation agreement statistic—is reported without any methodological detail. This is the key load-bearing issue.

major comments (4)

[Section 3.2.2 (Step 1)] The sentence 'Human validation confirmed that 95% of GPT-4o's judgments were consistent with human annotations' is the only independent check on the label-generation pipeline, but no sample size, annotation protocol, or disagreement breakdown is reported. This matters because GPT-4o generated the harmful variants, GPT-4o was used as the evaluator for harmfulness, and—if GPT-4o is also the ASR judge in Table 4—the same model both defines and measures the attack-success signal. The OOD results in Tables 6–7 are suggestive but do not break the loop, because JailBreakV-28K, FigStep, HADES, and MM-SafetyBench are themselves substantially LLM-generated and LLM-labeled. Please report the human-validation sample size, per-category agreement, inter-annotator agreement, and a human-evaluated ASR subset on FENCE.
[Section 4.1, Table 4] The ASR protocol is underspecified. No information is given about which model or human judges whether a response is harmful, how many samples are included in each 'mini' benchmark, or how the standard deviations were computed. Because the headline claim is that FENCE exposes higher ASRs than existing benchmarks, the missing protocol makes the comparison uninterpretable. Add the judgment protocol, sample counts, and confidence intervals or repeated-seed variation.
[Section 4.2, Tables 6–7] The fine-tuning description for Qwen2.5-VL 3B is incomplete: no learning rate, number of epochs, batch size, LoRA rank (if any), or train/validation split is given. The 99% defense-success result is also partially in-distribution by construction (the FENCE row), so the abstract's '99% ... across FENCE and four external benchmarks' should be qualified as post-fine-tuning and should distinguish ID from OOD performance. Please provide full training details and a breakdown.
[Section 4.2, Table 6] The comparison between the FENCE-tuned Qwen2.5-VL and the LlamaGuard baselines is not apples-to-apples: the paper does not state whether LlamaGuard models received any fine-tuning. If they are evaluated zero-shot, the comparison conflates model capacity with training signal. The OOD columns are still informative, but the FENCE column is in-distribution by construction and should be labeled as such. Additionally, Section 7 states the dataset will not be publicly released; for a dataset paper, this severely limits reproducibility and verification.

minor comments (6)

[Abstract] The abstract contains a typo: 'morereliableAIsystems' should be 'more reliable AI systems.' Also, '99 percent in-distribution accuracy' should specify that this is F1-score on FENCE, not a single accuracy number.
[Table 7] The table reports mean DSR as 66.69%, but the text in Section 4.2 says 66.29%. Please correct the inconsistency.
[Table 6] The header formatting is confusing: 'LLamaGuard 3Vision LLamaGuard 4Qwen2.5-VL(Ours) 8B 11B 3B' is not readable. Use separate rows or clear column headings.
[References and Related Work] The reference for HarmAug-Guard ('Lee et al.') is incomplete: no year or venue is provided. Also, the descriptions of VARCO Vision and Kanana1.5 Vision lack citations or version details.
[Section 7] The ethics statement says the dataset will not be publicly released and access is granted only upon request. For a benchmark paper, please provide a formal data-access plan or a safer public release of a validated subset.
[Table 3] The example row for TextImg has label 0 (0,0), but the input query text and image description might be misaligned; please clarify which component is benign/harmful in the examples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FENCE's claims are empirical benchmark results with external validation, not reductions to their inputs.

full rationale

The paper's central claims are empirical: FENCE produces higher ASRs across 15 VLMs (Table 4) and a 3B Qwen2.5-VL fine-tuned on FENCE reaches ~99% DSR on four external benchmarks (Tables 6-7). These are measurements, not derivations from definitions. The only self-referential element is GPT-4o's triple role in Section 3.2.2: GPT-4o transforms benign queries into harmful variants, its evaluation prompt determines whether output qualifies as harmful ('an evaluation prompt that determines whether the generated output qualifies as harmful'), and GPT-4o is itself one of the evaluated models in Table 4 (ASR 3.2%). This creates a partial loop for the GPT-4o row and for ground-truth label validity, but the paper reports an independent anchor: 'Human validation confirmed that 95% of GPT-4o's judgments were consistent with human annotations.' Although the sample size and protocol are not reported, the existence of human validation plus the four external benchmarks (JailBreakV-28K, FigStep, HADES, MM-SafetyBench) with pre-existing labels means the central generalization claim does not reduce to GPT-4o's own outputs. Section 6 openly flags that FENCE is 'built from synthetic adversarial prompts generated by GPT-4o' and that harm definitions involve subjectivity; this is a validity/reproducibility limitation, not circularity. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation steps were found. The missing human-validation details are a correctness risk, not a circularity risk.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the fidelity of GPT-4o-generated labels and the representativeness of the image-based attack construction, plus the transferability of finance-specific fine-tuning to general benchmarks. There are no free parameters or invented physical/formal entities.

assumptions (3)

ad hoc to paper GPT-4o is a reliable generator and judge of harmful content; its judgments match human annotations 95% of the time
Section 3.2.2: harmful queries are generated via GPT-4o role-playing and validated by GPT-4o evaluation; the 95% human agreement figure is given without sample size or protocol.
domain assumption Image-based attacks are a critical, realistic jailbreak vector in financial VLM deployments
Section 2.2 and 3.2.1 justify focusing on IA by claiming vision encoders are less aligned, but no deployed finance-VLM evidence is provided.
domain assumption Fine-tuning on a synthetic finance-specific dataset transfers to general-purpose jailbreak benchmarks
Section 4.2 uses OOD performance on JailBreakV-28K, FigStep, HADES, and MM-SafetyBench as evidence of generalization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FENCE: A Financial and Multimodal Jailbreak Detection Dataset." pith.science (2026). https://pith.science/paper/Z7U2NLOL

@misc{pith2026260218154,
  author       = {Pith},
  title        = {Pith review of: FENCE: A Financial and Multimodal Jailbreak Detection Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7U2NLOL}},
  note         = {Machine review of arXiv:2602.18154}
}

read the original abstract

Jailbreaking poses a significant risk to the deployment of Large Language Models (LLMs) and Vision Language Models (VLMs). VLMs are particularly vulnerable because they process both text and images, creating broader attack surfaces. However, available resources for jailbreak detection are scarce, particularly in finance. To address this gap, we present FENCE, a bilingual (Korean-English) multimodal dataset for training and evaluating jailbreak detectors in financial applications. FENCE emphasizes domain realism through finance-relevant queries paired with image-grounded threats. Experiments with commercial and open-source VLMs reveal consistent vulnerabilities, with GPT-4o showing measurable attack success rates and open-source models displaying greater exposure. A baseline detector trained on FENCE achieves 99 percent in-distribution accuracy and maintains strong performance on external benchmarks, underscoring the dataset's robustness for training reliable detection models. FENCE provides a focused resource for advancing multimodal jailbreak detection in finance and for supporting safer, more reliable AI systems in sensitive domains. Warning: This paper includes example data that may be offensive.

Figures

Figures reproduced from arXiv: 2602.18154 by the authors.

**Figure 2.** Distribution of FENCE across 15 financial [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Workflow for constructing FENCE, consisting of three stages: (1) transforming benign queries [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: t-SNE visualization of harmful queries from existing datasets and FENCE using embeddings from the text-embedding-3-small model. FENCE’s queries exhibit a higher degree of semantic overlap with benign queries, suggesting that distinguishing harmful from benign inputs i…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

[1]

Introduction The rapid advancement of large language models (LLMs) has accelerated the development of Multi- modal Large Language Models (MLLMs), includ- ing Vision Language Models (VLMs) (Zhang et al., 2024). These models extend traditional LLMs by integrating multiple input modalities—such as im- ages, text, audio, and video—enabling a deeper understand...

arXiv 2024
[2]

Related Work Recent studies on multimodal jailbreaks have fo- cusedmainlyontext-drivenpromptinjections,while systematic taxonomies of attack types remain lim- ited. Building on prior datasets and attack strate- gies, we categorize jailbreak attempts in VLMs into two broad types: Text-based Attacks (TA), where harmfulcontentappearsintextwhileimagesarebe- n...

2024
[3]

We then introduce our proposed dataset, FENCE

Datasets In this section, we review existing open datasets related to VLM jailbreaking, with a particular focus on IA. We then introduce our proposed dataset, FENCE. The key distinctions between existing IA- focused benchmarks and our dataset are summa- rized in Table 1. 3.1. Open Datasets JailBreakV-28K JailBreakV-28K (Luo et al.,
[4]

Experiments To assess the utility of FENCE, we conduct two complementary experiments. The first evaluates its effectiveness as a benchmark for identifying jail- break vulnerabilities in VLMs, and the second ex- amines its utility as a training resource for harm- ful query detection. Rather than proposing new modelarchitectures,ourobjectiveistodemonstrate ...

2024
[5]

Conclusion As VLMs gain traction in financial services, ensur- ing their safety and robustness against jailbreak attacks has emerged as a critical challenge. In this work, we introduced FENCE, the first bench- mark dataset explicitly designed to evaluate jail- break vulnerabilities and support mitigation efforts in finance-focused multimodal systems. By i...
[6]

Limitations and Future Work While FENCE marks an important step toward advancing financial AI safety, several limitations remain. First, its current scale and domain scope are narrower than those of large, general- purpose benchmarks, and its bilingual focus (Ko- rean–English) may limit broader linguistic general- ization. Second, as FENCE is built from s...
[7]

While FENCE includes potentiallyharmfuloroffensiveexamplesgenerated for research purposes, we acknowledge the ethi- cal risks associated with creating and sharing such data

Ethics Statement The primary goal of this work is to highlight safety vulnerabilities in VLMs, particularly within the finan- cial domain, to promote responsible model devel- opment and deployment. While FENCE includes potentiallyharmfuloroffensiveexamplesgenerated for research purposes, we acknowledge the ethi- cal risks associated with creating and shar...
[8]

Bibliographical References Josh Achiam, Steven Adler, Sandhini Agar- wal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Al- tenschmidt,SamAltman,ShyamalAnadkat,etal

Show all 13 references

[11]

arXiv preprint arXiv:2303.08774

Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Ko...

2024 arXiv
[12]

Muhammad Salar Khan and Hamza Umer

Perception-guidedjailbreakagainsttext-to- image models.arXiv preprint arXiv:2408.10848. Muhammad Salar Khan and Hamza Umer. 2024. Chatgptinfinance: Applications,challenges,and solutions. Heliyon, 10(2):e24890. Seongho Kim, Hyuk-Jun Kwon, and Hyeob Kim

2024 arXiv
[13]

Do Anything Now

Mobile banking service design attributes for the sustainability of internet-only banks: A case study of kakaobank.Sustainability, 15(8). RazLapid,RonLangberg,andMosheSipper.2024. Open sesame! universal black-box jailbreaking of large language models. Applied Sciences, 14(16). ...

2024 arXiv
[2023]

money laundering,

and MM-SafetyBench (Liu et al., 2024b) include IA samples, they rely on a single fixed generation strategy, limiting diversity. JailBreakV- 28K (Luo et al., 2024) incorporates multiple tech- niques but still contains only 28.6% IA data. In contrast, FENCE provides a fully IA d...

2024
[2024]

is the largest dataset of its kind, compris- ing 28,000 adversarial test cases. It includes 2,000 base malicious queries expanded into 20,000 text- based jailbreak prompts using various LLM jail- break strategies, along with 8,000 image-based inputsderivedfromrecentMLLMattacks...

2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Introduction The rapid advancement of large language models (LLMs) has accelerated the development of Multi- modal Large Language Models (MLLMs), includ- ing Vision Language Models (VLMs) (Zhang et al., 2024). These models extend traditional LLMs by integrating multiple input modalities—such as im- ages, text, audio, and video—enabling a deeper understand...

arXiv 2024

[2] [2]

Related Work Recent studies on multimodal jailbreaks have fo- cusedmainlyontext-drivenpromptinjections,while systematic taxonomies of attack types remain lim- ited. Building on prior datasets and attack strate- gies, we categorize jailbreak attempts in VLMs into two broad types: Text-based Attacks (TA), where harmfulcontentappearsintextwhileimagesarebe- n...

2024

[3] [3]

We then introduce our proposed dataset, FENCE

Datasets In this section, we review existing open datasets related to VLM jailbreaking, with a particular focus on IA. We then introduce our proposed dataset, FENCE. The key distinctions between existing IA- focused benchmarks and our dataset are summa- rized in Table 1. 3.1. Open Datasets JailBreakV-28K JailBreakV-28K (Luo et al.,

[4] [4]

Experiments To assess the utility of FENCE, we conduct two complementary experiments. The first evaluates its effectiveness as a benchmark for identifying jail- break vulnerabilities in VLMs, and the second ex- amines its utility as a training resource for harm- ful query detection. Rather than proposing new modelarchitectures,ourobjectiveistodemonstrate ...

2024

[5] [5]

Conclusion As VLMs gain traction in financial services, ensur- ing their safety and robustness against jailbreak attacks has emerged as a critical challenge. In this work, we introduced FENCE, the first bench- mark dataset explicitly designed to evaluate jail- break vulnerabilities and support mitigation efforts in finance-focused multimodal systems. By i...

[6] [6]

Limitations and Future Work While FENCE marks an important step toward advancing financial AI safety, several limitations remain. First, its current scale and domain scope are narrower than those of large, general- purpose benchmarks, and its bilingual focus (Ko- rean–English) may limit broader linguistic general- ization. Second, as FENCE is built from s...

[7] [7]

While FENCE includes potentiallyharmfuloroffensiveexamplesgenerated for research purposes, we acknowledge the ethi- cal risks associated with creating and sharing such data

Ethics Statement The primary goal of this work is to highlight safety vulnerabilities in VLMs, particularly within the finan- cial domain, to promote responsible model devel- opment and deployment. While FENCE includes potentiallyharmfuloroffensiveexamplesgenerated for research purposes, we acknowledge the ethi- cal risks associated with creating and shar...

[8] [8]

Bibliographical References Josh Achiam, Steven Adler, Sandhini Agar- wal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Al- tenschmidt,SamAltman,ShyamalAnadkat,etal