{"paper":{"title":"High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models","license":"http://creativecommons.org/licenses/by-nc-nd/4.0/","headline":"A small share of high-entropy tokens during generation concentrates most adversarial influence in vision-language models.","cross_cats":["cs.LG"],"primary_cat":"cs.CV","authors_text":"Jing Zhang, Jinhong Ni, Mengqi He, Shu Zou, Xin Shen, Xinyu Tian, Zhaoyuan Yang","submitted_at":"2025-12-26T01:01:25Z","abstract_excerpt":"Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, as a measure of model uncertainty, is highly correlated with VLM reliability. While prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token equally contributes to model instability, we reveal that a small fraction (around 20%) of high-entropy tokens, in the evaluated representative open-source VLMs with diverse architectures, concentrates a disproportionate share of adversarial influence during autoregressive generation. We dem"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"a small fraction (around 20%) of high-entropy tokens, in the evaluated representative open-source VLMs with diverse architectures, concentrates a disproportionate share of adversarial influence during autoregressive generation","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That high-entropy tokens can be identified reliably during generation and that concentrating perturbations on them produces comparable semantic degradation to global attacks without requiring post-hoc selection or model-specific tuning that would invalidate the transferability claim.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"High-entropy tokens act as concentrated multimodal failure points in VLMs, enabling sparse Entropy-Guided Attacks that achieve 93-95% success and 30-38% harmful rates with cross-model transfer.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A small share of high-entropy tokens during generation concentrates most adversarial influence in vision-language models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"1f8077993881cfb232972a684346efbb0dc5737bc45ee675bedbd820673b7b48"},"source":{"id":"2512.21815","kind":"arxiv","version":3},"verdict":{"id":"ec215b08-5ced-4a3d-a80c-8d4baa9b633b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T19:26:18.378293Z","strongest_claim":"a small fraction (around 20%) of high-entropy tokens, in the evaluated representative open-source VLMs with diverse architectures, concentrates a disproportionate share of adversarial influence during autoregressive generation","one_line_summary":"High-entropy tokens act as concentrated multimodal failure points in VLMs, enabling sparse Entropy-Guided Attacks that achieve 93-95% success and 30-38% harmful rates with cross-model transfer.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That high-entropy tokens can be identified reliably during generation and that concentrating perturbations on them produces comparable semantic degradation to global attacks without requiring post-hoc selection or model-specific tuning that would invalidate the transferability claim.","pith_extraction_headline":"A small share of high-entropy tokens during generation concentrates most adversarial influence in vision-language models."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2512.21815/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"b318806ae64168b412aa3aabbb7615d7f56ade2d02e8bd896fead889c1bbbaef"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}