{"id":"9e800117-cf5d-406a-ab9d-5f62bd737f06","arxiv_id":"2504.14395","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Hydra, an agentic reasoning framework that iteratively queries and critiques multiple vision models, improves object-level hallucination accuracy and maintains performance under adversarial image attacks on four LVLMs and three benchmarks.","lead":"This paper introduces Hydra, a training-free agentic wrapper that makes vision-language models more accurate and more resistant to image tampering by repeatedly querying several separate vision models and critiquing their answers. Generalists may care because it offers a modular, no-fine-tuning route to improve VLM reliability in high-stakes applications, though the evidence is limited to COCO-style object-presence benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adversarial robustness claim depends on unverified integrity of auxiliary vision models; attack success on DETR, Paligemma, BLIPvqa, Llama-3.2 captioner is never measured (Section 3.1, Section 4.1.3).","rationale":"The reader's conditional verdict already identifies the integrity of auxiliary vision models as a key weakness; I agree and elevate it as the single most load-bearing concern rather than the COCO domain overlap. The paper's abstract and Section 5 claim that Hydra 'surpasses plug-in VLMs and SOTA dehallucination methods, even without explicit adversarial defenses, demonstrating enhanced robustness.' That claim requires the auxiliary suite to remain reliable under attack, because the agent is blind to the image and reasons only from textual outputs. The paper never tests this condition, and attacks are explicitly designed to transfer across visual encoders. This is not a matter of disagreement with consensus; it is an internally unverified assumption. The concern is addressable with a concrete experiment: measure attack success on each auxiliary model using the same perturbations and image sets. If the auxiliary models are robust, the claim gains support; if they degrade, the headline robustness is an artifact. Because the issue is empirical and fixable, the appropriate disposition remains conditional acceptance rather than rejection. I partially agree with the reader because they also raise a second concern about DETR's COCO training overlap, which I did not develop; that concern primarily affects the generality of clean-image hallucination gains, whereas the auxiliary-integrity issue directly affects the adversarial robustness contribution and is therefore more central.","tokens_in":13583,"tokens_out":2831,"duration_ms":26311,"concrete_test":"Apply the exact Adversarial Illusions (epsilon=16/255, ImageBind surrogate) and AttackVLM (epsilon=8/255, Unidiffuser surrogate) perturbations to the same POPE, MME-Existence, and AMBER image samples used in the paper. Then evaluate each auxiliary model on the clean versus attacked images: DETR object-presence accuracy on the queried object classes, Paligemma and BLIPvqa VQA accuracy on the benchmark questions, and Llama-3.2 captioner's object recall and hallucination rate. If any auxiliary model's object-presence or VQA accuracy drops by more than 10 percentage points under attack, or if the captioner's CHAIR/Hal metrics worsen significantly, Hydra's verification signals are compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Hydra is adversarially robust without explicit defenses rests on the assumption that the auxiliary vision-language suite continues to provide trustworthy textual evidence under attack. Hydra's agent never sees the image; it only reads textual outputs from DETR, Paligemma, BLIPvqa, and the Llama-3.2 captioner (Section 3.1, Section 4.1.3). The paper's only justification is that 'a diverse suite lowers the risk of compromising to the adversarial attacks compared to a single model' (Section 3.1), but no experiment measures attack success on any auxiliary model. The attacks used, Adversarial Illusions and AttackVLM, are designed to transfer across models with similar visual encoders (Section 2.1, Section 4.1.3); DETR uses a ResNet backbone, Paligemma uses SigLIP, BLIPvqa uses a ViT, and Llama-3.2-Vision uses a CLIP-style encoder. Transferability of adversarial perturbations across encoders is a known phenomenon, and AttackVLM explicitly uses transfer- and query-based strategies. If the perturbation fools DETR into missing or adding objects, or corrupts Paligemma/BLIP captions, Hydra's cross-model verification would reinforce false information rather than correct it. The paper reports no oracle-like control, no per-model attack success rates, and no analysis of how often the agent is misled by compromised auxiliary outputs. Without this evidence, Hydra's apparent robustness could be an artifact of an un-attacked oracle suite, not a property of the agentic framework itself. This is the most load-bearing gap because it directly undermines the headline contribution of unified adversarial robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hydra, a training-free agentic framework that wraps plug-in vision-language models (VLMs) and refines their outputs through an iterative Action-Critique Loop. Hydra queries a suite of additional vision models (DETR, Paligemma, BLIPvqa, and Llama-3.2-Vision-Instruct) using only their textual outputs, then critiques and reconciles evidence from multiple sources to reduce object-level hallucinations and resist adversarial perturbations. The authors evaluate Hydra on four VLMs (mPLUG-Owl, LLaVA-1.5, MiniGPT-4, Qwen-VL-Chat), three hallucination benchmarks (POPE, MME-Existence, AMBER-Generative), two adversarial attacks (Adversarial Illusions, AttackVLM), and two preprocessing defenses (JPEG compression, feature squeezing). The main claimed results are consistent improvements in accuracy and F1 on POPE subsets across all four base VLMs, and a claim that Hydra maintains higher accuracy than plug-in VLMs and existing dehallucination methods even when the input images are adversarially perturbed.","tokens_in":13924,"tokens_out":3257,"duration_ms":30413,"significance":"If the adversarial-robustness claim were fully supported, the paper would make a meaningful contribution: it would show that a model-agnostic, training-free wrapper can simultaneously mitigate object-level hallucinations and resist joint-embedding adversarial attacks. The hallucination-mitigation results on POPE are consistent and reasonably convincing, and the integration of agentic reasoning with multi-model verification is a sensible research direction. The paper also usefully evaluates the interaction of preprocessing defenses with a hallucination-correction pipeline. However, the adversarial-robustness claim currently rests on an unexamined assumption that the auxiliary vision models remain reliable under attack, which is load-bearing because the agent never sees the image. That gap, plus the selective reporting of results, prevents the paper from fully substantiating its headline claim.","major_comments":[{"comment":"The adversarial-robustness claim is not adequately supported because the paper never measures attack success on the auxiliary vision models. Hydra's agent relies entirely on textual outputs from DETR, Paligemma, BLIPvqa, and the Llama-3.2 captioner (Section 4.1.3), and Section 3.1 asserts that a diverse suite 'lowers the risk of compromising to the adversarial attacks' without providing any per-model attack success rates, oracle controls, or analysis of how often the agent is misled by compromised auxiliary outputs. Since the injected attacks target the joint embedding space and are known to transfer across encoders, the observed robustness could be an artifact of the auxiliary models being unintentionally robust rather than a property of Hydra's reasoning. Please report attack success on each auxiliary model, or provide a control condition in which the auxiliary outputs are intentionally corrupted, to demonstrate that Hydra's cross-model verification is actually the source of robustness.","section":"3.1, 4.1.3"},{"comment":"The evidence for adversarial robustness is presented on a selective subset of the claimed evaluation grid. Figure 3 shows only LLaVA-1.5 on POPE-Popular under Adversarial Illusions and Qwen-VL-Chat on POPE-Adversarial under AttackVLM; Tables 3–5 omit LLaVA-1.5 entirely and cover only two of the four base VLMs on MME and AMBER. The abstract claims evaluation across four VLMs, three benchmarks, and two attacks, but the reader cannot verify the full cross product. Please provide complete results for all model–benchmark–attack combinations, or clearly state in the text which combinations were omitted and why.","section":"Figure 3, Tables 3–5"},{"comment":"The design conflates the agent backbone with one of the evidence sources: Llama-3.2-Vision-Instruct serves both as Hydra's reasoning agent and as the captioning model in the visual-language suite. This creates a possible failure mode in which the model used for reasoning and the model used for verification share the same vulnerabilities, weakening the claim that cross-model verification provides diverse independent evidence. The paper should either use a different captioning model or explicitly discuss how this shared-model dependency affects the robustness and diversity of the evidence suite.","section":"4.1.3"},{"comment":"The paper reports that performance sometimes improves after attack, which it labels an 'open question.' This may be an artifact of small sample sizes (e.g., 50 AMBER images and 300 POPE questions per subset) combined with the absence of statistical significance tests. Given that the main conclusions rely on differences of a few percentage points, please report confidence intervals or significance tests, and explain the post-attack improvement with a concrete analysis rather than leaving it as an unexplained observation.","section":"4.2.4, Tables 4 and 5"}],"minor_comments":[{"comment":"The citation for the MME benchmark is incorrect: [19] is a survey of multimodal large language models, not the MME evaluation paper by Fu et al. Please replace it with the appropriate reference.","section":"References, [19]"},{"comment":"The Action-Critique Loop is described only at a high level. Please provide the concrete prompt templates, the iteration-limit value, the consistency threshold used in the critiques, and the exact decision rule for when the agent finalizes an answer or enters the loop. This is needed for reproducibility.","section":"3.1"},{"comment":"The paper does not mention whether code or model configurations will be released. Given the large number of components (four base VLMs, three auxiliary models, two attacks, two defenses), a reproducibility appendix with the exact inference scripts and random seeds would strengthen the paper.","section":"4.1.3"},{"comment":"There is a typo in the introduction: 'approches' should be 'approaches.' Additionally, several reference entries in the bibliography appear to lack venue information or have inconsistent formatting (e.g., [1], [19]); a thorough proofread of the reference list is recommended.","section":"1"},{"comment":"The Limitations section mentions increased inference time but provides no quantitative estimate. Since Hydra involves iterative reasoning and multiple model queries, a brief measurement of wall-clock time or number of iterations per query would help readers gauge practical applicability.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The core hallucination-mitigation results appear solid, but the adversarial-robustness claim needs substantial additional evidence. I would recommend the editor request a revision that either provides the missing auxiliary-model attack analysis or substantially softens the robustness claims. The selective reporting in Figure 3 and Tables 3–5 is a concern for a paper whose abstract promises evaluation across the full grid. The paper also cites the MME benchmark incorrectly, which may indicate a broader citation-accuracy issue worth a quick check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on Hydra. The clean-image numbers are the real story: across four LVLMs, Hydra lifts POPE accuracy dramatically—mPLUG-Owl goes from 54 to 94.7 on the Random subset—and it beats Woodpecker and LogicCheckGPT when those are re-implemented with the same Llama-3.2 backbone. That is a solid, reproducible-looking result for a training-free post-hoc hallucination corrector. The agentic loop with a detector and two auxiliary VLMs is a new combination, and it's clearly described. I believe the hallucination-mitigation contribution stands.\n\nThe soft spot is exactly where the stress test points. The headline claim about adversarial robustness without defenses rests on the assumption that the auxiliary suite—DETR, Paligemma, BLIPvqa, and the Llama-3.2 captioner—still produces reliable text under attack, because the agent never sees the image. The paper never measures attack success on those models. If the perturbation transfers to DETR or the captioner, the cross-model verification could harden a false conclusion rather than correct it. Diversity of encoders is a plausible argument, but it is not evidence. This is not a fatal flaw; it is a missing experiment, and an easy one to run. The authors should report per-model attack success rates and an oracle control where the auxiliary suite is replaced by ground-truth labels.\n\nA few smaller issues: no error bars or significance tests for the headline comparisons; only a selective subset of the full model x attack x defense grid appears in Figure 3 and Tables 3-5; and no code or prompts are released. I also would have liked a sentence acknowledging that DETR is a COCO-trained closed-set detector and all three benchmarks use COCO images and classes—not a disqualifier, but it should be on the table.\n\nThe limitations section is honest about inference cost and scope, which makes me think the authors know what they have and what they don't. This is a paper worth sending to referees—conditional acceptance, with the auxiliary-model attack analysis and full result tables as required revisions.","headline":"Hydra's clean dehallucination results are strong and the agentic combination is new, but the adversarial-robustness headline depends on auxiliary models that are never put under attack.","tokens_in":14501,"tokens_out":2910,"would_cite":true,"duration_ms":26147,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic verification curbs VLM hallucinations and survives adversarial image attacks.","keywords":["agentic reasoning","vision-language models","hallucination mitigation","adversarial robustness","object-level hallucination","joint embedding space attacks","chain-of-thought reasoning","in-context learning"],"falsifier":"Run Adversarial Illusions and AttackVLM directly against the auxiliary suite (DETR, Paligemma, BLIPvqa, and the captioning model) and show that their object-presence answers shift on attacked images; if those shifts propagate into the agent's final answers, Hydra's robustness would not exceed the weakest auxiliary model. A cleaner test: adversarially attack the auxiliary models alone and measure whether Hydra's final accuracy degrades proportionally.","tokens_in":13386,"feed_emoji":"🛡️","tokens_out":6502,"duration_ms":51587,"temperature":0.7,"pith_summary":"The paper proposes Hydra, a training-free wrapper that places a large language model agent around any plug-in vision-language model (VLM). The agent repeatedly asks the VLM and a small suite of auxiliary vision models to describe or verify image content, critiques the answers, and refines its final response. Hydra targets object-level hallucinations—false claims that an object is present—and aims to keep outputs correct when images are adversarially perturbed in the joint embedding space. On POPE, MME-Existence, and AMBER-Generative, across four LVLMs, the paper reports that Hydra outperforms both the bare models and two post-hoc dehallucination methods, on clean and attacked inputs, with no adversarial training.","feed_headline":"Agentic loop cuts VLM hallucinations and resists adversarial attacks","feed_subtitle":"A training-free agent verifies images through several vision models, beating plug-in VLMs and post-hoc methods on clean and attacked inputs.","key_machinery":"The load-bearing mechanism is the Action-Critique Loop, an iterative cycle in which a single LLM agent moves between an Action Space (issuing queries to a vision-language suite) and a Critique Space (assessing consistency and factual correctness), with a shared memory of retrieved information. Five subtasks—initial perceptual querying, adaptive model critique, critique-driven decision-making, attribute-based inquiry formulation, and cross-model object discovery—turn the loop into a complete verification pipeline. The auxiliary suite's diversity is what the paper credits for robustness: because the agent fuses textual evidence from models with different visual encoders, an attack that fools one model need not fool the others.","core_discovery":"Hydra's central claim is that a single agentic loop can unify two problems usually treated separately: resisting adversarial perturbations and suppressing intrinsic hallucinations. The agent has no direct access to the image; it reasons only over textual outputs from the plug-in VLM and from auxiliary models. When initial object-presence judgments conflict, the agent generates attribute-based questions and queries multiple models again, iterating until the critiques agree. The paper reports gains such as raising one model's POPE-Random accuracy from 54% to 94.7%, sustaining above 90% accuracy on two other models, and retaining the highest accuracy after two adversarial attacks, without any explicit defense.","pith_inferences":["The paper does not report attack success on the auxiliary suite; a natural extension would be adaptive attacks specifically targeting DETR, Paligemma, and BLIPvqa, since Hydra's claimed robustness would collapse if those models are also fooled.","Because POPE, MME, and AMBER all draw from COCO images and classes, DETR's strong performance may partly reflect closed-set familiarity with those categories; testing on out-of-distribution object sets would clarify whether the mechanism generalizes.","The agent's reliance on attribute-based questions suggests the loop could extend beyond object-level verification to attribute- and relationship-level hallucinations if the auxiliary models can answer those question types reliably.","Hydra's inference cost grows with each iteration, so a test-time-compute versus accuracy trade-off curve, rather than a single accuracy number, would be the more informative deployment comparison."],"forward_implications":["If Hydra's results hold, a training-free wrapper can improve an LVLM's factual consistency on object-presence questions without modifying the LVLM's weights.","Adversarial robustness and hallucination mitigation can be addressed by the same mechanism: cross-model textual verification makes the system harder to fool in both settings.","Small auxiliary models under 3B parameters can provide large robustness gains, which matters for deployment under computational constraints.","Applying standard preprocessing defenses like JPEG compression and feature squeezing changes Hydra's trade-off: hallucination drops further but object coverage falls, so the choice of defense must consider the task.","Hydra's interpretable critiques give users a human-readable rationale for each answer, a direct corollary of the agent's structured decision trail."],"supporting_citations":[{"why":"Defines AttackVLM, the transfer- and query-based joint-embedding attack used to test Hydra under adversarial conditions.","marker":"[4]"},{"why":"Defines Adversarial Illusions, the white-box surrogate-based attack that targets nonexistent objects and is used in the evaluation.","marker":"[5]"},{"why":"Woodpecker is the post-hoc hallucination-correction baseline Hydra is compared against; Hydra's loop extends its object-verification idea.","marker":"[6]"},{"why":"LogicCheckGPT is the second post-hoc dehallucination baseline, providing the self-consistency comparison point.","marker":"[7]"},{"why":"POPE supplies the object-presence questions and Accuracy/F1/Yes-Ratio metrics used for most of the headline results.","marker":"[18]"},{"why":"MME's Existence subset provides the Accuracy, Accuracy+, and Total Score metrics for binary presence evaluation.","marker":"[19]"},{"why":"AMBER's Generative subset provides the caption-level CHAIR, Cover, Hallucination, and Cognition metrics.","marker":"[20]"},{"why":"DETR, the closed-set detector in Hydra's auxiliary suite, contributes the object lists the agent critiques against.","marker":"[25]"},{"why":"Paligemma serves as the auxiliary vision-language model in the suite whose textual answers are fused into the agent's evidence.","marker":"[26]"},{"why":"BLIPvqa is the auxiliary VQA model in the suite, providing additional image-grounded verification.","marker":"[27]"}],"fun_headline_variants":["Agentic loop unifies VLM hallucination and attack defense","Training-free agent boosts VLM robustness against attacks and errors","Hydra agent thwarts adversarial attacks and halts hallucinations","One agentic loop fixes VLM hallucinations and adversarial snaps","Cross-model agent verifies images to cut VLM errors and attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Hydra's reliability rests on the auxiliary vision models remaining trustworthy under attack, since the agent never sees the image and can only read their textual reports.","fun_headline_variants_meta":{"raw":{"variants":["Agentic loop unifies VLM hallucination and attack defense","Training-free agent boosts VLM robustness against attacks and errors","Hydra agent thwarts adversarial attacks and halts hallucinations","One agentic loop fixes VLM hallucinations and adversarial snaps","Cross-model agent verifies images to cut VLM errors and attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3020,"prompt_tokens":949,"completion_tokens":2071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1987}},"tokens_in":565,"tokens_out":2071,"duration_ms":13752,"temperature":1.0,"reasoning_tokens":1987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:49:17.324820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Adversarial Illusions and AttackVLM directly against the auxiliary suite (DETR, Paligemma, BLIPvqa, and the captioning model) and show that their object-presence answers shift on attacked images; if those shifts propagate into the agent's final answers, Hydra's robustness would not exceed the weakest auxiliary model. A cleaner test: adversarially attack the auxiliary models alone and measure whether Hydra's final accuracy degrades proportionally.","supporting_citations":[{"cited_title":"Adversarial illusions in{Multi-Modal} embeddings","cited_arxiv_id":null,"evidence_quote":"Defines Adversarial Illusions, the white-box surrogate-based attack that targets nonexistent objects and is used in the evaluation."}],"review_version":1}