{"id":"4a62c39b-dace-49fc-be9f-2db3ce55590d","arxiv_id":"2504.13690","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces TextVQA-C and GQA-C, showing that LLaVA 1.5 loses text-question accuracy most under blur and snow and object-question accuracy most under frost and impulse noise, though only one model is tested.","lead":"This paper tests how one vision-language model, LLaVA 1.5, responds to 19 types of image corruption on two visual-question-answering datasets, and introduces corrupted versions of TextVQA and GQA. The results show different failure patterns for text reading versus object reasoning, but the broad claims rest on a single model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported vulnerability ranking is not reproducible from the paper: Section 5.3 itself flags non-monotonic 'anomalous' severity points, yet no error bars, raw tables, benchmarks, or code are released, so the central empirical claim is currently unverified.","rationale":"The reader identified the generalization from a single VLM to all transformer-based VLMs as the weakest assumption. I agree that is a real limitation, but the more immediate load-bearing concern is that the empirical foundation for even the single-model claim is not verifiable. The manuscript flags its own non-monotonic severity curves in Section 5.3 ('anomalous value at severity 4 in GQA', 'anomalous value at severity 5' for snow), which is direct evidence that the reported point estimates contain noise or an artifact; without error bars, raw numbers, datasets, or code, the ranking's central contrasts cannot be checked. This is an internal verifiability problem rather than a disagreement with any field consensus. The reader's verdict of CONDITIONAL is reasonable, but the correct disposition under the reviewer guidelines is UNVERDICTED: the paper needs a reproducibility release before the empirical claims can be accepted, whereas the generalization concern alone would only justify CONDITIONAL. The proposed concrete check directly tests whether the central contrast survives once the data and pipeline are available.","tokens_in":14894,"tokens_out":2553,"duration_ms":21206,"concrete_test":"Release the exact corruption pipeline plus the raw per-sample accuracy tables for the 5,000 TextVQA-C and 12,578 GQA-C images, and recompute the severity curves for LLaVA 1.5 with bootstrap 95% confidence intervals; then check three specific claims: (1) whether the frost severity-4 rise in GQA and the snow severity-5 rise in TextVQA (Section 5.3) persist under the released code, (2) whether the 'Glass Blur most robust in TextVQA but steepest decline in GQA' contrast exceeds the interval width, and (3) whether the 'Zoom Blur least robust in TextVQA yet better tolerated in GQA' contrast exceeds the interval width. If the anomalies vanish or the task contrasts fall within noise, the central ranking is undermined; if the contrasts persist with tight intervals, the ranking is confirmed.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central claim is an empirical ranking: TextVQA degrades most under blur/snow, GQA under frost/impulse noise, explained by transformer low-frequency bias. For this ranking to be load-bearing, the underlying measurements must be trustworthy and the comparisons statistically meaningful. Section 4 reports only single accuracy values: no repeated runs, no standard errors, no error bars across samples or runs. Section 5 reports per-severity accuracies to the nearest percentage point, and Section 5.3 explicitly describes 'anomalous' values (frost in GQA rising from severity 3 to 4; snow in TextVQA rising from severity 4 to 5). A fixed deterministic model evaluated on a fixed sample would not produce non-monotonicity unless there is finite-sample noise or an implementation artifact. TextVQA-C, GQA-C, the evaluation code, and the raw accuracy table are not released, so one cannot distinguish a genuine task-level pattern from noise or a corruption-pipeline bug. The frequency-domain explanation (Sections 5.1-5.5, 6) is asserted but never tested by measuring the frequency content of corrupted images or model representations; it is an interpretation, not a derivation. The most load-bearing gap is therefore the trustworthiness of the reported numbers, which precedes and undercuts the generalization concern: if the LLaVA 1.5 curves cannot be independently confirmed, no claim about transformer-based VLMs can rest on them.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first systematic robustness study of a vision-language model under common image corruptions. The authors construct two new benchmarks, TextVQA-C and GQA-C, by applying the 19 corruption types from ImageNet-C at five severity levels to subsets of TextVQA and GQA, and they report LLaVA 1.5 accuracy on both tasks. Their central empirical claim is a task-specific vulnerability ranking: scene text understanding (TextVQA) degrades most under blur and snow, while object reasoning (GQA) is most sensitive to frost and impulse noise. They further propose that these patterns are explained by a low-frequency processing bias of transformer-based vision encoders.","tokens_in":15282,"tokens_out":3710,"duration_ms":34123,"significance":"If the empirical ranking and the benchmark datasets were fully substantiated, this would be a useful contribution to VLM robustness evaluation: TextVQA-C and GQA-C address a real gap, the use of concise answers reduces evaluation ambiguity, and the task-level comparison between OCR-centric and object-centric reasoning is a sensible design. The paper also makes a falsifiable architectural claim about frequency bias that could inform future robustness work. However, the current evidence base is narrow: results come from a single model with no error bars or repeated runs, the proposed benchmarks and code are not released, and the frequency-domain mechanism is asserted rather than measured. The strengths of the paper—the benchmark construction and the clear articulation of a task-dependent robustness map—are real, but they currently outrun the evidence presented.","major_comments":[{"comment":"The experiments evaluate only LLaVA 1.5, yet the abstract, Section 5.5, and the conclusion generalize to 'transformer-based VLMs' and 'transformer models.' LLaVA 1.5's CLIP ViT-L/14 encoder, linear projection, and Vicuna 7B decoder represent one point in a large architectural space, so the reported vulnerability ranking cannot support claims about transformer-based VLMs as a class. Either evaluate additional VLMs with different encoders, projection layers, and decoders, or restrict every claim in the abstract and conclusion to LLaVA 1.5.","section":"Sections 4.1 and 5.5"},{"comment":"The central empirical ranking is reported as single accuracy values with no repeated runs, confidence intervals, or error bars. Section 5.3 explicitly flags non-monotonic 'anomalous' severity points, including frost in GQA rising from severity 3 to 4 and snow in TextVQA rising from severity 4 to 5. A deterministic model evaluated on a fixed test set should not produce unexplained non-monotonicity unless there is finite-sample noise or a pipeline artifact. The paper must release per-sample results and provide bootstrap or paired statistical comparisons across severities to establish that the reported ranking is not noise.","section":"Sections 4.2 and 5.3"},{"comment":"The low-frequency-bias explanation is asserted rather than tested. The paper presents no spectral analysis of the corrupted images and no measurements of the model's internal representations, so the frequency-domain account is an interpretation consistent with selected results rather than evidence for the stated mechanism. Add quantitative support, such as Fourier power spectra of each corruption type, controlled high-pass/low-pass filtering experiments, or attention/saliency analyses, or explicitly downgrade the frequency explanation to a hypothesis.","section":"Sections 5.1-5.5 and 6"},{"comment":"The two proposed benchmarks, TextVQA-C and GQA-C, are the paper's main reusable contribution, but no dataset download link, evaluation code, or full raw accuracy table is provided. Without these artifacts, the benchmarks cannot be used by the community and the reported numbers cannot be independently checked. The datasets, evaluation code, and complete per-corruption/per-severity accuracy tables should be released as a condition for the benchmark contribution to be verifiable.","section":"Section 4.2"}],"minor_comments":[{"comment":"There is a duplicated and grammatically broken sentence: 'Both methods, Flamingo, and Frozen adopt the language modeling loss, where the language model generates texts conditioned on the image. have demonstrated the benefits...' This should be rewritten.","section":"Section 2.3"},{"comment":"There are several typographical issues, including 'W eather' and 'F rost' in Section 3.3 and 'naturally occuring' in Section 2.4; these should be corrected.","section":"Sections 3.3, 2.4"},{"comment":"The accuracy plots are difficult to verify because the axis labels and legends are not legible in the provided figures, while the text cites values such as 52.7% and 51.6%; ensure figures have readable axes and legends and include the exact numeric values in a supplementary table.","section":"Figures 4-7"},{"comment":"The evaluation protocol is underspecified: the exact matching procedure for TextVQA (normalization, synonyms, or the official TextVQA metric) and the sampling method and seed for the 5,000 TextVQA and 12,578 GQA subsets should be stated.","section":"Section 4.3"},{"comment":"The adversarial-attack paragraph cites several papers that are not connected to the corruption-robustness analysis; this material should be trimmed or explicitly tied to the paper's scope.","section":"Section 2.4"},{"comment":"The text says 'the authors generate multi-turn conversation data' when describing LLaVA's training; this should be rephrased as 'the LLaVA [27] authors,' and the LLaVA-1.5 paper should be cited specifically since the evaluated version is 1.5.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable preliminary study, but the headline claims currently exceed the evidence. I would encourage the editor to treat release of the datasets, evaluation code, and raw accuracy tables, together with either additional models or explicitly narrowed claims, as conditions for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward ImageNet-C-style corruption study applied to two VQA datasets with a single model (LLaVA 1.5). The new artifacts are the two corrupted datasets, TextVQA-C and GQA-C, and the accuracy curves. But the central ranking is not reproducible as submitted: no error bars, no raw numbers, no code or data release, and the paper itself reports non-monotonic severity points.\n\nWhat's new and good: The idea of testing VLM robustness on text-focused vs object-focused VQA is sensible. Using TextVQA and GQA is a reasonable choice. The paper correctly identifies that these tasks stress different visual skills. The descriptive ranking (blur/snow hit text, frost/impulse hit object reasoning) is plausible and, if confirmed with proper statistics, would be a useful data point.\n\nSoft spots: The biggest issue is trustworthiness of the numbers. Section 5.3 calls out \"anomalous\" rises at severity 4-5 for frost and snow. Without repeated runs or confidence intervals, you can't tell if that's finite-sample noise or a bug in the corruption pipeline. The paper generalizes from LLaVA 1.5 to \"transformer-based VLMs\" with no comparison across architectures. That's a clear overclaim. The frequency-domain explanation is asserted, not tested; they never measure frequency content of corruptions or model representations. Also, the related work has duplicated sentences and some citations that don't belong (e.g., refs 4 and 8 look off-topic). Minor compared to the reproducibility gap.\n\nThe central empirical pattern might hold up. But right now it's a plausible, unverified observation. The paper's own text flags the anomalies, so it's not hiding them—but it doesn't resolve them either.\n\nWho it's for: Anyone working on VLM robustness benchmarks. A serious referee could push the authors to release the corrupted datasets, add repeated runs and error bars, and temper the generalization. That would turn it into a solid benchmark contribution. Without that, it's a preliminary study.\n\nRecommendation: Send to peer review with a clear request for major revision focused on reproducibility and scope. It deserves referee time because the benchmark artifacts are potentially reusable and the empirical pattern is checkable. If the authors refuse to release data/code, that alone should sink it.","headline":"Useful but unreproducible corruption benchmark for VLMs—single model, no error bars, no release; ranking could be real but needs verification.","tokens_in":15679,"tokens_out":1984,"would_cite":false,"duration_ms":18006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that a transformer-based vision-language model fails in task-specific ways: blur and snow most severely degrade scene-text understanding, while frost and impulse noise most severely degrade object reasoning, and both…","keywords":["vision-language models","robustness","common corruptions","ImageNet-C","TextVQA-C","GQA-C","LLaVA 1.5","low-frequency bias"],"falsifier":"Run TextVQA-C and GQA-C on a second vision-language model with a different vision encoder, such as a BLIP-2-style model with a Q-Former or a model built on a convolutional encoder, and compare the per-corruption rankings with LLaVA 1.5's; if blur and snow do not remain the worst for text tasks, or frost and impulse noise do not remain the worst for object reasoning, the claimed task-specific map and its frequency explanation fail. A more direct check is to measure whether a corruption's spectral content, classified as high-frequency versus low-frequency disruption, predicts the observed accuracy drops in the way the paper claims.","tokens_in":14693,"feed_emoji":"📉","tokens_out":7630,"duration_ms":65948,"temperature":0.7,"pith_summary":"This paper sets out to map how a vision-language model's accuracy degrades when images are hit with the 19 common corruptions from ImageNet-C, at five severity levels. It contributes two new benchmarks, TextVQA-C for reading text in images and GQA-C for object-based reasoning, and runs 190 evaluations on LLaVA 1.5. The central finding is that failure modes are task-dependent: blur and snow hurt text reading most, while frost and impulse noise hurt object reasoning most, with brightness and JPEG compression doing little harm. The paper explains these rankings by the transformer's bias toward low-frequency information, arguing that corruptions which preserve low-frequency structure are tolerated and corruptions that inject high-frequency disruption are not. If correct, the result gives system builders a task-specific robustness map and points to the architectural bias a more robust vision-language model would need to correct.","feed_headline":"Blur breaks text-reading AI models; frost breaks object reasoning","feed_subtitle":"Which corruptions break LLaVA 1.5's tasks, and why; the answer is a low-frequency bias.","key_machinery":"The load-bearing object is the ImageNet-C corruption suite, 19 corruptions in four categories (noise, blur, weather, digital) at five severity levels, applied to 5,000 TextVQA and 12,578 GQA samples to form TextVQA-C and GQA-C. The argument is carried by ranking accuracy across the five severity levels and matching those rankings to a frequency-domain classification of each corruption: self-attention acts as a weighted average, giving the transformer a low-frequency bias, so corruptions that preserve low-frequency content should degrade little and those that inject high-frequency patterns should degrade a lot. The evaluated model is LLaVA 1.5, a CLIP ViT-L/14 encoder feeding linear-projected visual tokens into a Vicuna 7B decoder.","core_discovery":"On the paper's own terms, the discovery is that corruption robustness in a transformer vision-language model is not a single property; it depends on the task and on the spectral footprint of the corruption. Scene text understanding (TextVQA) degrades most under blur and snow corruptions, while object reasoning (GQA) is most sensitive to frost and impulse noise. The offered mechanism is that self-attention acts as a weighted average, emphasizing low frequencies, so corruptions like JPEG compression and brightness that preserve low-frequency structure cause small drops, whereas impulse noise, frost, snow, and blur that disrupt or replace high-frequency detail cause large drops. The paper presents TextVQA-C and GQA-C as reusable benchmarks, with clean baselines of 57.5% and 62.5%, so other models can be compared on the same degradation axis.","pith_inferences":["Beyond the paper: swapping the vision encoder for a convolutional backbone, while keeping the same decoder and instruction data, would test whether the low-frequency-bias explanation is causal; if the vulnerability rankings invert or flatten, the encoder's spectral behavior rather than the task is the controlling factor.","Beyond the paper: the severity-level anomalies, such as an accuracy rise from severity 4 to 5 for snow in TextVQA and frost in GQA, suggest that single-word exact-match scoring may be masking answer-pattern effects; a per-answer analysis would show whether these are evaluation noise or systematic reversion.","Beyond the paper: the spectral account implies a concrete, untested remedy: augmenting training with high-frequency-disrupting corruptions, or adding a high-pass branch to the visual encoder, should differentially improve the tasks that currently fail most."],"forward_implications":["TextVQA-C and GQA-C become reusable benchmarks: any vision-language model can be scored on the same 19 corruptions at five severities, producing a task-specific robustness map rather than a single average.","Deployments that rely on reading text in images should treat blur and snow as primary failure risks, while object-reasoning deployments should treat frost and impulse noise as primary risks.","The claimed low-frequency bias predicts that improving text-task robustness requires protecting mid-to-high frequency bands such as character edges, while object reasoning can tolerate more low-frequency-only degradation.","Frequency-preserving corruptions such as JPEG compression and brightness change are the least likely to surprise a deployed system, and high-frequency disruptions are the ones to defend against.","Designing a more robust vision-language model would mean correcting the encoder's low-frequency bias, especially for applications that need fine-grained visual detail."],"supporting_citations":[{"why":"Defines the 19 corruption types and five severity levels from which TextVQA-C and GQA-C are built; the paper's whole evaluation axis rests on it.","marker":"[1]"},{"why":"Supplies the CLIP ViT-L/14 vision encoder whose self-attention is invoked as the source of the low-frequency bias.","marker":"[2]"},{"why":"Defines LLaVA 1.5, the only model evaluated in the study.","marker":"[27]"},{"why":"Supplies Vicuna, the language decoder of LLaVA 1.5 whose instruction-following behavior is tested by the VQA tasks.","marker":"[26]"},{"why":"Supplies LLaMA 2.0 7B, the base model from which Vicuna is fine-tuned.","marker":"[25]"}],"fun_headline_variants":["VLM corruption robustness varies by task: blur hits text, frost hits reasoning","Low-frequency bias drives VLM's differential sensitivity to corruptions","Why blur breaks your VLM's reading but frost breaks its reasoning","New corruption benchmarks TextVQA-C and GQA-C for VLM evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLaVA 1.5, with a CLIP ViT-L/14 encoder, a linear projection, and a Vicuna 7B decoder, represents transformer-based vision-language models generally; the paper evaluates no other model, so if other architectures, training sets, or projection layers behave differently, the vulnerability rankings and the low-frequency-bias explanation do not generalize.","fun_headline_variants_meta":{"raw":{"variants":["VLM corruption robustness varies by task: blur hits text, frost hits reasoning","Low-frequency bias drives VLM's differential sensitivity to corruptions","Why blur breaks your VLM's reading but frost breaks its reasoning","New corruption benchmarks TextVQA-C and GQA-C for VLM evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1828,"prompt_tokens":891,"completion_tokens":937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":859}},"tokens_in":507,"tokens_out":937,"duration_ms":8837,"temperature":1.0,"reasoning_tokens":859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:01:45.081736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TextVQA-C and GQA-C on a second vision-language model with a different vision encoder, such as a BLIP-2-style model with a Q-Former or a model built on a convolutional encoder, and compare the per-corruption rankings with LLaVA 1.5's; if blur and snow do not remain the worst for text tasks, or frost and impulse noise do not remain the worst for object reasoning, the claimed task-specific map and its frequency explanation fail. A more direct check is to measure whether a corruption's spectral content, classified as high-frequency versus low-frequency disruption, predicts the observed accuracy drops in the way the paper claims.","supporting_citations":[{"cited_title":"Benchmarking neural network robustness to common corruptions and perturbations,","cited_arxiv_id":null,"evidence_quote":"Defines the 19 corruption types and five severity levels from which TextVQA-C and GQA-C are built; the paper's whole evaluation axis rests on it."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP ViT-L/14 vision encoder whose self-attention is invoked as the source of the low-frequency bias."},{"cited_title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,","cited_arxiv_id":null,"evidence_quote":"Supplies Vicuna, the language decoder of LLaVA 1.5 whose instruction-following behavior is tested by the VQA tasks."}],"review_version":1}