{"id":"839b0684-2432-43fe-98eb-1c019f95a64d","arxiv_id":"2608.07742","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"BRUCE is a new benchmark with metrics RCI and T-RCI that quantify how vision-language model reasoning for chemistry and math degrades as image corruption escalates.","lead":"This paper introduces BRUCE, a benchmark that tests how well vision-language models keep their scientific reasoning when input images are corrupted by blur, low contrast, or similar distortions. It also proposes two new metrics, RCI and T-RCI, for measuring how quickly reasoning quality drops as corruption gets worse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RCI/T-RCI definitions are unspecified, so the central 'rapidity' claim may depend on arbitrary severity-schedule choices.","rationale":"The reader's weakest assumption concerned the choice of corruption types, severity levels, and aggregation rules. My concern sharpens one component of that: the aggregation rules for RCI and T-RCI are the most load-bearing piece because the entire central claim is about the metrics quantifying 'how rapidly' degradation occurs. If the metrics are not invariant to the severity schedule or the traversal order, then the benchmark's comparative rankings across models and domains would not support the claims made. The abstract omits the metric formulas, so the risk is currently unassessable from the abstract alone; the proposed test requires the full paper's equations and a synthetic experiment. This does not conflict with the UNVERDICTED verdict — it strengthens the case that the paper cannot be judged without the methodology. I therefore recommend leaving the reader's verdict unchanged, while noting that this is a specific, testable concern rather than a general lack of information.","tokens_in":741,"tokens_out":2345,"duration_ms":24935,"concrete_test":"Once the full text is available, take the formal definitions of RCI and T-RCI and test three invariance properties on one model-dataset pair: (1) Recomputed RCI under a linear severity schedule versus a logarithmic schedule with the same endpoints and number of steps; if the ranking of models changes, the metric is schedule-dependent. (2) Verify that RCI is monotone with respect to pointwise dominance of accuracy-versus-severity curves; if a strictly worse model receives a better RCI, the metric is not a valid robustness measure. (3) Check whether T-RCI is invariant under permutation of the order in which corruption-type blocks are traversed; if not, the interpretable failure analysis is order-dependent rather than intrinsic to the model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RCI and T-RCI quantify how rapidly VLM reasoning degrades under escalating visual corruption. This requires the metrics to be stable properties of the model, not artifacts of the benchmark's arbitrary implementation choices. The abstract gives no mathematical definitions, so we cannot rule out schedule dependence. For example, if RCI is computed as the mean accuracy drop per severity step, its value depends on the number and spacing of severity levels; a model with gradual early degradation and steep late degradation could be ranked differently simply by changing where the severity samples are placed. Similarly, if T-RCI is a traversal integral, the ordering and grouping of corruption types could change the result. The abstract also does not justify that different corruption types (blur, low contrast, etc.) are commensurable, so aggregating them into a single RCI could hide domain-specific failure modes rather than enabling the promised interpretable analysis. Without the formulas and a sensitivity analysis, the benchmark's headline numbers may not be valid measurements of robustness under progressive perturbation scaling.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BRUCE, a benchmark for evaluating vision-language model robustness under progressively escalating image corruptions in scientific reasoning tasks. The abstract introduces two metrics, RCI (Robustness Corruption Index) and T-RCI (Traversal-RCI), intended to quantify how rapidly multimodal reasoning performance degrades as corruption severity increases. The benchmark is applied to chemistry and mathematics reasoning tasks, and corruption-induced failures are classified into four reasoning domains: OCR-dependent, spatial, symbolic, and semantic. The abstract provides no mathematical definitions, no experimental results, and no validation of the proposed metrics.","tokens_in":895,"tokens_out":2839,"duration_ms":28953,"significance":"The motivation is timely: robustness evaluation for scientific vision-language models currently relies heavily on clean-task accuracy, and a benchmark that characterizes degradation trajectories could be a useful community resource. The proposed four-domain failure taxonomy is a sensible high-level structure for interpretable analysis. However, because the abstract does not define RCI and T-RCI or provide any empirical evidence, the scientific contribution is not yet assessable. If the full paper supplies rigorous definitions, invariance or sensitivity analyses, and comparative experiments, the benchmark could become a valuable tool, but those elements are essential and currently missing.","major_comments":[{"comment":"The central claim that RCI and T-RCI quantify \"how rapidly\" performance deteriorates is not assessable because the functional forms of these metrics are entirely unspecified. Without formulas showing how severity levels are parameterized and aggregated, there is no way to determine whether the measured \"traversal rate\" is a stable property of a model or an artifact of the chosen severity schedule. The paper must present the exact definitions and, ideally, a sensitivity analysis over severity spacing and number of levels.","section":"Abstract, paras. 2–3"},{"comment":"RCI aggregates multiple corruption types (blur, low contrast, etc.) into a single index, but the abstract does not address commensurability across qualitatively different perturbations. If corruption types are simply averaged, the index depends on the arbitrary list of included corruptions and their severity ranges. The authors should either prove that RCI is invariant to such choices or report a sensitivity analysis showing that model rankings do not change when the corruption schedule is varied.","section":"Abstract, para. 2"},{"comment":"The claimed \"interpretable failure analysis\" rests on a taxonomy of four reasoning domains and \"fine-grained corruption specific failure subtypes,\" but the abstract gives no operational definition of how failures are assigned to these categories. Without a rule-based or human-annotation protocol with reliability statistics, the failure analysis cannot be reproduced or compared across studies. The full paper must specify the labeling procedure and report agreement measures.","section":"Abstract, para. 3"}],"minor_comments":[{"comment":"The parenthetical expansion of BRUCE is incomplete: the opening parenthesis after \"BRUCE\" is never closed, and the phrase \"a multimodal reasoning fragility framework\" reads as an appositive that is not grammatically integrated.","section":"Abstract, para. 1"},{"comment":"The term \"Traversal-RCI\" is introduced without any explanation of what is traversed; a brief description of the conceptual meaning (e.g., moving across severity levels or across corruption types) would improve clarity.","section":"Abstract, para. 2"},{"comment":"The abstract does not mention whether each evaluated dataset provides ground-truth labels for the failure subtypes, which would be needed to support the quantitative failure analysis.","section":"Abstract, para. 3"}],"recommendation":"uncertain","confidential_remarks":"The manuscript as provided is abstract-only, and the core technical content—definitions of RCI and T-RCI, experimental setup, and validation—is unavailable. The referee cannot render a soundness judgment without that material. I recommend that the editor obtain the full manuscript before making a decision, and that the review be completed with access to the complete text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version first: this is an abstract-only submission about a benchmark for testing how vision-language models cope with corrupted images in scientific reasoning tasks. The idea is timely and the proposed failure taxonomy (OCR, spatial, symbolic, semantic) is a useful way to look at degradation. But the two metrics that carry the central claim — RCI and T-RCI — are not defined anywhere we can see, and the abstract gives no sensitivity analysis. That means the main quantitative contribution is currently a black box.\n\nWhat's genuinely new: the combination of corruption escalation with a four-domain failure analysis for chemistry and math VLM tasks. That's not in the cited prior work, and it fills a small but real gap. The authors are right that most evaluations focus on clean accuracy. So the framing is good.\n\nThe soft spots are proportional to the missing detail. The RCI/T-RCI metrics could be schedule-dependent — the severity spacing, the ordering of corruption types, and the aggregation rule all matter. If the full paper doesn't include formulas, validation against established benchmarks, and a sensitivity analysis to the corruption grid, the headline numbers may say more about the benchmark than about the models. That said, this is not a fatal flaw from what we can see; it's the usual risk with new self-defined metrics, and the authors may well address it in the main text. We simply can't verify it from the abstract alone.\n\nI'd also note the stress test's concern about commensurability of different corruption types. Aggregating blur and low contrast into one index is only meaningful if the severity scales are aligned. Without that, the 'traversal' rate could hide exactly the failure modes the paper promises to expose. Again, fixable — but it needs to be addressed explicitly.\n\nWho is this for? Researchers in multimodal robustness and AI-for-science. The paper would be a useful data point if the benchmark and metrics hold up. Right now it's an interesting proposal, not a verified contribution.\n\nBottom line: if the full paper provides the definitions, the sensitivity analysis, and external validation, it deserves a serious referee. If those are missing, it's a desk reject. Since we only have the abstract, I'd recommend that a serious editor give it a shot at peer review, with the expectation that the reviewers will insist on the missing evidence.","headline":"Abstract-only benchmark for VLM corruption robustness; the framing is good and the failure taxonomy is useful, but RCI/T-RCI are undefined in the abstract, so soundness can't yet be assessed.","tokens_in":1393,"tokens_out":3049,"would_cite":false,"duration_ms":29137,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BRUCE is a benchmark that measures how quickly scientific vision-language models lose reasoning accuracy as input images are progressively corrupted, using two new metrics, RCI and T-RCI, to quantify the speed of that collapse.","keywords":["visual-language models","robustness benchmark","corruption escalation","multimodal reasoning","scientific reasoning","robustness metrics","failure analysis","image perturbations"],"falsifier":"A concrete observation that would settle the claim: if applying the same corruption types in a different order (for example, blur before contrast versus contrast before blur) or with coarser or finer severity steps substantially reorders models according to RCI or T-RCI, then the measured rate of deterioration is partly an artifact of the escalation schedule. Alternatively, if the four reasoning-domain failure types can be predicted entirely from a generic low-level image-quality score, the metrics would not be measuring reasoning-specific fragility.","tokens_in":545,"feed_emoji":"📉","tokens_out":3109,"duration_ms":28891,"temperature":0.7,"pith_summary":"This paper introduces BRUCE, a benchmark for measuring how robust scientific vision-language models are when input images are progressively degraded by blur, low contrast, and other corruptions. Most evaluation compares clean-task accuracy; BRUCE instead asks how quickly reasoning performance collapses as corruption severity increases. It defines two metrics, Robustness Corruption Index (RCI) and Traversal-RCI (T-RCI), to quantify the rate of deterioration, and classifies the resulting errors into OCR-dependent, spatial, symbolic, and semantic reasoning failures. The goal is to make robustness in multimodal scientific reasoning measurable, comparable, and interpretable rather than treating it as a single average accuracy number.","feed_headline":"Benchmark tracks how fast AI vision models break as images corrupt","feed_subtitle":"Two metrics, RCI and T-RCI, measure the speed of reasoning collapse across OCR, spatial, symbolic, and semantic tasks.","key_machinery":"The central object is BRUCE, a corruption-escalation benchmark that systematically applies increasing levels of image perturbations, such as blur and contrast degradation, to scientific reasoning tasks. The load-bearing metrics are RCI, which aggregates the overall accuracy drop caused by corruption, and T-RCI, which measures how rapidly that drop occurs as severity increases. The supporting taxonomy maps causal failures into four high-level reasoning domains -- OCR-dependent, spatial, symbolic, and semantic -- providing the interpretive layer that turns raw accuracy losses into a diagnosis of which reasoning abilities break first.","core_discovery":"The central claim is that reasoning robustness in scientific vision-language models can be characterized by how performance changes along a progression of corruption severity, not just by accuracy at a fixed noise level. BRUCE applies a wide range of perturbations to chemistry and mathematical reasoning tasks and measures the point and speed at which each model's reasoning degrades, using RCI to capture the overall corruption-induced drop and T-RCI to capture the traversal rate as severity escalates. The paper argues that these metrics reveal distinct fragility patterns: OCR-dependent reasoning fails under different corruptions and at different rates than spatial, symbolic, or semantic reasoning, and mapping failures to these four domains gives an interpretable account of why a model breaks. If correct, the benchmark provides a standard way to ask not only whether a vision-language model is robust, but how its robustness is distributed across reasoning types and perturbation scales.","pith_inferences":["RCI and T-RCI could be applied outside the scientific domains in the paper to general-purpose vision-language models, but that extrapolation is not established in the abstract.","The two scalar metrics compress a high-dimensional corruption space; a hidden assumption is that severity ordering is commensurable across corruption types, which the abstract does not defend.","A testable extension is to check whether the failure taxonomy predicts human error patterns or only model-specific artifacts."],"forward_implications":["Model rankings can change depending on the corruption-severity schedule, so reporting RCI and T-RCI alongside clean accuracy gives a fuller picture of practical reliability.","The failure taxonomy lets developers identify which reasoning skill a model loses first under visual stress, such as OCR dependence vs spatial reasoning.","Scientific vision-language model accuracy claims become more meaningful when accompanied by corruption-traversal metrics, since real images are rarely pristine.","The taxonomy may support targeted robustness training, such as augmenting specific corruption types that cause early semantic or OCR failures."],"supporting_citations":[],"fun_headline_variants":["BRUCE benchmark tracks how AI reasoning collapses as corruptions escalate","New metrics show speed of reasoning breakdown in vision-language models","Benchmark maps fragility across OCR, spatial, symbolic, and semantic reasoning","How corruption severity exposes weak spots in scientific vision-language AI","RCI and T-RCI: quantifying the rate of reasoning failure under image distortion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's chosen corruption types, severity levels, and aggregation rules accurately capture a general property called reasoning robustness; if a different corruption schedule changes which models look robust, then the measured traversal rate is a property of the benchmark rather than of the models.","fun_headline_variants_meta":{"raw":{"variants":["BRUCE benchmark tracks how AI reasoning collapses as corruptions escalate","New metrics show speed of reasoning breakdown in vision-language models","Benchmark maps fragility across OCR, spatial, symbolic, and semantic reasoning","How corruption severity exposes weak spots in scientific vision-language AI","RCI and T-RCI: quantifying the rate of reasoning failure under image distortion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1553,"prompt_tokens":932,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":548,"tokens_out":621,"duration_ms":6191,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:19:55.632330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete observation that would settle the claim: if applying the same corruption types in a different order (for example, blur before contrast versus contrast before blur) or with coarser or finer severity steps substantially reorders models according to RCI or T-RCI, then the measured rate of deterioration is partly an artifact of the escalation schedule. Alternatively, if the four reasoning-domain failure types can be predicted entirely from a generic low-level image-quality score, the metrics would not be measuring reasoning-specific fragility.","supporting_citations":[],"review_version":1}