{"id":"16a8ca8d-7e0a-4fdc-ba26-4ea0b1c66957","arxiv_id":"2411.14725","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AbilityLens unifies 11 public benchmarks into six perception abilities with accuracy and stability metrics, and reveals ability conflicts during MLLM training linked to data mixing and model size.","lead":"AbilityLens is a new benchmark that tests how well multimodal AI models see by combining 11 existing tests into six perception skills and scoring both accuracy and consistency. It also tracks how these skills change during training, exposing conflicts where improving one ability hurts another.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3)-(4) define stability relative to the fixed 18-model cohort; without sensitivity analysis, the stability rankings and the claimed closed-source stability gap are not established as benchmark-independent.","rationale":"The paper's headline contribution is a unified evaluation tool, and the part of that tool most in need of scrutiny is the stability axis, because Eq. (3)-(4) explicitly normalize against the evaluated model set. This is not an internal inconsistency, but it makes the benchmark's central stability comparison conditional on an unstated assumption: that the 18 models approximately span the distribution of relevant MLLMs. The paper argues from Table 4 that closed-source models are more stable, but without a cohort-sensitivity analysis those numbers are not comparable across papers or model sets. The reader's weakest assumption already flags z-score standardization and cohort skew, and this concern is the concrete version of that assumption. If a sensitivity test shows the rankings are stable, the concern does not land and the conditional verdict is unchanged. If it shows instability, the authors should either report stability on raw sub-metric variances or provide cohort-invariant statistics; either way the benchmark remains useful but the stability claims need softening. I see no basis for rejection, so the reader's CONDITIONAL verdict should remain.","tokens_in":18839,"tokens_out":6912,"duration_ms":71810,"concrete_test":"Recompute stability scores from Table 4 and the stability curves in Figure 2 under leave-one-out over the 18 candidate models, and under two augmented cohorts: (i) add 5 weaker open-source models, and (ii) remove the closed-source models. If the top-5 stability rankings change by more than one position, or if the closed-vs-open stability gap shifts by more than roughly 5 points, the stability conclusions are cohort artifacts and should be re-reported with cohort-invariant statistics such as raw per-sub-metric variances or bootstrap confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AbilityLens's stability metric is defined relative to the candidate set itself. In Eq. (3), z_ij = (m_ij - mu_j) / sigma_j uses the mean and standard deviation of the 18 evaluated models on each sub-metric, and Eq. (4) then sets stability to std(z_i) across sub-metrics. Consequently, a model's stability score is not an intrinsic property of the model but a measure of how consistently its position deviates from the cohort. The central comparison in Sec. 4.1 ('closed-source models are more stable') and the stability columns in Table 4 can therefore change if the cohort is augmented or replaced: a uniformly dominant model would look perfectly stable against a weak cohort, while a genuinely balanced model can look unstable when it sits near the cohort mean on some sub-metrics and far above it on others. The paper reports no sensitivity analysis, and the NeurIPS checklist item 7 explicitly concedes that no error bars are provided, so cohort effects cannot be separated from model properties. Since the online training-dynamics analysis in Sec. 4.2 and Sec. 5 also relies on the same stability scores, the ability-conflict conclusions inherit this cohort dependence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AbilityLens, a benchmark assembled from 11 public datasets and organized around six perception abilities: counting, OCR, attribute recognition, entity extraction, grounding, and structural-data understanding. It defines a baseline-corrected accuracy score (Eqs. 1-2) and a z-score-based stability score (Eqs. 3-4), evaluates 18 open- and closed-source MLLMs, and uses the benchmark to monitor training dynamics of LLaVA-OV-SI checkpoints. The authors report that closed-source models are more stable, that perception abilities exhibit early convergence and ability conflicts, and that data mixing ratio and LLM size are the primary causes of ability conflict. They also discuss fine-tuning and model merging as partial remedies.","tokens_in":19147,"tokens_out":3321,"duration_ms":32806,"significance":"If the claims hold, AbilityLens is a useful contribution: it is efficient (about 0.5 hours for a 7B model on 8 H100s), built from public data, and ships with code and a leaderboard, which is a reproducibility-friendly design. The distinction between accuracy and stability, and the use of training-dynamics monitoring to expose ability conflicts, are valuable and actionable directions for MLLM development. The paper's empirical findings are also falsifiable: specific models and checkpoints are enumerated, and the benchmark is released. However, the robustness of the stability metric and the strength of the causal claims about ability conflict must be established before the central conclusions can be accepted.","major_comments":[{"comment":"","section":""},{"comment":"","section":""},{"comment":"","section":""},{"comment":"","section":""}],"minor_comments":[{"comment":"","section":"Table 2 caption; Table 7; Sec. 4"},{"comment":"","section":"Fig. 1"},{"comment":"","section":"Checklist item 12"},{"comment":"","section":"Sec. 4.1"},{"comment":"","section":"Figs. 5-6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark contribution with a reproducible framework, but the stability metric's cohort dependence and the absence of error bars/sensitivity analysis are central to the paper's main claims. I believe these issues are fixable within the scope of a major revision, so I do not recommend rejection. The claimed causal primacy of data mixing and LLM size should also be softened or supported by more evidence. The manuscript would also benefit from addressing the licensing item in the checklist, as the benchmark is constructed from multiple existing datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: AbilityLens is a solid benchmark-building paper, not a breakthrough. What is genuinely new is packaging eleven existing benchmarks into six perception abilities with a baseline-corrected weighted accuracy score and a z-score-based stability metric, then applying the same tool to monitor training checkpoints. That is a real contribution, and the public release makes it usable. The motivation figure showing models ranking differently on MME, MMBench, and SeedBench for entity is a fair example of why benchmark variance matters.\n\nThe stability metric is the most interesting part and the softest spot. Equations (3)-(4) define stability as the spread of a model's z-scores across sub-metrics, where the mean and standard deviation are computed from the 18 evaluated models. So stability is a cohort-relative property. A model that is uniformly strong looks perfectly stable; a balanced model can look unstable if it sits near the cohort mean on some sub-metrics and far above on others. The paper's claim that closed-source models are more stable, and the stability rankings in Table 4, could shift if the cohort were changed. The stress-test concern about this is correct. It is not fatal if AbilityLens is treated as a fixed public leaderboard with a fixed cohort, but the paper should say that explicitly and show a sensitivity analysis with a few cohort perturbations.\n\nSecond, the paper has no error bars and says so in the checklist. For a benchmark paper that ranks models and reports training curves, that is a real weakness. Several conclusions, especially OCR degradation after 10k steps, could be noise. The sample-reduction procedure is also unspecified: Table 2 shows big reductions (SeedBench 2447 to 1223, ChartQA 2500 to 400) but the text only says redundant samples are removed. That needs detail.\n\nThird, the claim that the primary reason of ability conflict is data mixing ratio and LLM model size is too strong. The evidence is a handful of training runs with one architecture family, a few LLM backbones, and two data perturbations. It is a reasonable hypothesis, not a demonstrated cause. The paper would be stronger with 'suggests' or 'is associated with'.\n\nThe citation pattern looks fine; UniBench and EUREKA are acknowledged as closest prior work. The benchmark construction is internally consistent, and the accuracy part of the evaluation is not undercut by the stability caveat.\n\nWho is this for: people building or evaluating MLLMs who want a holistic perception benchmark and a quick way to track ability-level training dynamics. It is not a theory paper and not a strong causal study. I would send it to a serious referee. The right outcome is conditional acceptance asking for sensitivity analysis, sampling details, error bars, and softened causal language.","headline":"A useful and reusable perception benchmark with a cohort-dependent stability metric; the benchmark work should get a fair review, but the causal claims about ability conflict need softening.","tokens_in":19608,"tokens_out":3277,"would_cite":true,"duration_ms":33696,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language model perception cannot be judged by accuracy alone: rankings flip across benchmarks and abilities conflict during training, so this paper builds AbilityLens, a six-ability benchmark scoring accuracy and stability, and…","keywords":["multimodal large language models","perception evaluation","benchmark","stability","accuracy","ability conflict","training dynamics","baseline correction"],"falsifier":"Recompute the ability rankings and stability scores with a different cohort of 18 comparable models, or with a different random sample drawn from the same 11 source benchmarks; if top-model accuracy-stability rankings shift materially or the OCR mid-training peak vanishes under the same data recipe, the central claim fails.","tokens_in":18672,"feed_emoji":"👁️","tokens_out":9628,"duration_ms":90888,"temperature":0.7,"pith_summary":"Perception is the core skill that separates multimodal large language models from text-only ones, but the paper argues that no single existing benchmark can tell how well a model really sees: rankings of the same models flip across different perception tests. AbilityLens pools more than 12,000 samples from 11 public benchmarks into six perception abilities—counting, OCR, attribute recognition, entity extraction, grounding, and structural data understanding—and scores each model twice, once for accuracy and once for stability. Accuracy comes from baseline-corrected sub-metrics so that True/False, multiple-choice, and VQA questions are not distorted by their different random-guess rates; stability is the spread of a model's z-scores across those sub-metrics among 18 candidate models. With this lens the paper finds that the strongest open models match closed ones in accuracy but trail them noticeably in stability, and that during training abilities do not improve in lockstep: OCR peaks at mid-training and then declines while other abilities keep improving. The paper identifies the data mixing ratio and the size of the LLM backbone, rather than the visual encoder, as the main drivers of this ability conflict.","feed_headline":"AbilityLens splits accuracy from stability across six skills","feed_subtitle":"A six-ability benchmark built from 11 public tests exposes conflicts and stability gaps that accuracy-only scores hide.","key_machinery":"The load-bearing object is AbilityLens, an ability-decomposed benchmark assembled from 11 public test suites. The identity that carries the argument is the baseline correction $\\tilde{m}_{ij} = (m_{ij} - BL_j)/(UB_j - BL_j)$, which rescales every sub-metric between its random-guess baseline and its upper bound, so that the weighted accuracy score $A_i = \\sum_j n_j \\tilde{m}_{ij} / N$ is question-type agnostic. The stability score is $I_i = \\mathrm{std}(z_i)$, where $z_{ij} = (m_{ij} - \\mu_j)/\\sigma_j$ is the model's relative position among 18 candidate models on sub-metric $j$; a high standard deviation means the model's standing swings across question types, domains, and metrics. Together the two scores turn a collection of heterogeneous benchmarks into a common scale on which training checkpoints can be compared and ability conflicts can be localized to a specific skill.","core_discovery":"AbilityLens is a unified evaluation instrument, and its central claim is that perception ability is two-dimensional: accuracy and stability, with stability defined as the standard deviation of a model's z-scores across sub-metrics relative to a cohort of 18 recent MLLMs. The benchmark is constructed by selecting roughly 1,000 or more test samples per ability from 11 source benchmarks, applying baseline correction to every sub-metric, and aggregating with sample-count weights. Evaluated on 18 open and closed models, AbilityLens reveals that no model dominates all six abilities, that closed-source models reach accuracy comparable to the best open models while being substantially more stable, and that tracking checkpoints of a LLaVA-OneVision reproduction shows early convergence plus a specific ability conflict: OCR accuracy and stability rise, peak around 10,000 training steps, and then fall while other abilities keep rising. The paper's causal analysis attributes the conflict to data mixing ratio and LLM model size, not the vision encoder, and shows that a lightweight model-merging step (Task Arithmetic) mitigates the conflict better than continued fine-tuning at a fraction of the compute.","pith_inferences":["Because Eq. (3) is defined against the 18-model cohort, AbilityLens rankings are candidate-dependent; extending the benchmark to a larger or differently sampled cohort would test whether the stability gaps and conflict findings transfer.","The same six-ability decomposition and baseline-correction scheme could be applied to video or audio-visual models, where stability across frames or modalities may be a stronger signal than aggregate accuracy.","If data mixing ratio is the primary conflict driver, then ordering training data by ability convergence time—entity early, counting late, OCR mid-training—could reduce conflicts without changing total data volume.","The baseline-correction formula assumes each sub-metric's upper bound is attainable; on saturated or floored metrics, both the accuracy and z-score weights may need renormalization."],"forward_implications":["If accuracy and stability are treated as separate axes, an accuracy leader that is erratic across question types will no longer outrank a steadier model, and benchmark designers can report both numbers instead of one.","Monitoring checkpoints per ability during training will reveal conflicts that average-score curves hide; in the paper's LLaVA-OneVision reproduction, OCR degrades after 10,000 steps while the average keeps rising.","The causal finding that data mixing ratio and LLM size drive ability conflict means training strategies should rebalance data proportions and strengthen the backbone, not just swap in a better vision encoder.","For open-source development, the stability gap with closed models implies that consistency across question types is an achievable and necessary target, not a luxury.","When a conflict hits only one ability, model-merging (Task Arithmetic) recovers it more cheaply and more effectively than continued fine-tuning, while conflicts across several abilities remain open."],"supporting_citations":[{"why":"Supplies the True/False-style perception items (counting, OCR, position, entity) whose 50% random baseline the correction formula must neutralize.","marker":"[12]"},{"why":"Supplies multiple-choice items across several abilities, giving the question type whose lower random baseline the aggregation has to align.","marker":"[40]"},{"why":"Contributes large multiple-choice subsets for counting, entity, and attribute abilities, which are sample-reduced for the unified set.","marker":"[18]"},{"why":"Adds fine-grained perception MCQ sub-metrics used for counting, localization, attribute, and diagram abilities.","marker":"[9]"},{"why":"Provides the unified evaluation pipeline and postprocessing that allow samples and metrics from 11 source benchmarks to be pooled into one framework.","marker":"[43]"},{"why":"Provides the training recipe whose checkpoints are tracked online and whose mid/one-vision stages are compared for ability conflict.","marker":"[19]"},{"why":"The model-merging method tested as a lightweight intervention to recover OCR ability after conflict.","marker":"[14]"}],"fun_headline_variants":["AbilityLens splits perception into accuracy and stability","Six-skill benchmark reveals hidden ability conflicts","Why OCR peaks then declines in LLM training","Stability, not just accuracy, exposes MLLM gaps","Data mixing ratio drives ability conflicts in MLLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the 18 models chosen as the comparison cohort and the reduced samples selected from each of the 11 source benchmarks are representative enough that the z-score and weighting steps yield a benchmark-independent measurement of perception ability.","fun_headline_variants_meta":{"raw":{"variants":["AbilityLens splits perception into accuracy and stability","Six-skill benchmark reveals hidden ability conflicts","Why OCR peaks then declines in LLM training","Stability, not just accuracy, exposes MLLM gaps","Data mixing ratio drives ability conflicts in MLLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1321,"prompt_tokens":1032,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":648,"tokens_out":289,"duration_ms":3264,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:57:46.348824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the ability rankings and stability scores with a different cohort of 18 comparable models, or with a different random sample drawn from the same 11 source benchmarks; if top-model accuracy-stability rankings shift materially or the OCR mid-training peak vanishes under the same data recipe, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes large multiple-choice subsets for counting, entity, and attribute abilities, which are sample-reduced for the unified set."}],"review_version":1}