Redundancy Principles for MLLMs Benchmarks

Chunyi Li; Guangtao Zhai; Haodong Duan; Kai Chen; Xiangyu Zhao; Xiaohong Liu; Xinyu Fang; Xiongkuo Min; Zicheng Zhang

REVIEW 3 major objections 6 minor 2 cited by

Redundancy Principles for MLLMs Benchmarks

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most existing MLLM benchmarks carry at least 50% redundant test instances, because sampling half the questions still reproduces model rankings with correlation above 0.95.

desk verdict A useful, open-data empirical survey of benchmark redundancy with a real finding about Top-50 vs Bottom-50 models, but the headline 'half the instances are redundant' is inflated by a part-whole correlation and needs split-half reliability. read the letter →

arxiv 2501.13953 v2 pith:S2M3TIQX submitted 2025-01-20 cs.CL cs.AI

Zicheng Zhang , Xiangyu Zhao , Xinyu Fang , Chunyi Li , Xiaohong Liu , Xiongkuo Min , Haodong Duan , Kai Chen

show 1 more author

Guangtao Zhai

This is my paper · ORCID

classification cs.CLcs.AI

keywords MLLMbenchmarkredundancyrankcorrelationinstancesamplingdesignprinciplesmultimodallargelanguagemodelscross-benchmarkperformanceranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that redundancy in MLLM benchmarks is pervasive, measurable, and partly avoidable at three levels: within-benchmark capability dimensions, individual test questions, and benchmarks that target the same domain. Its guiding principle is correlation-based: if two evaluation components rank a large population of MLLMs in nearly the same order, the second component adds little information. Using public evaluation records from over 100 MLLMs on more than 20 benchmarks, the paper finds that for most benchmarks at least half of the test instances are redundant, and that ranking the strongest models requires more questions than ranking the weakest ones. The paper converts these observations into design principles: keep dimensions independent, choose the smallest instance count that preserves ranking, and deliberately decide whether a domain benchmark should overlap with its peers or fill a gap.

What carries the argument

The central object is the Performance Correlation Redundancy Framework. It defines redundancy as the correlation between MLLM performance rankings on two dimensions, two sampled instance sets, or two benchmarks, measured by Spearman rank correlation, Pearson linear correlation, and $R^2$. For dimensions and cross-benchmark pairs, an item's redundancy is the average correlation with all other items; for instances, the full-benchmark ranking is compared with rankings from randomly sampled subsets at ratio $A\%$, repeated 100 times and averaged. The framework's operative threshold is 0.95: once a sampled ranking correlates with the full ranking above that level, the unsampled instances are declared redundant.

What would settle it

Take two deliberately disjoint benchmarks, one pure OCR and one pure social reasoning, run a large set of MLLMs on both, and compute the Spearman correlation of their rankings; if the correlation exceeds 0.95 despite no shared task content, then correlation is not a valid proxy for content redundancy and the framework's central mapping fails.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is quantitative. Across 18 mainstream MLLM benchmarks, sampling 50% of the test instances produces model rankings whose Spearman and Pearson correlations with the full-benchmark ranking exceed 0.95, so the omitted half contributes almost no ranking information. The same correlation logic applied to MMBench's 20 dimensions shows that capability dimensions are substantially more redundant for the bottom half of models than for the top half, meaning redundancy is a property of the model population as much as of the benchmark. In the mathematics domain, four popular benchmarks are not strongly redundant with one another; MathVista stands apart because roughly 30-40% of its questions sit outside traditional mathematics, and removing those questions raises its correlation with the other math benchmarks. The paper reads these results as evidence that redundancy can be diagnosed and pruned, and that a benchmark's intended role, broad domain representative versus specialized probe, should determine how much overlap with other benchmarks it seeks.

Load-bearing premise

The load-bearing premise is that nearly identical model rankings across two evaluation sets mean the sets measure the same content; if the rankings correlate because of a shared general ability rather than overlapping questions, the redundancy numbers overstate true duplication.

Editorial extensions

If this is right

Benchmark builders could cut most mainstream MLLM benchmark question counts roughly in half without materially changing the model ranking they report.
Ranking the top-performing MLLMs demands more questions than ranking weaker models, so instance counts should be chosen with the target capability tier in mind.
Dimensions that consistently rank MLLMs alike, such as Image Emotion and Social Relation in MMBench, can be consolidated into one score rather than reported as independent.
A broad-coverage domain benchmark should show high cross-benchmark redundancy with its peers, while a specialized benchmark should show low redundancy; unrelated tasks inside a domain benchmark dilute its measured redundancy.
Models with uniformly poor performance should be excluded from redundancy audits, because their consistent underperformance inflates correlations and obscures true dimension independence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If rank correlations across genuinely different tasks are driven by a shared general-ability factor rather than by content overlap, then the paper's numeric redundancy estimates overstate how much actual test content is duplicated; the paper itself flags this premise as its main assumption.
The same sampling-curve machinery could be turned into a per-benchmark quality certificate that reports the minimum sample size needed to recover the full ranking at a chosen confidence level, making redundancy a routine statistic rather than a one-off analysis.
One direct consequence for model developers is that half-sampled benchmarks remain nearly as reliable for ordering models but noticeably less reliable for absolute score comparisons, since $R^2$ saturation requires over 90% of instances.
Because redundancy changes with the model population, a benchmark that is informative for today's strongest models may become redundant as capabilities separate differently, so redundancy estimates should be recomputed when the frontier shifts.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know about arXiv:2501.13953. First, it is a genuinely useful empirical survey: the authors run a simple correlation framework over VLMEvalKit's open leaderboard data, with code and data available, and they report saturation curves for 18 benchmarks stratified by Top-50 and Bottom-50 models. The Top-50 vs Bottom-50 asymmetry is a real finding, and the paper is honest enough to list the main assumption as a limitation in Section 5. Second, the headline quantitative claim—"at least 50% of instances are redundant"—is overstated, and the stress-test note is correct: Eq. (3) correlates the sampled subset ranking with the full-set ranking that contains it. That is a part-whole correlation. For PLCC at rho(0.5)=0.95, the implied split-half correlation between two disjoint halves is about 0.805, which is below the paper's own "nearly identical" threshold. So the conclusion that the remaining half adds little information does not follow from the measured numbers. The correct check is split-half reliability: correlate two disjoint halves and report that.

The dimension-redundancy analysis avoids that particular circularity, but it inherits the paper's stated assumption that rank correlation implies shared capability. That assumption is plausible but fragile; the paper's own limitation admits it. The cross-benchmark section removes "general VQA" and CLEVR-derived questions from MathVista post hoc and reports that redundancy rises, which is suggestive but not a controlled analysis. There are no confidence intervals or significance tests anywhere; with roughly 50 to 100 models, those would matter, especially for the R^2 claims about needing 90% of instances.

What is genuinely new: the Top-50/Bottom-50 stratification, the per-benchmark saturation curves, and the explicit framework for choosing whether a new benchmark should be redundant with a domain. The open code and data make it reproducible, and that should be credited.

Who is this for: benchmark designers and anyone running evaluation suites who wants a cheap, interpretable way to gauge whether their questions are pulling their weight. The framework is a reasonable starting tool; the specific percentages should not be quoted as measured facts until the split-half analysis is done.

Verdict: worth a serious referee, and I would send it to review. The right path is a major revision that adds split-half reliability, confidence intervals, and a less assertive reading of the saturation curves. If the authors do that, the paper becomes a solid reference.

Referee Report

3 major / 6 minor

Summary. The paper proposes a redundancy-analysis framework for MLLM benchmarks at three levels: redundancy among capability dimensions within a benchmark, redundancy among test instances within a benchmark, and cross-benchmark redundancy within a domain. The framework quantifies redundancy through rank correlations of MLLM performance rankings, using VLMEvalKit records for more than 20 benchmarks and OpenCompass results for math benchmarks. The main reported findings are that many benchmark dimensions are highly correlated for weaker models, that most benchmarks retain ranking fidelity after removing at least 50% of their instances according to a 0.95 correlation threshold, and that MathVista is less redundant with other math benchmarks until general-VQA and CLEVR-derived questions are removed. The paper concludes with practical principles for benchmark design and an appendix with additional redundancy maps and recommendations.

Significance. If the quantitative claims survive correction, the paper would provide a broadly useful and low-cost methodology for auditing benchmark redundancy, with practical value for benchmark designers and evaluators. The work is commendably grounded in open data, ships code, and includes explicit limitations, which makes the analysis easier to check and reuse. The three-level decomposition (dimension, instance, cross-benchmark) is a sensible organizing framework, and the Top-50/Bottom-50 comparison is a useful angle. However, the headline instance-redundancy claim is built on a part-whole correlation, and the cross-benchmark quantitative support is thinner than the narrative suggests; both points need to be addressed before the paper's central conclusions can be accepted.

major comments (3)

[Section 3.2, Eq. (3)] The central claim that 'at least 50% of the instances are redundant' rests on the correlation between a ranking computed on a random A% subset and the ranking computed on the full instance set. Because the subset is a proper part of the full set, this is a part-whole correlation, not an independent measure of redundancy. For equal-sized disjoint halves X and Y with equal variance and split-half correlation r, the PLCC between X and X+Y is sqrt((1+r)/2), so the reported threshold rho(0.5)=0.95 corresponds to r=0.805. Two random halves of the benchmark therefore agree with each other at only about 0.80, well below the 0.95 threshold the paper uses to define 'nearly identical' rankings. The same inflation applies qualitatively to SRCC and to R^2. The quantitative claim that half the instances can be dropped without affecting the ranking is thus overstated by construction. Please report split-half reliability, i.e., the correlation between two disjoint halves averaged over repeated splits, and base the redundancy conclusions on that quantity, or on an alternative criterion such as the smallest sample size needed to match the full ranking within a specified error bound.
[Section 3.3] The cross-benchmark analysis in the math domain supports the practical recommendation that MathVision and MathVerse are 'more suitable for benchmarking the mathematical capabilities of MLLMs in a narrow sense,' but the supporting evidence is incomplete. The text states that removing general-VQA and CLEVR-derived questions from MathVista 'significantly increases' redundancy with other math benchmarks, yet it reports no numerical correlations, no confidence intervals, and no significance test for this increase. With only 37 MLLMs from the OpenCompass leaderboard, the rank correlations have wide sampling distributions. Please report the correlation matrices and their uncertainties before and after the removal, and test whether the increase is statistically significant. In addition, because the framework itself notes that low correlation can indicate either unique content or noise, the interpretation of MathVista's low redundancy as 'noise' is underdetermined without a more direct content-overlap analysis.
[Section 2; Section 5] The framework's prior assumption is that strongly correlated rankings imply redundant capabilities or benchmarks. As the Limitations section acknowledges, this assumption 'may not always hold.' This is not a circularity, but it is a load-bearing interpretive step: the quantitative redundancy estimates will overstate content overlap if rank correlations are driven by a shared general capability factor rather than by task similarity. The paper would be substantially strengthened by a validation study on a subset of benchmarks, comparing the rank-correlation-based redundancy estimates against direct evidence of overlap, such as exact or near-duplicate questions, shared answer distributions, or task-taxonomy overlap. If the two measures diverge, the reported percentages should be re-interpreted as ranking-redundancy rather than content-redundancy.

minor comments (6)

[Section 1.3] The heading contains a typo: 'Benifits' should be 'Benefits'.
[Figure 5 caption] The caption says 18 benchmarks but the parenthetical list contains 17 entries; either add the missing benchmark or correct the count.
[Section 3.2] The sentence 'We adopt a similarity threshold of 0.95 for partitioning2' is missing a period, and the footnote marker placement is awkward; also, 'partitioning' alone does not convey what is being partitioned.
[Section 3.3, Figure 8 caption] The caption writes 'MathVersion' in one place; this should be 'MathVerse'.
[Section 3.2, footnote 2] The citation to Hauke and Kossowski (2011) supports a comparison of Pearson and Spearman coefficients, not a threshold of 0.95 for 'nearly identical' rankings; a different justification or a sensitivity analysis around the threshold would be more appropriate.
[Appendix A] The inline formulas for PLCC and R2 appear scrambled, with missing fraction bars and square-root signs; please typeset them properly.

Circularity Check

1 steps flagged · score 6.0 of 10

Instance-redundancy claim is inflated by part-whole correlation; the rest of the framework is a non-circular measurement.

self definitional [Section 2.2 (Eq. 3); Section 3.2]
"To quantify the redundancy of the benchmark at a sampling ratio of A%, we calculate the correlation coefficient between Rsample and RGT ... RGT is the MLLM ranking based on the full M instances within the MLLM benchmark. ... a majority of existing MLLM benchmarks exhibit significant redundancy in their instances ... with at least 50% of the instances being redundant."

The instance-redundancy metric ρ(A%) is defined as CORR(RA%t, RGT), where RA%t is the ranking on a random A%-subset and RGT is the ranking on the full M-instance set. Because the subset is contained in the full set, the two rankings share A% of the data by construction; this is a part-whole correlation, not an independent redundancy estimate. For equal-variance halves under PLCC, ρ(0.5)=sqrt((1+r)/2), so the paper's threshold ρ=0.95 corresponds to a split-half correlation of only r≈0.805. The paper then interprets ρ(A%)≥0.95 as 'the remaining 1−A% instances contribute little additional information' and concludes that at least 50% of instances are redundant.

full rationale

The dimension-redundancy (Section 2.1) and cross-benchmark redundancy (Section 2.3) analyses are not circular: they compare rankings computed on disjoint dimensions or disjoint benchmarks and report the resulting correlations. The paper's stated prior assumption, while acknowledged in the Limitations as possibly invalid, is an assumption rather than a derived result, so it is a validity concern rather than a circularity. Self-citations to VLMEvalKit and MMBench are data/tool citations backed by open-source resources and external evaluation records; they are not load-bearing self-referential evidence. The 0.95 threshold is justified by an external statistical reference (Hauke and Kossowski, 2011), not by the authors' own prior work. The one significant circular/construction issue is in Section 3.2: the instance-redundancy metric correlates a sampled subset with the full set containing that subset, so the high ρ(A%) values are partly an artifact of the definition. This inflates the paper's central claim that at least 50% of instances are redundant in most benchmarks. The rest of the framework is a self-contained measurement exercise, and the paper candidly lists limitations about the correlation-to-redundancy assumption. Overall, the instance-redundancy conclusion is partially forced by construction, giving a score of 6 rather than a fully circular derivation score of 8-10.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entity. It relies on the domain assumption that ranking correlation reflects redundancy, on open leaderboard data, and on hand-set thresholds and cohort splits. The free parameters are analysis choices rather than fitted model parameters, but they materially affect the quantitative claims.

free parameters (3)

0.95 rank-correlation threshold = 0.95
Chosen as the cutoff for considering two rankings 'nearly identical' in Figure 5 and Section 3.2. It is not fitted to the data, but it directly determines the claim that 50% of instances are redundant, so it is a hand-set decision affecting the headline number.
Top-50 and Bottom-50 model cohorts = 50 per cohort
Models are split into top and bottom halves of the overall performance distribution. This hand-chosen split size drives the paper's main qualitative finding that Bottom-50 models show much higher redundancy, and results may vary with another split.
Sampling ratios A% from 20% to 90% = 20 to 90 percent
The instance-redundancy curves are reported over this range. The range is reasonable, but the absence of a principled criterion for the minimum sampling ratio leaves the saturation point dependent on the chosen grid.

assumptions (3)

domain assumption Similar capabilities of MLLMs produce strongly correlated performance rankings.
Stated as the 'prior assumption' at the start of Section 2 and explicitly flagged as potentially invalid in the Limitations section. This is the load-bearing premise that connects correlation to redundancy.
domain assumption Model rankings from VLMEvalKit and OpenCompass are reliable and comparable.
The entire analysis rests on third-party leaderboard data. The paper does not verify the accuracy or consistency of those records, such as prompt differences or metric aggregations across benchmarks.
domain assumption Spearman, Pearson, and R-squared correlations are adequate summaries of benchmark similarity.
The paper chooses these three metrics in Section 2.4 and does not justify that they capture the full complexity of task overlap, a point the authors concede in the Limitations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Redundancy Principles for MLLMs Benchmarks." pith.science (2026). https://pith.science/paper/S2M3TIQX

@misc{pith2026250113953,
  author       = {Pith},
  title        = {Pith review of: Redundancy Principles for MLLMs Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S2M3TIQX}},
  note         = {Machine review of arXiv:2501.13953}
}

read the original abstract

With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of redundancy and propose targeted principles for constructing effective MLLM benchmarks. In this paper, we focus on redundancy from three key perspectives: 1) Redundancy of benchmark capability dimensions, 2) Redundancy in the number of test questions, and 3) Cross-benchmark redundancy within specific domains. Through the comprehensive analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we aim to quantitatively measure the level of redundancy lies in existing MLLM evaluations, provide valuable insights to guide the future development of MLLM benchmarks, and offer strategies to refine and address redundancy issues effectively. The code is available at https://github.com/zzc-1998/Benchmark-Redundancy.

Figures

Figures reproduced from arXiv: 2501.13953 by the authors.

**Figure 1.** Where Redundancy Exists? and Why Evaluate Redundancy? Abstract With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of re… view at source ↗

**Figure 2.** A quick look at the redundancy framework, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Visualizations of dimensions redundancy for MMBench ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Bar plots of dimensions redundancy for MMBench ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Visualizations of average instance redundancy for (a) Top-50 MLLMs and (b) Bottom-50 MLLMs across [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Benchmark-specific instance redundancy for (a) Top-50 MLLMs and (b) Bottom-50 MLLMs. The [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Examples of tasks excluded from the MathVista benchmark. (a), (b), and (c) showcase tasks derived from [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Cross-benchmark redundancy map. MathVision and MathVersion are more focused on the core domain of mathematics (with relatively higher redundancy across other math benchmarks), making them more suitable for benchmarking the mathematical capabilities of MLLMs in a nar…

**Figure 9.** Figure 9: Visualizations of dimensions redundancy for AI2D ( [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: Visualizations of dimensions redundancy for SEED-Bench ( [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Affordance Benchmark for MLLMs
cs.CL 2025-06 conditional novelty 6.0 of 10

A new 2,000-question benchmark finds multimodal AI models recognize object affordances far worse than humans, with top model Gemini-2.0-Pro at 18.05% versus 85.34% human best.
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models
cs.AI 2025-05 conditional novelty 6.0 of 10

Current vision-language models fall far short of humans on spatial reasoning, especially when they must generate answers directly instead of choosing from options.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages · cited by 2 Pith papers

[1]

It is defined as: SRCC = 1 − 6 Pn i=1 d2 i n(n2 − 1) , where: di = rank(xi) − rank(yi), and n is the number of data points

The SRCC measures the rank-based relation- ship between predicted and true values. It is defined as: SRCC = 1 − 6 Pn i=1 d2 i n(n2 − 1) , where: di = rank(xi) − rank(yi), and n is the number of data points. A higher SRCC indicates a stronger monotonic relationship between the rankings of predicted and ground truth values

work page
[2]

It is computed as: PLCC = Pn i=1(xi − ¯x)(yi − ¯y)pPn i=1(xi − ¯x)2pPn i=1(yi − ¯y)2 , where: • xi and yi are the data points, • ¯x and ¯y are the means of x and y, respectively

The PLCC quantifies the linear relationship between predicted and true values. It is computed as: PLCC = Pn i=1(xi − ¯x)(yi − ¯y)pPn i=1(xi − ¯x)2pPn i=1(yi − ¯y)2 , where: • xi and yi are the data points, • ¯x and ¯y are the means of x and y, respectively. A higher PLCC indicates a stronger linear relation- ship between predicted and ground truth values

work page
[3]

arXiv preprint arXiv:2412.10360

Apollo: An exploration of video understand- ing in large multimodal models. arXiv preprint arXiv:2412.10360. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. 2024. Dynamath: A dy- namic visual benchmark for evaluating mathemati- cal reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836. A Metrics Equation ...

arXiv 2024
[6]

It is defined as: R2 = 1 − Pn i=1(yi − ˆyi)2 Pn i=1(yi − ¯y)2 where: • yi are the ground truth values, • ˆyi are the predicted values, • ¯y is the mean of the ground truth values

The R2 score represents the proportion of variance in the ground truth values that is explained by the predictions. It is defined as: R2 = 1 − Pn i=1(yi − ˆyi)2 Pn i=1(yi − ¯y)2 where: • yi are the ground truth values, • ˆyi are the predicted values, • ¯y is the mean of the ground truth values. An R2 score closer to 1 indicates a better fit be- tween the ...

work page 2016
[7]

9, it is evident that the dimension ‘lifeCycles’ exhibits the highest redundancy, particularly with ‘typesOf’

Key Observations from the Redundancy Maps: • In Fig. 9, it is evident that the dimension ‘lifeCycles’ exhibits the highest redundancy, particularly with ‘typesOf’. • Similarly, in Fig. 10, the ‘Instance Identity’ dimension shows the highest redundancy and is most closely related to ‘Scene Under- standing’

work page
[8]

Bottom-50 Redun- dancy: • A clear pattern emerges when comparing the Top-50 and Bottom-50 redundancy maps

Trends in Top-50 vs. Bottom-50 Redun- dancy: • A clear pattern emerges when comparing the Top-50 and Bottom-50 redundancy maps. Nearly all Bottom-50 dimensions display significantly higher redundancy than their Top-50 counterparts. This observation sup- ports our conclusion that dimensions tend to exhibit greater redundancy for Bottom- 50 MLLMs compared t...

work page
[9]

Implications for Redundancy Analysis: • To ensure a reasonable and accurate evalu- ation during redundancy analysis, it is cru- cial to exclude MLLMs with consistently poor performance. Including such models could skew the analysis by disproportion- ately inflating redundancy, as their universal underperformance does not provide mean- ingful insights into...

work page
[10]

Calculate the dimensional redundancy within the benchmark, with particular attention to di- mensions exhibiting overall high redundancy

Dimension Redundancy Check. Calculate the dimensional redundancy within the benchmark, with particular attention to di- mensions exhibiting overall high redundancy. Analyze the redundancy heatmap to identify pairs of dimensions with exceptionally strong correlations, as these may indicate overlap- ping capabilities being assessed. For such cases, evaluate...

work page

Show all 12 references

[11]

Compute the instance redundancy curve to determine whether a smaller subset of bench- mark instances can produce results compa- rable to the full instance set

Instance Redundancy Check. Compute the instance redundancy curve to determine whether a smaller subset of bench- mark instances can produce results compa- rable to the full instance set. If significant instance redundancy is identified, the bench- mark should be reviewed, and ...
[12]

If the benchmark is intended to serve as a rep- resentative for a specific domain, measure its cross-benchmark redundancy relative to other benchmarks within the domain

Cross-benchmark Redundancy Check. If the benchmark is intended to serve as a rep- resentative for a specific domain, measure its cross-benchmark redundancy relative to other benchmarks within the domain. Higher redun- dancy indicates stronger representativeness, making it a re...

2016
[2016]

In Euro- pean conference on computer vision, pages 235–251

A diagram is worth a dozen images. In Euro- pean conference on computer vision, pages 235–251. Springer. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava- onevision: Easy visual task trans...

2023 arXiv
[2024]

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei- Chiu Ma, and Ranjay Krishna

Mme: A comprehensive evaluation benchmark for multimodal large language models. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei- Chiu Ma, and Ranjay Krishna. 2025. Blink: Multi- modal large language models can see but not perceiv...

2025 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

It is defined as: SRCC = 1 − 6 Pn i=1 d2 i n(n2 − 1) , where: di = rank(xi) − rank(yi), and n is the number of data points

The SRCC measures the rank-based relation- ship between predicted and true values. It is defined as: SRCC = 1 − 6 Pn i=1 d2 i n(n2 − 1) , where: di = rank(xi) − rank(yi), and n is the number of data points. A higher SRCC indicates a stronger monotonic relationship between the rankings of predicted and ground truth values

work page

[2] [2]

It is computed as: PLCC = Pn i=1(xi − ¯x)(yi − ¯y)pPn i=1(xi − ¯x)2pPn i=1(yi − ¯y)2 , where: • xi and yi are the data points, • ¯x and ¯y are the means of x and y, respectively

The PLCC quantifies the linear relationship between predicted and true values. It is computed as: PLCC = Pn i=1(xi − ¯x)(yi − ¯y)pPn i=1(xi − ¯x)2pPn i=1(yi − ¯y)2 , where: • xi and yi are the data points, • ¯x and ¯y are the means of x and y, respectively. A higher PLCC indicates a stronger linear relation- ship between predicted and ground truth values

work page

[3] [3]

arXiv preprint arXiv:2412.10360

Apollo: An exploration of video understand- ing in large multimodal models. arXiv preprint arXiv:2412.10360. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. 2024. Dynamath: A dy- namic visual benchmark for evaluating mathemati- cal reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836. A Metrics Equation ...

arXiv 2024

[4] [6]

It is defined as: R2 = 1 − Pn i=1(yi − ˆyi)2 Pn i=1(yi − ¯y)2 where: • yi are the ground truth values, • ˆyi are the predicted values, • ¯y is the mean of the ground truth values

The R2 score represents the proportion of variance in the ground truth values that is explained by the predictions. It is defined as: R2 = 1 − Pn i=1(yi − ˆyi)2 Pn i=1(yi − ¯y)2 where: • yi are the ground truth values, • ˆyi are the predicted values, • ¯y is the mean of the ground truth values. An R2 score closer to 1 indicates a better fit be- tween the ...

work page 2016

[5] [7]

9, it is evident that the dimension ‘lifeCycles’ exhibits the highest redundancy, particularly with ‘typesOf’

Key Observations from the Redundancy Maps: • In Fig. 9, it is evident that the dimension ‘lifeCycles’ exhibits the highest redundancy, particularly with ‘typesOf’. • Similarly, in Fig. 10, the ‘Instance Identity’ dimension shows the highest redundancy and is most closely related to ‘Scene Under- standing’

work page

[6] [8]

Bottom-50 Redun- dancy: • A clear pattern emerges when comparing the Top-50 and Bottom-50 redundancy maps

Trends in Top-50 vs. Bottom-50 Redun- dancy: • A clear pattern emerges when comparing the Top-50 and Bottom-50 redundancy maps. Nearly all Bottom-50 dimensions display significantly higher redundancy than their Top-50 counterparts. This observation sup- ports our conclusion that dimensions tend to exhibit greater redundancy for Bottom- 50 MLLMs compared t...

work page

[7] [9]

Implications for Redundancy Analysis: • To ensure a reasonable and accurate evalu- ation during redundancy analysis, it is cru- cial to exclude MLLMs with consistently poor performance. Including such models could skew the analysis by disproportion- ately inflating redundancy, as their universal underperformance does not provide mean- ingful insights into...

work page

[8] [10]

Calculate the dimensional redundancy within the benchmark, with particular attention to di- mensions exhibiting overall high redundancy

Dimension Redundancy Check. Calculate the dimensional redundancy within the benchmark, with particular attention to di- mensions exhibiting overall high redundancy. Analyze the redundancy heatmap to identify pairs of dimensions with exceptionally strong correlations, as these may indicate overlap- ping capabilities being assessed. For such cases, evaluate...

work page