{"id":"46afee98-2a55-41f7-9a33-6876045b6835","arxiv_id":"2507.23509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Image classifiers of different architectures use significantly different amounts and locations of image pixels to reach a decision, and misclassifications use slightly more pixels.","lead":"This paper measures the smallest set of pixels each image classifier needs to make its prediction, using a tool called ReX. It finds that different model architectures focus on different amounts and locations of the image, and that models use more pixels when they get an answer wrong.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central architecture comparison may be an artifact of ReX's approximate search and fixed OOD baseline rather than a true difference in minimal sufficient pixel sets.","rationale":"The reader's weakest-assumption analysis identified ReX's approximate MPS as the key vulnerability, and my reading agrees. The paper's headline conclusions are all measured in terms of these approximate sets, and the algorithm's acknowledged intractability (DP-complete for MPS computation, NP-complete for ranking) means that the approximation is not just a minor implementation detail. The magnitude of the architecture differences is so large that an unbiased approximation is necessary for the conclusion; if the approximation error correlates with architecture, the entire comparative study is called into question. I also note that the position claim (RQ2) is supported only by descriptive Dice and Hausdorff averages in Tables 2 and 3, with no hypothesis test or confidence interval, so the abstract's 'statistically different ... in ... position' is not actually established by the reported statistics. However, this is secondary to the approximation concern: even if position were statistically tested, the positions are still ReX's approximate MPSs. The proposed sensitivity check directly tests whether the architecture-level size differences are stable under reasonable changes to the baseline and search budget. If the ordering replicates, the central comparison gains credibility; if it does not, the conditional verdict would need to move toward rejection. The reader's verdict is already CONDITIONAL, and this concern is exactly the condition that needs to be met, so no verdict change is warranted at this stage.","tokens_in":12005,"tokens_out":3221,"duration_ms":37672,"concrete_test":"Run ReX on a random 100-image subset of the ImageNet validation set under three configurations: (a) default baseline=0, 20 iterations; (b) baseline=per-dataset mean RGB, 100 iterations; (c) baseline=Gaussian-blurred image, 100 iterations, each with 5 random seeds. Recompute the Kruskal-Wallis architecture comparison and the Inception-vs-ConvNext pairwise effect for MPS area. If the architecture ordering (Inception ~0.23 > ResNet ~0.14 > ViT ~0.10 > ConvNext/EVA ~0.06–0.08) does not replicate across all configurations, the reported differences are an artifact of ReX's baseline/search configuration rather than of the models' true minimal sufficient pixel sets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that different architectures have statistically different MPS size and position, and that incorrect classifications have larger MPSs. Every one of these conclusions rests on ReX's approximate MPS being an unbiased proxy for the model's true minimal sufficient pixel set. ReX (Section 3) uses a single default baseline (masking value 0, applied after processing, so virtually all mutants are out-of-distribution), 20 random 4-part partitions, and an approximate responsibility ranking that the authors state is NP-complete and therefore not exact. The reported size differences are large (Inception ~23–25% of image area vs ConvNext/EVA ~5–8%), but they could reflect systematic differences in how architectures respond to OOD masked inputs or in how often the approximate search terminates early, rather than genuine differences in the causal minimality of the pixel sets. The paper explicitly acknowledges that ReX's output is 'approximately minimal, but not necessarily the minimum,' and provides no validation that approximation error is unbiased across architectures. Without such validation, the architecture-level ranking and the 2.6% misclassification effect size may be artifacts of the search procedure and baseline choice rather than properties of the models themselves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using minimal sufficient pixel sets (MPSs), computed by the black-box causal XAI tool ReX, as a measure of 'concentration' for image classifiers. It compares 15 ImageNet-fine-tuned models from five architectures (ConvNext, EVA, Inception, ResNet, ViT) on 1000 ImageNet validation/test images (plus 50 Caltech-256 images), measuring MPS size relative to image area, location via Dice and Hausdorff distances, and the association between misclassification and MPS size. The paper reports statistically significant architecture differences in MPS size (Kruskal-Wallis, Friedman), descriptive location differences, and a 2.6% increase in MPS area for incorrect classifications from a mixed linear model.","tokens_in":12233,"tokens_out":5193,"duration_ms":50502,"significance":"If the MPS estimates are faithful, the paper provides a novel and useful comparative methodology: moving beyond accuracy to measure how much input evidence each model family uses. The empirical pattern—ConvNext/EVA using 5–8% of image area versus Inception's 23–25%—is striking and could inform model selection and OOD robustness discussions. The paper's strengths include the use of non-parametric matched tests, a same-seed replication across models, a Caltech-256 sanity check, and explicit acknowledgment of ReX's approximation. However, the study's central claims depend on ReX's approximate search being unbiased across architectures and on the location claim being backed by inferential statistics, both of which need further support.","major_comments":[{"comment":"The paper's abstract and introduction claim that different architectures have 'statistically different concentration, in both size and position,' but the location claim is supported only by average Dice and Hausdorff values with no significance test, confidence interval, or adjustment for multiple comparisons. Please add an inferential analysis (e.g., permutation tests on the per-image DC/Hausdorff distributions, or a mixed model with image and model as random effects) and report effect sizes with confidence intervals.","section":"§4.3, RQ2 (Tables 2–3)"},{"comment":"ReX is approximate—20 random 4-part partitions, a responsibility ranking that is NP-complete, and the paper's own statement that the MPS is 'approximately minimal, but not necessarily the minimum.' Because the default baseline (0 after processing) makes virtually all mutants out-of-distribution, the large architecture differences in Table 1 could reflect different architectures' sensitivity to OOD inputs or different approximation-error profiles rather than true differences in minimal sufficient pixel sets. Please provide robustness checks (varying iteration count, baseline value, and random seed; or cross-validating against another sufficient-set method) to show the architecture ranking is stable.","section":"§3, Algorithmic Overview"},{"comment":"The mixed linear model used to claim a 2.6% increase in MPS size for incorrect classifications is not described in enough detail to assess validity or reproduce. Please state the fixed and random effects, how model accuracy and architecture were controlled, whether the outcome was transformed, and provide a full coefficient table. Also check the Caltech-256 standard error, reported as '0.01%,' which appears implausible relative to the 3.3% estimate.","section":"§4.2–4.3, RQ3"}],"minor_comments":[{"comment":"Report standard deviations or confidence intervals for the mean MPS area ratios, and state the number of images per cell; the current table gives no sense of spread, and the caption's '3.6× larger' claim is not directly implied by the averaged entries (e.g., Inception-ResNet V2 0.254 vs ConvNext-V2 Large 0.089 is about 2.9×).","section":"Table 1"},{"comment":"The header calls the measure 'Hausdorff coefficient'; Hausdorff distance is a metric, not a coefficient, so please standardize the terminology.","section":"Table 3"},{"comment":"The Friedman test reporting (ConvNext p<0.01, Inception p>0.01, etc.) is unclear; please report test statistics and whether p-values are adjusted (e.g., Bonferroni) for the multiple intra-architecture comparisons.","section":"§4.3, RQ1"},{"comment":"The Caltech-256 validation uses only 50 images; the sentence 'evidence of inter-architecture differences are not found for the EVA, Inception or ResNet models' is ambiguous (intra-architecture?), and the small sample should be stated as a limitation in the main text.","section":"§4.3 and Table 4"},{"comment":"Add an explicit y-axis label and units; the violin plot is central to the size comparison, and the caption's color coding should be repeated in the figure itself.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest empirical study, but the lack of any inferential test for the location claim and the absence of validation of ReX approximation error across architectures leave the central 'statistically different concentration' claim only partially supported. I would encourage the authors to add the requested robustness analyses and statistical details; I believe major revision is appropriate rather than rejection, since the core dataset and analyses are reproducible in principle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth reading and worth a serious referee, but don't buy the position claim and treat the size ranking as a property of ReX's operational definition rather than a clean measure of true minimal pixel sets.\n\nWhat is genuinely new: Jiang et al. compared transformers with CNNs using patch-based MSEs; this paper compares 15 models across five architectures using pixel-level MPSs and finds a consistent, striking ordering—ConvNext/EVA sit around 5–8% of image area while Inception models sit around 23–25%. That ordering is the real empirical contribution, and it is backed by Kruskal-Wallis and Friedman tests. The RQ3 analysis is also decent: incorrect classifications are associated with a 2.6% larger MPS (SE 0.4%, p < 0.01) after controlling for model accuracy. The authors are explicit that ReX's search is approximate and that the MPS is not necessarily minimum, and they keep claims about human interpretability modest.\n\nNow the soft spots, in proportion. First, the big one: ReX uses 20 random 4-part partitions, an approximate responsibility ranking, and a fixed baseline of zero, so virtually every mutant is OOD. The paper acknowledges both the approximation and the OOD character, but it never validates that approximation error is unbiased across architectures. If the search terminates differently for some model families, or if some models cope with OOD masks better, the size ordering could partly be a search-and-baseline artifact. The differences are large and consistent, so I doubt this is the whole story, but the interpretation should be conditional. A sensitivity analysis with a different baseline and more iterations/seeds would settle it.\n\nSecond, RQ2: the abstract claims statistically different position, but the results section gives only descriptive Dice coefficients and Hausdorff distances for location. No significance test is reported. That mismatch should be fixed.\n\nThird, reproducibility: no code, no data, no spread in Table 1, and the Caltech-256 check uses only 50 images, so treat it as a sanity check, not confirmation. ReX is prior work from the same group; self-citation is not itself a flaw, but the lack of external replication of the MPS measure compounds the approximation concern.\n\nBottom line: this is a genuine new behavioral fingerprint for model selection and auditing, with one load-bearing caveat (approximation and OOD sensitivity) and one overstated axis (position). A strong revision can address both. I'd send it to review, bring it to the reading group, and cite it with a caveat.","headline":"The cross-architecture MPS size ordering is a real new empirical result, but the position claim is statistically unsupported and the central measure needs sensitivity analysis before being taken as ground truth.","tokens_in":12748,"tokens_out":5331,"would_cite":true,"duration_ms":53472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that image classifiers can be compared by the smallest set of pixels that preserves their top classification, and that this minimal set differs systematically by architecture and grows when the model is wrong.","keywords":["minimal sufficient pixel sets","causal explanations","image classification","model comparison","architecture comparison","ReX","model concentration","misclassification detection"],"falsifier":"Run ReX with a much larger number of random starting partitions, or an exact exhaustive search on small cropped images, and compare the resulting MPS sizes across architectures; if the ordering among ConvNext, EVA, Inception, ResNet, and ViT changes materially with search effort or seed, the claimed architecture-level concentration differences are an artifact of the approximation.","tokens_in":11801,"feed_emoji":"👁️","tokens_out":6793,"duration_ms":63989,"temperature":0.7,"pith_summary":"The paper's goal is to measure how much of an input image a classifier truly needs, by computing the smallest set of pixels that still produces the same top classification, and to use that quantity to compare architectures. Applying this measure to 15 ImageNet-finetuned models from five architecture families, the authors report that different architectures have statistically different 'concentration' in both the size and the location of these minimal pixel sets. The starkest contrasts are quantitative: ConvNext and EVA models decide from roughly 5-8% of image area, while Inception models need about 23-25%; misclassified images carry pixel sets about 2.6% larger than correctly classified ones. A sympathetic reader should care because, if accurate, this gives a new, model-internal axis for choosing among classifiers and a post-hoc warning signal for likely errors.","feed_headline":"Some vision models need only 5% of the image to decide","feed_subtitle":"Minimal-pixel analysis: ConvNext and EVA use 5-8% of an image, Inception 23-25%, and errors demand more pixels.","key_machinery":"The central object is the minimal sufficient pixel set (MPS): an approximately smallest subset of an image's pixels that, with all other pixels set to a baseline, still makes the model output the original top classification. The machinery that produces MPSs is ReX, a black-box causal-explanation algorithm that partitions the image, tests masked 'mutant' images against the model, assigns each pixel a causal responsibility score, ranks pixels by that score, and adds pixels until the classification is restored. Because exact minimality is DP-complete and the ranking step is NP-complete, ReX's output is approximately minimal and guaranteed sufficient, not minimal; the paper uses the highest-responsibility MPS per image and compares sets across models by size ratio, Sørensen-Dice coefficient, and Hausdorff distance.","core_discovery":"On the paper's own terms, the central discovery is a set of stable statistical differences in minimal sufficient pixel sets across models. For the same images, EVA and ConvNext architectures consistently yield small MPSs (5.4-8.1% of image area across the six models), Inception models yield large ones (23.0-25.0%), and ResNet and ViT lie in between; a Kruskal-Wallis test across architectures gives p<0.01, and a Friedman test shows intra-architecture differences for all families except Inception. Position also differs: average Sørensen-Dice overlaps between best-performing models are low, Hausdorff distances are high, and in some images the MPSs are entirely disjoint. On the validation set, a mixed linear model estimates that incorrect classifications are associated with a 2.6% increase in MPS area (standard error 0.4%, p<0.01). The authors interpret small MPSs as evidence that very large pretrained models are 'myopic' and may be overfit, and suggest MPS size could be used as a supplementary check on a model's output.","pith_inferences":["An implicit extension is to treat MPS size as a rough robustness probe: a model that accepts a very small set of zero-masked pixels is accepting strongly out-of-distribution inputs, so the architecture ordering found here may predict relative tolerance to other masked or corrupted inputs.","A natural testable extension is to vary the masking baseline: if the 5-8% versus 23-25% ordering persists under blur, mean-color, and noise baselines, the concentration differences are about the models themselves; if it is baseline-specific, the metric partly measures sensitivity to a particular perturbation family.","Because the authors compare only the single highest-responsibility MPS, a further extension would measure the full multiplicity of explanations per image; architectures that offer several disjoint alternative pixel sets may be less brittle than their smallest MPS alone suggests.","The 2.6% area increase for errors points to a confidence-free misclassification detector: in deployment, compute the MPS area and flag decisions whose area is far above the model's own correct-decision distribution, a test that does not require ground-truth labels."],"forward_implications":["Model families can be told apart by where and how much they look: with identical hyperparameters and seed, ConvNext and EVA rely on roughly 5-8% of the image while Inception relies on roughly 23-25%, so accuracy tables alone under-describe a classifier.","Small MPSs indicate that large pretrained models will happily classify heavily masked, out-of-distribution inputs, which the authors connect to overfitting and 'myopic' decision-making.","The statistically significant 2.6% increase in MPS area for wrong answers supports using MPS size as a post-classification diagnostic.","MPS overlap and distance measures show that architectures often base decisions on different image regions, so the visual evidence used by different models is genuinely different even when their classifications agree."],"supporting_citations":[{"why":"Supplies the ReX algorithm that computes each model's minimal sufficient pixel set, the core measurement used throughout.","marker":"[7]"},{"why":"Establishes the equivalence between the CKS explanation definition and actual causality, grounding the MPS definition.","marker":"[6]"},{"why":"Defines causal responsibility, which ReX uses to rank pixels when constructing the MPS.","marker":"[5]"},{"why":"Shows that exact explanation computation is DP-complete, motivating ReX's approximate search.","marker":"[12]"},{"why":"The prior patch-based SAG comparison of transformers and CNNs that this paper extends by moving to pixel-level MPSs.","marker":"[18]"},{"why":"Defines the Inception architecture family, whose models produce the largest MPSs in the study.","marker":"[37]"},{"why":"Defines the ConvNext architecture family, whose models produce some of the smallest MPSs.","marker":"[21]"},{"why":"Defines the EVA models, which together with ConvNext differ most sharply from the other architectures.","marker":"[13]"},{"why":"Defines the Vision Transformer architecture class included in the comparison.","marker":"[11]"},{"why":"Defines the ResNet architecture, a third convolutional family whose MPS behavior sits between Inception and ConvNext.","marker":"[16]"}],"fun_headline_variants":["Vision models differ in minimal pixels needed to classify","Some vision models need only 5% of an image to decide","ConvNext and EVA use minimal pixels, Inception uses many","Misclassified images demand more pixels, study finds","Pixel-concentration analysis reveals model-specific vision quirks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison collapses if ReX's approximate pixel sets do not faithfully track each model's true decision-relevant pixels, since exact minimality is computationally intractable and ReX uses only 20 iterations from random partitions; size and location differences could then be artifacts of the search rather than model behavior.","fun_headline_variants_meta":{"raw":{"variants":["Vision models differ in minimal pixels needed to classify","Some vision models need only 5% of an image to decide","ConvNext and EVA use minimal pixels, Inception uses many","Misclassified images demand more pixels, study finds","Pixel-concentration analysis reveals model-specific vision quirks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2671,"prompt_tokens":936,"completion_tokens":1735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1655}},"tokens_in":552,"tokens_out":1735,"duration_ms":12878,"temperature":1.0,"reasoning_tokens":1655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:39:41.340724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReX with a much larger number of random starting partitions, or an exact exhaustive search on small cropped images, and compare the resulting MPS sizes across architectures; if the ordering among ConvNext, EVA, Inception, ResNet, and ViT changes materially with search effort or seed, the claimed architecture-level concentration differences are an artifact of the approximation.","supporting_citations":[{"cited_title":"Kelly, Daniel Kroening, and Youcheng Sun","cited_arxiv_id":null,"evidence_quote":"Supplies the ReX algorithm that computes each model's minimal sufficient pixel set, the core measurement used throughout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the equivalence between the CKS explanation definition and actual causality, grounding the MPS definition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines causal responsibility, which ReX uses to rank pixels when constructing the MPS."},{"cited_title":"Complex- ity results for explanations in the structural-model ap- proach","cited_arxiv_id":null,"evidence_quote":"Shows that exact explanation computation is DP-complete, motivating ReX's approximate search."},{"cited_title":"Compar- ing the decision-making mechanisms by transformers and cnns via explanation methods","cited_arxiv_id":null,"evidence_quote":"The prior patch-based SAG comparison of transformers and CNNs that this paper extends by moving to pixel-level MPSs."},{"cited_title":"Rethinking the in- ception architecture for computer vision","cited_arxiv_id":null,"evidence_quote":"Defines the Inception architecture family, whose models produce the largest MPSs in the study."},{"cited_title":"A convnet for the 2020s","cited_arxiv_id":null,"evidence_quote":"Defines the ConvNext architecture family, whose models produce some of the smallest MPSs."},{"cited_title":"Eva: Exploring the limits of masked visual representation learning at scale","cited_arxiv_id":null,"evidence_quote":"Defines the EVA models, which together with ConvNext differ most sharply from the other architectures."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Defines the ResNet architecture, a third convolutional family whose MPS behavior sits between Inception and ConvNext."}],"review_version":1}