Pith. sign in

REVIEW 5 major objections 6 minor 12 references

Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A binary Vision Transformer reaches 90% accuracy on a 285-image COCO action subset.

desk verdict The paper's central claim is confounded: the 90% ViT result comes from a binary task while every comparator is evaluated on three classes, so the headline gap reflects task difficulty, not architecture. read the letter →

arxiv 2506.11678 v1 pith:72GGEVK4 submitted 2025-06-13 cs.CV cs.AI

MingZe Tang , Madiha Kazi This is my paper
classification cs.CVcs.AI
keywords humanactionrecognitionVisionTransformerCOCOsubsetsmalldataCLIPconvolutionalneuralnetworkexplainableAILeGrad
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether pretrained transformer representations can recognize human actions from a very small image collection. On a curated, label-audited COCO subset of 285 images spanning sitting, standing, and walking_running, the authors report that a fine-tuned binary Vision Transformer reaches 90% mean test accuracy with zero variance over five runs, compared with roughly 35% for CNNs, 40.7% for an FNN, and 61–64% for CLIP-based classifiers. A one-way ANOVA (F = 61.37, p < 0.001) is presented as evidence that these accuracy differences are statistically meaningful. SHAP and LeGrad heatmaps further suggest that the ViT grounds its predictions in pose-specific regions such as lower limbs for walking_running, while simpler networks attend to background textures. The authors read these results as evidence for the data efficiency of transformer representations and for explainability as a way to diagnose where small-data models fail.

What carries the argument

The load-bearing mechanism is the pretrained Vision Transformer (ViT), fine-tuned on the small COCO subset after resizing images to 224×224, with self-attention as the global-context operation that lets the model bind pose-relevant parts of the image. The argument is carried by comparing this ViT against carefully constructed baselines: an FNN, a three-layer CNN, a regularized CNN_gen, and two fixed-feature CLIP pipelines with a small MLP head, one of which appends cosine-similarity scores against each label's text embedding. A secondary mechanism is the explainability pair SHAP and LeGrad, which the paper uses to show that the ViT's saliency concentrates on the lower limbs for walking_running and on torsos and hips for standing and sitting, whereas simpler models disperse attention or latch onto background textures. The binary formulation matters: removing the intermediate 'standing' class is what isolates the visually separated action pair and produces the 90% result.

What would settle it

Retrain all seven models on an identical three-class sitting/standing/walking_running split with the same folds, or run the binary ViT on each of the three binary class pairs; if the ViT no longer exceeds CLIP by a wide margin under a matched task, the data-efficiency conclusion is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that, on this particular two-class sitting-versus-walking_running task, a fine-tuned binary Vision Transformer attains 90% mean test accuracy across five seeds, with standard deviation zero, clearly beating both convolutional baselines (≈34–35%) and CLIP-based pipelines (≈62–64%). The paper also reports that extending the same ViT to a three-class problem drops accuracy to 57.2%, below the CLIP embeddings' 63.9%, which the authors attribute to 'standing' sharing visual features with both sitting and walking_running. Using LeGrad attribution maps, the paper argues that the ViT's success comes from concentrating on semantically relevant body regions—lower limbs for gait actions, torsos for standing, hips and bench structures for sitting—rather than on background elements. The broader claim is that transformer representations are data-efficient on small, balanced action subsets, and that binary action discrimination is where that efficiency is most visible.

Load-bearing premise

The load-bearing assumption is that a 90% accuracy on a two-class task is directly comparable to a 34–64% accuracy on a three-class task; if binary and multiclass accuracies are not commensurable, the headline gap does not by itself show that transformers are more data-efficient.

Editorial extensions

If this is right

  • If the binary ViT result holds, coarse two-way action discrimination, such as stationary sitting versus active gait, can be done very accurately from a few hundred labeled images with a pretrained transformer.
  • Saliency methods like LeGrad can be used before deployment to check whether a small-data action model is reasoning from pose regions or from background confounds.
  • The multiclass ViT result implies that introducing a visually intermediate class such as standing is the main remaining bottleneck, so future gains are more likely from resolving posture ambiguity than from architecture choice alone.
  • A fixed CLIP encoder with a lightweight head is a competitive alternative on the multiclass task and may be preferable when a binary formulation is not available.
  • Accuracy numbers on this subset should be read as task-specific rather than as evidence that one architecture family generally dominates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that much of the reported 90% versus 64% gap reflects task difficulty, with a 50% chance baseline for the binary task and a 33% chance baseline for the three-class task, rather than a general architectural advantage; the multiclass ViT's 57.2% being below CLIP's 63.9% is consistent with that reading.
  • A natural testable extension is to train every model on every binary pair from the same three classes; if the ViT wins all pairs, the data-efficiency story is stronger, and if it only wins the sitting-versus-walking_running pair, the story is about that specific discrimination.
  • The paper's label-auditing step suggests that annotation noise can silently change which model looks best; applying the same audit to the validation split and reporting per-class confusion would let readers separate architecture effects from label effects.
  • I infer that adding temporal context, or using a pose-estimation backbone rather than raw pixels, may close the multiclass gap faster than adding more transformer capacity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript curates a 285-image three-class COCO subset (sitting, standing, walking_running), trains or fine-tunes FNN, CNN, CNN_gen, CLIP, CLIP Cosine, a binary ViT, and a multiclass ViT, and reports mean test accuracy, precision, recall, and F1 over five seeds (42-46). It reports that the binary ViT reaches 90.0% accuracy with zero standard deviation, whereas multiclass models reach roughly 35% (CNNs), 62-64% (CLIP), and 57.2% (multiclass ViT). A one-way ANOVA is reported as F=61.37, p<0.001, and SHAP/LeGrad visualizations are used to argue that the binary ViT focuses on pose-specific regions such as lower limbs for walking_running. The paper concludes that transformer representations are data-efficient for human action recognition on small COCO subsets.

Significance. The paper has some strengths: it fixes an explicit stratified split, reports five-seed statistics, and provides qualitative error and explanation analyses that could inform future small-data action recognition studies. However, the central comparison is confounded because the headline ViT result is obtained on a binary task while every other model is evaluated on a three-class task. As a result, the 90% versus 35-64% gap does not support the paper's data-efficiency conclusion, and the reported ANOVA tests a global hypothesis rather than the specific pairwise claims in the abstract. The useful contribution is therefore a preliminary benchmark description and an explainability case study, not a valid architecture comparison. If the authors rerun all models on the same task or restrict the conclusions to within-task comparisons, the work could become a worthwhile empirical note.

major comments (5)
  1. [§3.1, Table 2; §2.2.3] The central comparison is confounded by task type. ViT Binary is trained on two classes (sitting versus walking_running, chance 50%), while FNN, CNN, CNN_gen, CLIP, CLIP Cosine, and ViT MultiClass are trained on three classes (chance 33%). Listing all seven models in one table and directly comparing mean accuracies in §3.1 and the abstract therefore conflates architecture quality with task difficulty. Section 4.2 itself states that 'standing' shares visual features with both other classes and causes the multiclass ViT accuracy to drop to 57.2%, which confirms that the label space drives much of the gap. The data-efficiency claim in the abstract does not follow from these experiments; the authors should either evaluate all models on the same binary task, evaluate a binary version of every baseline, or report chance-normalized metrics and restrict all claims to within-task comparisons.
  2. [§3.4] The one-way ANOVA compares the five accuracy values of seven models and only establishes that at least one group mean differs; it does not test the specific pairwise differences claimed in the abstract, such as ViT Binary significantly exceeding CLIP or CNN. Because model identity is fully confounded with task type (binary versus multiclass), F=61.37 cannot be attributed to architecture. A proper analysis would require post-hoc pairwise comparisons with correction, or a model that includes task type as a factor, and it should also address the zero-variance ViT Binary group.
  3. [§2.2.4 versus §2.3] The evaluation protocol for CLIP contradicts the protocol for all other models. Section 2.2.4 states that CLIP uses an 80/20 train/test split, while Section 2.3 states that every architecture uses a fixed 80/10/10 stratified train/validation/test split. This inconsistency means the CLIP results in Table 2 were obtained under a different data-split and validation procedure, further undermining comparability across models. The authors should use the identical split and validation protocol for every model.
  4. [Table 2, §3.3] ViT Binary is reported as 0.900 ± 0.000 over five independent trials with seeds 42-46. With stochastic training, early stopping, and five different seeds, an exactly zero standard deviation is surprising and needs an explanation. The authors should report the per-run test accuracies, the test-set size, and a confusion matrix, or explain why the results are deterministic. Without this, readers cannot assess whether the 90% figure is a stable finding or an artifact of a very small test set.
  5. [§2.1] The manual dataset audit is described only qualitatively: the authors say that 'all such noise samples were removed prior to model training,' but they do not state how many images were removed, what criteria were used, or whether the removal was independently verified. Since the dataset contains only 285 images, this manual editing can materially change every reported accuracy and is not reproducible. The authors should provide the filtered image IDs, a detailed audit protocol, and a discussion of how the removals affect class balance and the comparison across models.
minor comments (6)
  1. [§1] The introduction contains a typo: 'walking_unning' should be 'walking_running'.
  2. [§2.3, §3.4] The text contains spacing errors in 'ANOV A' and 'one -way ANOV A'; these should be corrected to 'ANOVA' and 'one-way ANOVA'.
  3. [Table 2] The table formatting runs the row label into the value ('CNN_base0.343'); ensure proper spacing so the model name and accuracy are visually separated.
  4. [§2.2.5] The text says concatenating the 3-dimensional cosine-similarity vector to the 512-dimensional CLIP embedding 'doubles' the feature size; 512+3 does not double it. Clarify the intended feature dimension.
  5. [§3.1] The sentence says 'The two binary vision transformer based models' but only one binary ViT is reported; correct the plural or describe the second model.
  6. [§2.1, §4.2] The paper does not state whether code and the curated dataset will be released. Given the manual audit step, a data-release or reproducibility statement is important.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline numbers are measured accuracies; the binary-vs-multiclass comparison is confounded by task difficulty, but no result reduces to its input by construction.

full rationale

The paper contains no formal derivation whose conclusion is equivalent to its assumptions. The central quantitative claims are empirical test accuracies from held-out splits (Table 2), not predictions derived from fitted parameters renamed as outputs. The headline 90% versus 63.9% gap is computed across different classification tasks (binary vs. three-class), which is a serious internal-validity concern, but it is not circularity: the accuracies are not defined to equal each other or to force the architecture comparison. The interpretability analysis (LeGrad and SHAP) describes the same models it is used to explain, which is standard post-hoc explanation rather than a claim that the maps were independently predicted from pose annotations; no equation or construction collapses here. There are no load-bearing self-citations: the cited works (COCO, CLIP, LeGrad, SHAP, the ViT checkpoint) are independent external resources or methodological tools, and the authors do not invoke their own prior work to justify any premise. The paper's own error analysis in Section 4.2 explicitly states that the binary-versus-multiclass gap arises because standing shares visual features with both sitting and walking_running, confirming that the reported difference is task-compositional rather than a circular derivation. Overall, no identified step satisfies the hard requirement of exhibiting a specific reduction to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on new theoretical quantities or fitted constants. The free parameters listed are hand-chosen training and data-splitting choices that affect the reported accuracy gaps. The main assumptions are that pretrained representations transfer, that attribution maps are faithful, and that the manual data audit is unbiased. No invented entities are introduced.

free parameters (3)
  • Fixed 80/10/10 stratified split and seeds 42-46 = 80/10/10, seeds 42-46
    All models are evaluated on one fixed split; the reported order of accuracies may be specific to this split. The split seed is global but the actual test image IDs are not given.
  • Early stopping patience = 5 epochs
    Used for all models; the convergence point changes test accuracy and can change model rankings.
  • CNN_gen augmentation and regularization settings = rotation +/-25 degrees, shift +/-20 percent, brightness 0.8-1.2x, L2 1e-4, dropout 0.2
    These hand-chosen settings are specific to the small dataset and directly affect CNN_gen accuracy.
assumptions (4)
  • domain assumption Pretrained ViT and CLIP checkpoints provide transferable representations for human pose classification.
    Used in sections 2.2.3 and 2.2.4; the superiority claim depends on transfer learning being effective for this task.
  • domain assumption SHAP and LeGrad attribution maps faithfully reflect the features the models actually use.
    Section 4.1 uses these maps to conclude ViT attends to pose regions; no ablation or ground-truth pose verification is provided.
  • ad hoc to paper The manual audit removed only genuine noise and did not bias the comparison.
    Section 2.1 and Figure 5 describe removing unclassifiable and mislabeled images without formal criteria or a list of removed IDs; if the audit favored models that ignore context, the results change.
  • domain assumption The COCO-derived activity labels are valid ground truth for sitting, standing, and walking_running.
    Section 2.1 states labels are from COCO entries, but the paper does not explain label provenance and gives an example of a sitting label it considers wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets." pith.science (2026). https://pith.science/paper/72GGEVK4

@misc{pith2026250611678,
  author       = {Pith},
  title        = {Pith review of: Pose Matters: Evaluating Vision Transformers and CNNs for Human Action Recognition on Small COCO Subsets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72GGEVK4}},
  note         = {Machine review of arXiv:2506.11678}
}
read the original abstract

This study explores human action recognition using a three-class subset of the COCO image corpus, benchmarking models from simple fully connected networks to transformer architectures. The binary Vision Transformer (ViT) achieved 90% mean test accuracy, significantly exceeding multiclass classifiers such as convolutional networks (approximately 35%) and CLIP-based models (approximately 62-64%). A one-way ANOVA (F = 61.37, p < 0.001) confirmed these differences are statistically significant. Qualitative analysis with SHAP explainer and LeGrad heatmaps indicated that the ViT localizes pose-specific regions (e.g., lower limbs for walking or running), while simpler feed-forward models often focus on background textures, explaining their errors. These findings emphasize the data efficiency of transformer representations and the importance of explainability techniques in diagnosing class-specific failures.

Figures

Figures reproduced from arXiv: 2506.11678 by the authors.

Figure 1
Figure 1. Random sample for each class superior but also why certain architectures succeed or fail under limited data constraints. 2 Description of Data and Methods 2.1 Data A curated subset of the COCO benchmark (Lin et al., 2015) was employed, in which each en￾try originally comprised a license code, file name, COCO URL, Flickr URL, capture date, image di￾mensions, unique identifier, and an activity label (sitting, standing… view at source ↗
Figure 2
Figure 2. (a) Distribution of image labels (b) Height vs. width scatter by class (c) Aspect ratio histogram [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Model Accuracy Comparison a higher false positive rate. Recall and F1 score reflect similar patterns - the transformer models dominate on both metrics, while the dense and con￾volutional baselines lag behind. 3.3 Variability Across Runs Standard deviations highlight stability - ViT Binary has zero variance, indicating deterministic behav￾ior under the fixed seed values. Both CLIP models present higher variability (±… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Model Predictions for Error Analysis walking_running, standing). The empirical evaluation demonstrates that both the baseline CNN and the augmented variant suc￾cumb to overfitting, mainly because the dataset is undersized (Yamashita et al., 2018). With far fewer indepe…
Figure 5
Figure 5. Figure 5: Example of Poor Dataset Annotation from "Sitting" Label The exemplar image ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Example of Unclassifiable Data 6 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: LeGrad overlays and model predictions for one “sitting”, “walking_running” and “standing” image (top [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: SHAP Explainer for Feed-forward Neural Network Model [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: SHAP Explainer for Convolutional Neural Network Model [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Walid Bousselham, Angie Boggust, Sofian Chaybouti, Hendrik Strobelt, and Hilde Kuehne. 2025. http://arxiv.org/abs/2404.03214 Legrad: An explainability method for vision transformers via feature formation sensitivity

    arXiv 2025

  4. [4]

    Jason Brownlee. 2022. https://machinelearningmastery.com/when-to-use-mlp-cnn-and-rnn-neural-networks/ When to use mlp, cnn, and rnn neural networks

    work page 2022

  5. [5]

    Muqing Li, Ziyi Zhu, Ruilin Xu, Yinqiu Feng, and Lingxi Xiao. 2024. https://doi.org/10.54097/qg7hakzu Research on image classification and semantic segmentation model based on convolutional neural network . Journal of Computing and Electronic Information Management, 12:94--100

    work page doi:10.54097/qg7hakzu 2024

  6. [6]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. http://arxiv.org/abs/1405.0312 Microsoft coco: Common objects in context

    arXiv 2015

  7. [7]

    Scott Lundberg and Su-In Lee. 2017. http://arxiv.org/abs/1705.07874 A unified approach to interpreting model predictions

    arXiv 2017

  8. [8]

    Bala Priya. 2023. https://www.pinecone.io/learn/softmax-activation/ Softmax activation function: Everything you need to know

    work page 2023

Show all 12 references
  1. [9]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. http://arxiv.org/abs/2103.00020 Learning transferable visual models from natural lang...

    2021 arXiv

  2. [10]

    Sreekanth. 2024. https://huggingface.co/Sreekanth3096/vit-coco-image-classification Sreekanth3096/vit-coco-image-classification · hugging face . Sreekanth3096/vit-coco-image-classification · Hugging Face

    2024

  3. [11]

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. 2022. http://arxiv.org/abs/2106.10270 How to train your vit? data, augmentation, and regularization in vision transformers

    2022 arXiv

  4. [12]

    Rikiya Yamashita, Mizuho Nishio, Richard Kinh Gian Do, and Kaori Togashi. 2018. https://doi.org/10.1007/s13244-018-0639-9 Convolutional neural networks: an overview and application in radiology . Insights Imaging, 9:611--629

    2018 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.