Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Leveraging Vision-Language Pre-training for Human Activity Recognition in Still Images

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning CLIP lifts still-image activity recognition from 41% to 76%.

desk verdict A cleanly written small-data CLIP fine-tuning study whose headline 76% vs 41% is plausible but rests on unvalidated labels and a ~29-image test set; the strong 'decisively proves' claim doesn't hold. read the letter →

arxiv 2506.13458 v1 pith:PSHTJBXI submitted 2025-06-16 cs.CV cs.CL

Cristina Mahanta , Gagan Bhatia This is my paper · ORCID
classification cs.CVcs.CL
keywords humanactivityrecognitionstillimagesCLIPvision-languagepre-trainingtransferlearningimageclassificationMSCOCO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that contrastive vision-language pretraining, specifically fine-tuning a CLIP encoder, sharply outperforms neural networks trained from scratch when the task is to recognize human activities in a single still photograph with no motion information. On a curated set of 285 MSCOCO validation images split into walking/running, sitting, and standing, the best from-scratch CNN reaches about 41% test accuracy while the fine-tuned CLIP model reaches 76%. The paper presents this as evidence that a single still image can carry enough pose-related signal for a pretrained multimodal model to classify everyday activities, and that practitioners should fine-tune such a model rather than train a small CNN from scratch. It also reports that CLIP beats a pretrained ViT and SigLIP2, and that raw CLIP image embeddings outperform cosine-similarity scores against label text.

What carries the argument

The load-bearing object is CLIP's contrastively trained joint vision-language embedding space, used in three configurations. In the configuration that produces the headline result, a CLIP image encoder is fine-tuned end-to-end with a three-class cross-entropy head (CLIP_IC), allowing the pretrained representations to adapt to the still-image activity task. The other two configurations, a small MLP on raw CLIP image vectors (CLIP_EM) and an MLP on cosine-similarity scores between image and label-text embeddings (CLIP_CS), act as ablations separating the value of the pretrained visual features from the value of the text-alignment signal. The paper also uses LeGrad attention maps to show that attention sometimes drifts to background objects, which it offers as the main reason accuracy stays below 80%.

What would settle it

Have three independent annotators re-label the same 285 images under a written protocol; if agreement is low (for example, below 90%), or if a CNN trained from scratch on the corrected labels reaches CLIP's 76%, then the reported advantage is an artifact of the ground-truth labels rather than of vision-language pretraining.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuning a contrastively pretrained CLIP model on just 285 labelled still images yields 76% test accuracy for three-way human activity classification, nearly doubling the roughly 41% achieved by a CNN trained from scratch with augmentation and regularisation. The paper argues the advantage is statistically significant and comes from CLIP's joint image-text pretraining, which provides feature separations that survive the absence of motion cues. It further claims that the same pretraining helps ViT and SigLIP2 but to a lesser degree, and that the fine-tuned CLIP classifier beats a shallow MLP trained on CLIP embeddings, which in turn beats one trained only on cosine similarities between image and label-text embeddings.

Load-bearing premise

The 285 images are assumed to be correctly and consistently labelled as walking/running, sitting, or standing, but the paper never says who assigned the labels, what instructions they followed, or how disagreements were resolved.

Editorial extensions

If this is right

  • Fine-tuning a pretrained vision-language model on a small labelled set is enough to lift three-class static activity recognition from 41% to 76%, with no motion or temporal input.
  • Raw CLIP image embeddings preserve more class information than cosine-similarity scores to label text, so practitioners should avoid compressing the embedding to label-similarity features.
  • The remaining error pattern, small, occluded, or low-resolution people and borderline poses, defines a concrete ceiling around 80% for static-image CLIP classifiers.
  • The statistical comparison across models (ANOVA, p<0.001) supports treating the gap between CLIP and the other backbones as more than within-run noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not describe how the 285 activity labels were created, so the headline gap should be re-run on an independently labelled set before treating 76% as a deployment number.
  • A natural extension the paper leaves implicit is zero-shot CLIP: if the joint image-text space is already well aligned, text prompts alone may classify these activities without any fine-tuning, which would isolate how much of the 76% comes from pretraining versus adaptation.
  • Because the errors concentrate on low-amplitude motion and occlusion, adding a second frame or a lightweight temporal module could plausibly push past the 80% ceiling the paper identifies.
  • The same pipeline could be tested on finer-grained activities such as bending, carrying, or interacting with objects, where the pose differences are smaller and the pretraining advantage may shrink or grow.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies three-class still-image human activity recognition (walking/running, sitting, standing) on a curated subset of 285 MSCOCO validation images. It compares scratch-trained CNN and FNN baselines, an augmented CNN (CNN_gen), fine-tuned ViT, CLIP, and SigLIP2 backbones, and MLPs trained on CLIP image embeddings or image-text cosine similarities. The headline result is that the fine-tuned CLIP_IC model reaches 76% accuracy versus about 41% for CNN_gen, which the authors interpret as demonstrating that contrastive vision-language pretraining decisively improves still-image action recognition. The paper also reports image-resolution statistics, an augmentation ablation, a global ANOVA, and LeGrad saliency maps for interpretability.

Significance. If the central comparison were properly supported, the result would be a useful small-scale confirmation that a fine-tuned CLIP backbone outperforms a small CNN trained from scratch on a three-way activity classification task. The paper has positive features: it is transparent about class balance and image-resolution statistics, it fixes a random seed and reports five-run means, and it includes concrete failure-case and explainability analyses. However, the evidentiary value is currently limited by the absence of validated ground-truth labels, the very small test set, and the lack of pairwise statistical comparisons, so the claim of a 'decisive' improvement is not yet established.

major comments (4)
  1. [§2.1 (Data)] The activity labels walking/running, sitting, and standing are not part of MSCOCO's annotation schema, yet the paper does not state who created the labels for the 285 selected images, what definitional instructions were used, or what inter-annotator agreement was achieved. Because every accuracy value in Tables 2–7 is computed against these labels, the headline 76% accuracy for CLIP_IC is only as trustworthy as the label set; if labels were produced by a single annotator selecting only clear poses, the reported gap between CLIP and the CNNs could reflect label artifacts rather than a genuine advantage of contrastive pretraining. This is a load-bearing point that must be addressed with a documented annotation protocol and an agreement measure.
  2. [§2.3 and Table 7 (Experimental Approach and Results)] The 80%–10%–10% split of 285 images yields a test set of roughly 29 images, so the 0.759 test accuracy for CLIP_IC corresponds to about 22 images and the 0.386 for CNN_gen to about 11 images; this difference of roughly 11 images is asserted without confidence intervals. The reported standard deviation of exactly 0.000 for CLIP_IC over five runs requires explanation and, if the runs were identical by design, provides no variance information. The one-way ANOVA (F=23.4562, p<0.001) is a global test across all seven models and does not establish that CLIP_IC is significantly better than CNN_gen or SigLIP2; the authors should report pairwise tests with multiple-comparison correction or confidence intervals on the accuracy differences.
  3. [§2.2 and §2.3 (Model configurations)] The transfer-learning results are not reproducible as reported: the paper says the ViT, CLIP, and SigLIP2 backbones are fine-tuned end-to-end under cross-entropy, but it does not state the optimizer, learning rate, batch size, number of epochs, weight decay, or input preprocessing for these models. In addition, the relationship between the 'binary classification' experiments of Table 5 and the multiclass CLIP_IC model of Table 7 is unclear; if the binary models were diagnostic only, this should be stated explicitly. Since the central 76% result depends entirely on the fine-tuning configuration, this information is not optional.
  4. [§2.3 and Table 3 (Augmentation selection)] The augmentation policy for CNN_gen was selected from validation-set results in Table 3, where the validation set contains roughly 28 images. This is model selection on a very small sample, so the reported test accuracy for CNN_gen may be optimistically biased; the comparison between CLIP_IC and CNN_gen is therefore not a controlled comparison of pretraining alone. The authors should either hold out the augmentation choice on a separate validation split or report the variability of validation accuracy across augmentation configurations.
minor comments (5)
  1. [Abstract and Conclusion] The abstract claims that CLIP pretraining 'decisively improves' still-image action recognition 'in real deployments,' but the experiments use 285 curated still images and the discussion itself notes that accuracy still struggles to exceed 80%; the wording should be tempered.
  2. [Table 7 caption] The caption of Table 7 refers to 'CLIP IM' while the text and Table 6 use 'CLIP EM' for the model trained on CLIP image embeddings; please unify the notation.
  3. [Figure 3] Figure 3 is described as 'Evaluation of ViT model trained on binary classes,' but the caption does not state which two classes are used or whether the confusion matrix is from the training or test split; please clarify.
  4. [§2.2 (CNN and FNN)] The paper says the FNN is a 'parameter-matched dense network,' but no parameter counts are provided; please add the counts or remove the claim.
  5. [Throughout] There are several typographical and formatting issues, including 'Explainablity' in Section 4, 'ANOV A' in Section 2.3, 'given a image' in Section 2.2, and inconsistently spaced 'i.e.' occurrences; a careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline results are measured performances from external pretrained models or scratch baselines, not derived from the paper's inputs.

full rationale

The paper's central claim is an empirical comparison: scratch CNNs score 41% and fine-tuned CLIP scores 76% on a 285-image subset of MSCOCO. No quantity is fitted and then renamed as a prediction; no parameter is defined in terms of the target result; and no load-bearing argument is justified by a self-citation. The CLIP, ViT, and SigLIP2 backbones are external pretrained resources cited to independent sources (Radford et al. 2021; Dosovitskiy et al. 2021; Tschannen et al. 2025), and the accuracy values are reported test-set measurements, not identities derivable from dataset statistics. The only choices informed by data are the augmentation policy selected from validation results and hyperparameters, which is standard model selection rather than circular reasoning. The unvalidated provenance of the activity labels is a legitimate correctness and reproducibility concern, but it does not make any step circular: the labels are inputs, and the accuracies do not reduce to them by construction. The explainability and error analyses are post-hoc interpretations, not derived predictions. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the correctness and representativeness of hand-made activity labels plus assumptions about statistical reliability and pretraining data leakage. There are no new entities; the only hand-chosen components are the augmentation policy and unspecified fine-tuning hyperparameters.

free parameters (2)
  • CNN_gen augmentation policy = vertical flip, perspective transform, random resized crop
    Selected after inspecting the validation accuracies in Table 3; this hand-picked policy affects the CNN_gen result and was not independently justified.
  • Fine-tuning hyperparameters (learning rate, batch size, epochs, optimizer)
    The paper reports no learning rates, batch sizes, epochs, optimizers, or head sizes for CLIP, ViT, or SigLIP2 fine-tuning; these choices influence every reported accuracy.
assumptions (4)
  • domain assumption The activity labels assigned to the 285 COCO images are correct.
    COCO provides object annotations, not walking/sitting/standing labels; the paper does not describe the labeling protocol, annotators, or agreement (Section 2.1).
  • domain assumption The selected images are representative of real-world still-image activity recognition.
    Images were selected by URL with unspecified criteria to depict exactly one activity; no random sampling from a defined population is described (Section 2.1).
  • domain assumption CLIP's pretraining data does not overlap with the 285 evaluation images.
    No leakage check is reported, and CLIP was trained on 400 million web image-text pairs (Radford et al., 2021), which could contain similar photos (Section 2.2).
  • domain assumption A test set of about 29 images yields reliable accuracy estimates.
    The 10% test split is about 29 images; accuracy on this sample has a large standard error, and the paper does not compute confidence intervals (Section 2.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Vision-Language Pre-training for Human Activity Recognition in Still Images." pith.science (2026). https://pith.science/paper/PSHTJBXI

@misc{pith2026250613458,
  author       = {Pith},
  title        = {Pith review of: Leveraging Vision-Language Pre-training for Human Activity Recognition in Still Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSHTJBXI}},
  note         = {Machine review of arXiv:2506.13458}
}
read the original abstract

Recognising human activity in a single photo enables indexing, safety and assistive applications, yet lacks motion cues. Using 285 MSCOCO images labelled as walking, running, sitting, and standing, scratch CNNs scored 41% accuracy. Fine-tuning multimodal CLIP raised this to 76%, demonstrating that contrastive vision-language pre-training decisively improves still-image action recognition in real-world deployments.

Figures

Figures reproduced from arXiv: 2506.13458 by the authors.

Figure 1
Figure 1. Examples of the training data From the full COCO set, we selected 285 images depicting exactly one of the three human activi￾ties—walking/running (98 images), sitting (95 im￾ages), or standing (92 images) —yielding a nearly balanced three-way classification problem. All im￾ages were downloaded directly via their URLS, and none were discarded due to corruption or missing annotations, confirming complete data integrit… view at source ↗
Figure 2
Figure 2. Exploratory data analysis of image dimensions: (a) width vs. height scatter, (b) height/width box [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Evaluation of ViT model trained on binary [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Explainability of model Model Test Accuracy Precision Recall F1 Score CLIPIC 0.759±0.000 0.743±0.000 0.740±0.000 0.737±0.000 Siglip2 0.634±0.028 0.635±0.025 0.648±0.021 0.636±0.025 ViT 0.462±0.017 0.462±0.029 0.437±0.035 0.434±0.026 CNNgen 0.386±0.114 0.420±0.127 0.409…
Figure 5
Figure 5. Figure 5: Error analysis of the models 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 linked inside Pith

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Walid Bousselham, Angie Boggust, Sofian Chaybouti, Hendrik Strobelt, and Hilde Kuehne. 2024. http://arxiv.org/abs/2404.03214v2 Legrad: An explainability method for vision transformers via feature formation sensitivity

    arXiv 2024

  4. [4]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. http://arxiv.org/abs/2010.11929 An image is worth 16x16 words: Transformers for image recognition at scale

    arXiv 2021

  5. [5]

    Sergey Ioffe and Christian Szegedy. 2015. http://arxiv.org/abs/1502.03167 Batch normalization: Accelerating deep network training by reducing internal covariate shift

    arXiv 2015

  6. [6]

    Lecun, L

    Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. https://doi.org/10.1109/5.726791 Gradient-based learning applied to document recognition . Proceedings of the IEEE, 86(11):2278--2324

    doi:10.1109/5.726791 1998

  7. [7]

    Bengio, and Geoffrey Hinton

    Yann LeCun, Y. Bengio, and Geoffrey Hinton. 2015. https://doi.org/10.1038/nature14539 Deep learning . Nature, 521:436--44

    doi:10.1038/nature14539 2015

  8. [8]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. http://arxiv.org/abs/1405.0312 Microsoft coco: Common objects in context

    arXiv 2015

Show all 12 references
  1. [9]

    Keiron O'Shea and Ryan Nash. 2015. http://arxiv.org/abs/1511.08458 An introduction to convolutional neural networks

    2015 arXiv

  2. [10]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  3. [11]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. http://arxiv.org/abs/2103.00020v1 Learning transferable visual models from natural la...

    2021 arXiv

  4. [12]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. 2025. http://arxiv.org/abs/2502.14786v1 Si...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.