REVIEW 5 major objections 4 minor 1 cited by
SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization
T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read SeFAR shows that reversing short fine-grained temporal chunks, while preserving context, yields state-of-the-art semi-supervised fine-grained action recognition.
desk verdict Solid FixMatch-style SSL framework for fine-grained action recognition with honest ablations, but the SOTA claim rests on only three re-implemented baselines and no variance estimates. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the dual-level temporal elements: from $K$ sampled frames, SeFAR builds $M$ fine-grained elements $\{p_i\}_{i=1}^M$, each of length $L$, together with a context element $p_{\mathrm{context}}$ of length $2L$. The strong augmentation is $A_{\mathrm{strong}}(\{p_i\}) = \{\overleftarrow{p_i}\}$ while $p_{\mathrm{context}}$ stays in its original order, forming the weak-to-strong pair for consistency regularization. The stabilizing mechanism is the Adaptive Regulation: the teacher predicts each unlabeled video $U$ times, the maximum confidence $\mu^*$ and its standard deviation $\sigma^*$ are extracted, and an adaptive coefficient $\eta = \tau_1 \tau_2$ is computed from $\tau_1 = \mathrm{sigmoid}(e^{\mu^*}-e)$ and $\tau_2 = \mathrm{sigmoid}(1/(\beta\sigma^*+\epsilon))-0.5$ to reweight the unsupervised loss. These pieces carry the claim by converting the generic FixMatch recipe into a training loop tailored to fine-grained temporal semantics.
What would settle it
Build a benchmark whose classes differ only by the motion direction inside a single short clip, with the surrounding context identical across classes (for example, "hand turns left" vs. "hand turns right" with the same background and body pose). If SeFAR's accuracy on that benchmark is no better than, or worse than, a spatial-only augmentation baseline, the semantic-preservation assumption behind chunk reversal is falsified. A simpler check: on any fine-grained dataset, compare the teacher's pseudo-label accuracy on weakly augmented videos with the true-label accuracy of the same teacher on strongly augmented videos; a large gap indicates the strong augmentation has broken the label.
Extended reading notes
Core claim
The central claim is that consistency regularization for fine-grained actions should be built from temporal, not spatial, structure: a video is decomposed into dual-level temporal elements, and the strong augmentation reverses the short fine-grained elements while preserving the order of the context element, producing a "moderate temporal perturbation" that is strong enough to be a useful training signal yet weak enough to keep the action's semantics and temporal directionality intact. On top of this, SeFAR stabilizes learning by having the teacher predict each unlabeled video several times, computing the mean confidence and standard deviation of those predictions, and deriving an adaptive coefficient that up-weights confident predictions and down-weights unstable ones. The paper reports that this combination achieves state-of-the-art accuracy on FineGym and FineDiving across data scopes and label rates, outperforms prior semi-supervised methods on UCF101 and HMDB51, and yields features that improve fine-grained question answering when used as a visual encoder in an MLLM.
Load-bearing premise
The load-bearing premise is that reversing each short fine-grained chunk while leaving the context chunk in its original order is a strong augmentation that perturbs the student's input enough to be useful without changing what the correct label is; if local motion direction is what distinguishes a class, and the preserved context cannot disambiguate it, the consistency loss would teach the student to ignore the defining cue.
Editorial extensions
If this is right
- With only 5% to 10% of labels, SeFAR outperforms prior semi-supervised action recognition methods on FineGym and FineDiving at event, set, and element granularities.
- The same recipe transfers to coarse-grained datasets, beating previous SSL action recognition methods on UCF101 and HMDB51 under the same label scopes.
- A SeFAR encoder trained on 5% FineGym data improves fine-grained Gym-QA accuracy over standard CLIP-style visual encoders in an MLLM, suggesting that semi-supervised FAR pretraining can substitute for large labeled video corpora in specialized domains.
- Reversing only fine-grained elements outperforms spatial-only augmentation, temporal warping, full shuffle, full reverse, and speed-based perturbations, especially on direction-sensitive action pairs.
- The adaptive weighting helps most at low labeling rates: on FineDiving the gain from Adaptive Regulation grows as the label rate drops from 10% to 1%.
Reading between the lines
- Beyond the paper, the same dual-level reversal recipe could be tested on other temporally sensitive semi-supervised tasks, such as surgical phase recognition or gesture segmentation, where short local motions matter but global order carries meaning.
- Beyond the paper, the hand-designed sigmoid coefficients in Adaptive Regulation could be replaced by a calibrated uncertainty estimate over pseudo-labels, which would make the weighting principle dataset-agnostic rather than tuned to these benchmarks.
- Beyond the paper, because direction-opposite pairs were tested only on the constructed Gym-New subset, a full-label-space analysis of forward/backward and other direction-sensitive classes would reveal whether the augmentation's semantic-preservation guarantee holds across all fine-grained categories.
- Beyond the paper, the MLLM result suggests that semi-supervised FAR training could serve as a low-cost domain adapter for video question answering in specialized fields, replacing large labeled video corpora with a small labeled subset and unlabeled video.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SeFAR, a semi-supervised framework for fine-grained action recognition (FAR) built on the FixMatch/Mean-Teacher paradigm. The method (i) samples dual-level temporal elements, consisting of multiple short fine-grained elements plus a longer context element; (ii) uses as strong augmentation a 'moderate temporal perturbation' that reverses each fine-grained element while keeping the context element in its original temporal order; and (iii) reweights the unsupervised losses with 'Adaptive Regulation' coefficients derived from the teacher's repeated predictions. The authors evaluate on FineGym (Gym99/Gym288 at element level, plus sub-event and within-set subsets), FineDiving, UCF-101, and HMDB-51, report state-of-the-art results against existing semi-supervised action recognition baselines, ablate each component and each temporal perturbation variant, and show that features from a 5%-trained SeFAR can replace frozen visual encoders in several MLLMs on a constructed Gym-QA benchmark. The paper includes a detailed appendix with data processing, baseline analysis, and research-question discussions, and the authors state that code and preprocessing scripts will be released.
Significance. If the empirical claims hold, SeFAR is a useful contribution: it is one of the first dedicated studies of semi-supervised fine-grained action recognition, it develops a plausible temporal strong-augmentation for FAR (reversing fine-grained elements while preserving context order), and it demonstrates a practical downstream use by improving MLLM fine-grained video question answering. The paper's strengths are its extensive ablations (component-level and augmentation-level, including sub-event and within-set FineGym splits), the construction of the Gym-New directionality stress test, the Gym-QA benchmark for MLLM evaluation, and the promise of released code and preprocessing scripts. The central limitation is that the headline 'state-of-the-art' claim rests on a narrow, partially self-run baseline set and lacks variance estimates; the direct concurrent method FinePseudo is identified in the appendix but not compared. Those gaps are fixable but need to be addressed before the claim is fully supported.
major comments (5)
- [Table 1 and Appendix A] The state-of-the-art claim for fine-grained datasets is not supported by the current comparison set. Table 1 compares SeFAR only with MemDPC, LTG, and SVFormer, all originally designed for coarse-grained action recognition, and the appendix's 'Baseline Analysis' states that these were evaluated by the authors themselves and that most other works were not reproducible. The same appendix identifies a directly competing published method, FinePseudo (Dave, Rizve, and Shah, ECCV 2025), which addresses exactly semi-supervised fine-grained action recognition, and defers the comparison to future work. A head-to-head comparison with FinePseudo under the same data splits, backbone, and label rates is needed before the phrase 'state-of-the-art' can be used; without it, the observed gains could reflect a strong but incomplete baseline set.
- [Tables 1 and 2] No error bars, confidence intervals, or multiple-seed results are reported anywhere in the main tables. At label rates of 5% and 10% in semi-supervised video recognition, the choice of labeled subset and initialization can change accuracy by several points, and the reported margins over baselines are as large as roughly 7-10 points. The authors should report the mean and standard deviation over at least three random labeled-subset/seed repetitions, or otherwise provide the experimental protocol that makes single-run numbers directly comparable, so that the claimed improvements can be interpreted as statistically meaningful rather than as split noise.
- [Methodology, Preliminaries, Eq. (1)] The teacher update rule is written as θt ← −ωθs + (1−ω)θt, with a negative coefficient on the student weights. This is inconsistent with the text, which says the teacher is an exponential moving average of consecutive student models; the standard EMA update is θt ← ωθs + (1−ω)θt. As printed, the equation would make the teacher weights diverge from an average and cannot be what the experiments implement. Please correct the sign and confirm the exact update used in the code.
- [Methodology, Adaptive Regulation, Eq. (7)] The Adaptive Regulation coefficients are not fully specified and are not reproducible from the paper. The constants β and ε in Eq. (7) are described qualitatively ('related to the model dropout', 'a steady parameter') but no values or ranges are given. Moreover, because τ2 = sigmoid(1/(βσ*+ε)) − 0.5 can be negative for small 1/(βσ*+ε), the product η = τ1·τ2 can be negative, which would flip the sign of the unsupervised loss; the paper does not discuss whether such negative weighting is intended or clipped. Please specify β, ε, the exact computation of σ*, and the handling of negative η, or provide code-level details.
- [Appendix C, RQ6] The limitations stated in RQ6 are reasonable, but they should be acknowledged in the main text as well: the method explores only temporal augmentation and does not combine it with modern spatial strong augmentations, and it relies solely on RGB input. These scope restrictions are not fatal, but the RGB-only point matters for the MLLM experiment in Table 5, which is a single visual-encoder replacement study rather than a general demonstration that SeFAR features suffice for multimodal fine-grained understanding.
minor comments (4)
- [Eq. (3)] The pseudo-label notation is informal: y-hat_j is written as the maximum predicted probability, while the cross-entropy H(y-hat_j, ...) requires a class label (typically a one-hot arg-max). Please write y-hat_j = argmax F_t(A_weak(x_j)) and keep the confidence threshold separate, e.g., 1(max F_t(...) > τ).
- [Abstract and Contributions] The claim that this is 'the first work' on semi-supervised fine-grained action recognition is qualified only by the concurrent FinePseudo work in the appendix. Please soften the claim in the abstract/introduction to 'one of the first' or 'the first open-source framework' to avoid the contradiction.
- [Table 4] The 'Sth.-Sth.' column header is not explained in the main text; it refers to Something-Something V2, but the reader must infer the SSL protocol, label rates, and evaluation split used there. Please add a sentence or footnote describing the Something-Something V2 setup.
- [Implementation Details] Several important hyperparameters are given in prose only or not at all: γ1 and γ2 in Eq. (4) are said to be set to 2, but the warmup coefficient ξ uses M_n without a stated value, and the teacher prediction count U is given as 10 only later in the text. Collecting all such values in the implementation-details paragraph would improve reproducibility.
Circularity Check
No circularity: SeFAR's central claims are empirical benchmark results, and its design choices are validated by ablations rather than derived from the claims themselves.
full rationale
The paper's central assertions are experimental: SeFAR attains state-of-the-art accuracy on FineGym, FineDiving, UCF101, and HMDB51 under semi-supervised settings. These are benchmark outcomes, not quantities derived from equations whose outputs are fed back as inputs. The proposed components—dual-level temporal elements, moderate temporal perturbation, and adaptive regulation—are defined autonomously: Eq. 5 defines the sampling of fine-grained and context elements, Eq. 6 defines the reversal of fine-grained elements while preserving context order, and Eq. 7 defines adaptive coefficients from teacher prediction confidence and variance. None of these definitions presupposes the final accuracies; their effectiveness is then tested through ablations in Tables 3, 4, and 6, and through constructed subsets such as Gym-New. The appendix explicitly identifies a directly competing method, FinePseudo, and defers comparison to future work; this is a completeness and correctness concern about the 'state-of-the-art' claim, not circular reasoning. The paper's many self-citations appear in a peripheral list of 'advancement of deep learning' works and do not carry a load-bearing argument. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is merely relabeled. Therefore, no circular step can be exhibited from the text, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Sampling combination {2-2-4} =
{2-2-4} (default)
- Teacher prediction count U =
10
- Adaptive Regulation constants beta and epsilon =
not reported
- Warmup coefficient xi =
xi = sin(n/Mn)
assumptions (5)
- standard math Cross-entropy and EMA teacher-student updates yield useful pseudo-labels under FixMatch-style consistency regularization.
- domain assumption Eight RGB frames sampled into short segments capture enough visual detail for fine-grained action discrimination.
- domain assumption Reversing fine-grained temporal elements while retaining context order preserves action category semantics.
- domain assumption Teacher prediction variance across U stochastic forward passes is a valid uncertainty signal for weighting pseudo-labels.
- domain assumption FineGym and FineDiving element-level annotations are reliable.
Cite this review
Pith. "Pith review of SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization." pith.science (2026). https://pith.science/paper/Y6LY7SES
@misc{pith2026250101245,
author = {Pith},
title = {Pith review of: SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y6LY7SES}},
note = {Machine review of arXiv:2501.01245}
}
read the original abstract
Human action understanding is crucial for the advancement of multimodal systems. While recent developments, driven by powerful large language models (LLMs), aim to be general enough to cover a wide range of categories, they often overlook the need for more specific capabilities. In this work, we address the more challenging task of Fine-grained Action Recognition (FAR), which focuses on detailed semantic labels within shorter temporal duration (e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating fine-grained labels and the substantial data needed for fine-tuning LLMs, we propose to adopt semi-supervised learning (SSL). Our framework, SeFAR, incorporates several innovative designs to tackle these challenges. Specifically, to capture sufficient visual details, we construct Dual-level temporal elements as more effective representations, based on which we design a new strong augmentation strategy for the Teacher-Student learning paradigm through involving moderate temporal perturbation. Furthermore, to handle the high uncertainty within the teacher model's predictions for FAR, we propose the Adaptive Regulation to stabilize the learning process. Experiments show that SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and FineDiving, across various data scopes. It also outperforms other semi-supervised methods on two classical coarse-grained datasets, UCF101 and HMDB51. Further analysis and ablation studies validate the effectiveness of our designs. Additionally, we show that the features extracted by our SeFAR could largely promote the ability of multimodal foundation models to understand fine-grained and domain-specific semantics.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance
FinePhys adds a learned PhysNet that re-estimates 3D skeletons through an Euler-Lagrange parameterization and uses the fused skeletons as 2D heatmap guidance for a latent diffusion model, reporting improved generation...
Reference graph
Works this paper leans on
-
[2]
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6. Cubuk, E. D.; Zoph, B.; Shlens, J.; and Le, Q. V . 2020. Ran- daugment: Practical automated data augmentation with a re- duced search space. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern ...
work page 2023
-
[3]
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2341–2352
Timebalance: Temporally-invariant and temporally- distinctive video representations for semi-supervised action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2341–2352. Dave, I. R.; Rizve, M. N.; and Shah, M. 2025. Finepseudo: improving pseudo-labelling through temporal-alignablity for semi-supervised fi...
arXiv 2025
-
[4]
arXiv preprint arXiv:2303.03378
Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. Gao, P.; Zhang, R.; Liu, C.; Qiu, L.; Huang, S.; Lin, W.; Zhao, S.; Geng, S.; Lin, Z.; Jin, P.; et al. 2024. Sphinx- x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935. Gowda, S. N.; Rohrbach, M.; Keller, F.; and S...
arXiv 2024
-
[7]
arXiv preprint arXiv:2208.01897
Combined CNN transformer encoder for enhanced fine-grained human action recognition. arXiv preprint arXiv:2208.01897. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. In International conference on machine learning, 19730–19742. PMLR. Li, K.; He, Y .;...
arXiv 2024
-
[10]
In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 9568– 9578
Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 9568– 9578. Vemprala, S. H.; Bonatti, R.; Bucker, A.; and Kapoor, A
-
[11]
Chatgpt for robotics: Design principles and model abilities. IEEE Access. Wang, J.; Wang, Y .; Liu, S.; and Li, A. 2021. Few-shot fine- grained action recognition via bidirectional attention and contrastive meta-learning. In Proceedings of the 29th ACM International Conference on Multimedia, 582–591. Wang, L.; Xiong, Y .; Wang, Z.; Qiao, Y .; Lin, D.; Tan...
work page 2021
-
[12]
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26077–26087
FSC: Few-point Shape Completion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26077–26087. Xiao, J.; Jing, L.; Zhang, L.; He, J.; She, Q.; Zhou, Z.; Yuille, A.; and Li, Y . 2022. Learning from temporal gradi- ent for semi-supervised action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and...
work page 2022
-
[13]
In Proceedings of the IEEE/CVF interna- tional conference on computer vision, 7209–7219
Multiview pseudo-labeling for semi-supervised learn- ing from video. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, 7209–7219. Xu, J.; Rao, Y .; Yu, X.; Chen, G.; Zhou, J.; and Lu, J. 2022a. Finediving: A fine-grained dataset for procedure-aware ac- tion quality assessment. In Proceedings of the IEEE/CVF conference on comput...
Show all 16 references
-
[14]
arXiv preprint arXiv:2307.03166
Videoglue: Video general understanding evaluation of foundation models. arXiv preprint arXiv:2307.03166. Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y . 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of...
2019 arXiv
-
[15]
run- ning
VideoGen-of-Thought: A Collaborative Frame- work for Multi-Shot Video Generation. arXiv preprint arXiv:2412.02259. Zhou, B.; Andonian, A.; Oliva, A.; and Torralba, A. 2018. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (E...
2018
-
[16]
forward” vs. “backward
(Left), and our proposed SeFAR (Right) applied to FineGym-New dataset. Our method capitalizes ondual-level temporal elements modeling , which yields diverse tempo- ral features, and moderate temporal perturbation, which en- hances the model’s focus on temporal feature modeling...
-
[329]
Hong, J.; Fisher, M.; Gharbi, M.; and Fatahalian, K
Springer. Hong, J.; Fisher, M.; Gharbi, M.; and Fatahalian, K. 2021. Video pose distillation for few-shot, fine-grained sports ac- tion recognition. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 9254–9263. Huang, H.; Qiao, X.; Chen, Z.; Chen, H....
2021
-
[2021]
arXiv preprint arXiv:2101.06329
In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2...
2015 arXiv
-
[2022]
In European conference on computer vision, 242–259
Learn2augment: learning to composite videos for data augmentation in action recognition. In European conference on computer vision, 242–259. Springer. Goyal, R.; Ebrahimi Kahou, S.; Michalski, V .; Materzyn- ska, J.; Westphal, S.; Kim, H.; Haenel, V .; Fruend, I.; Yian- ilos, ...
2017
-
[2023]
arXiv preprint arXiv:2310.09478
Minigpt-v2: large language model as a unified inter- face for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y .; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y .; Gonzalez, J. E.; et al
-
[2024]
arXiv preprint arXiv:2404.01121
CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening. arXiv preprint arXiv:2404.01121. Singh, A.; Chakraborty, O.; Varshney, A.; Panda, R.; Feris, R.; Saenko, K.; and Das, A. 2021. Semi-supervised action recognition with temporal contrastive learning. In Procee...
2021 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.