{"id":"ceeab4bc-1a31-4e87-ba74-2b507ae3f349","arxiv_id":"2501.01245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SeFAR combines dual-level temporal sampling, local temporal reversal, and uncertainty-based loss weighting to set new state-of-the-art results in semi-supervised fine-grained action recognition on FineGym and FineDiving.","lead":"This paper introduces SeFAR, a method for recognizing very similar sports actions (like gymnastics moves) when only a small fraction of videos have labels. It combines dual-scale video sampling, a temporal reversal trick for training, and adaptive weighting of uncertain pseudo-labels, reporting top results on FineGym, FineDiving, UCF101, and HMDB51.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not yet supported because the dedicated concurrent FAR method FinePseudo is not compared and no variance estimates are reported; a direct comparison could overturn the headline.","rationale":"The reader's weakest assumption concerns whether reversing the fine-grained temporal elements preserves action semantics. This is a plausible mechanism-level risk, and the paper's Gym-New construction is a reasonable but partial check: Table 4 shows Mod-Perturb gains only 0.6 points over spatial-only augmentation on Gym-New, weaker than on the full datasets, so direction-sensitive classes are not obviously harmed but also not strongly helped. I therefore regard that as a secondary concern rather than the most load-bearing one. The more decisive issue is empirical: the core claim is a state-of-the-art claim, and the paper's own appendix admits the dedicated concurrent competitor FinePseudo was not compared and that only three baselines were reproducible. Under low label rates, the absence of seed variance further weakens the comparison. These two gaps can be settled by one head-to-head experiment with multiple seeds. If SeFAR still wins with a margin exceeding run-to-run noise, the central claim stands; otherwise it should be softened. I retain the CONDITIONAL verdict because the method's design is coherent and the ablations are informative, but the empirical SOTA assertion requires that additional support.","tokens_in":20149,"tokens_out":9534,"duration_ms":97719,"concrete_test":"Run FinePseudo's released code or official protocol on the exact FineGym Gym99/Gym288 and FineDiving splits with the same 8-frame ViT input, same 5% and 10% labeling rates, and same top-1 metric; repeat SeFAR and FinePseudo over at least three seeds and compare mean and standard deviation. If FinePseudo's mean is within or above SeFAR's mean on any reported scope, the state-of-the-art claim in the abstract is not supported; if SeFAR remains ahead by more than the run-to-run variability, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that SeFAR achieves state-of-the-art performance on FineGym and FineDiving. The least secure support for that claim is the FAR baseline table itself. Table 1 compares only three methods (MemDPC, LTG, and SVFormer), and the appendix's Baseline Analysis states that these were evaluated by the authors for the first time and that many other works were not reproducible. The appendix also identifies a directly competing published method, FinePseudo (Dave, Rizve, and Shah, ECCV 2025), which tackles exactly semi-supervised fine-grained action recognition, and defers the comparison to future work. Without a head-to-head result against FinePseudo, the phrase 'state-of-the-art' is unsupported. In addition, no error bars or multiple-seed statistics are reported anywhere; at 5-10% labeling rates, semi-supervised accuracy can vary by several points between seeds, so the observed margins of up to roughly 7-10 points need a variance estimate before being interpreted as genuine improvements. If a direct comparison shows FinePseudo is comparable or better, or the margins collapse across seeds, the headline claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SeFAR, a semi-supervised framework for fine-grained action recognition (FAR) built on the FixMatch/Mean-Teacher paradigm. The method (i) samples dual-level temporal elements, consisting of multiple short fine-grained elements plus a longer context element; (ii) uses as strong augmentation a 'moderate temporal perturbation' that reverses each fine-grained element while keeping the context element in its original temporal order; and (iii) reweights the unsupervised losses with 'Adaptive Regulation' coefficients derived from the teacher's repeated predictions. The authors evaluate on FineGym (Gym99/Gym288 at element level, plus sub-event and within-set subsets), FineDiving, UCF-101, and HMDB-51, report state-of-the-art results against existing semi-supervised action recognition baselines, ablate each component and each temporal perturbation variant, and show that features from a 5%-trained SeFAR can replace frozen visual encoders in several MLLMs on a constructed Gym-QA benchmark. The paper includes a detailed appendix with data processing, baseline analysis, and research-question discussions, and the authors state that code and preprocessing scripts will be released.","tokens_in":20374,"tokens_out":7391,"duration_ms":66983,"significance":"If the empirical claims hold, SeFAR is a useful contribution: it is one of the first dedicated studies of semi-supervised fine-grained action recognition, it develops a plausible temporal strong-augmentation for FAR (reversing fine-grained elements while preserving context order), and it demonstrates a practical downstream use by improving MLLM fine-grained video question answering. The paper's strengths are its extensive ablations (component-level and augmentation-level, including sub-event and within-set FineGym splits), the construction of the Gym-New directionality stress test, the Gym-QA benchmark for MLLM evaluation, and the promise of released code and preprocessing scripts. The central limitation is that the headline 'state-of-the-art' claim rests on a narrow, partially self-run baseline set and lacks variance estimates; the direct concurrent method FinePseudo is identified in the appendix but not compared. Those gaps are fixable but need to be addressed before the claim is fully supported.","major_comments":[{"comment":"The state-of-the-art claim for fine-grained datasets is not supported by the current comparison set. Table 1 compares SeFAR only with MemDPC, LTG, and SVFormer, all originally designed for coarse-grained action recognition, and the appendix's 'Baseline Analysis' states that these were evaluated by the authors themselves and that most other works were not reproducible. The same appendix identifies a directly competing published method, FinePseudo (Dave, Rizve, and Shah, ECCV 2025), which addresses exactly semi-supervised fine-grained action recognition, and defers the comparison to future work. A head-to-head comparison with FinePseudo under the same data splits, backbone, and label rates is needed before the phrase 'state-of-the-art' can be used; without it, the observed gains could reflect a strong but incomplete baseline set.","section":"Table 1 and Appendix A"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported anywhere in the main tables. At label rates of 5% and 10% in semi-supervised video recognition, the choice of labeled subset and initialization can change accuracy by several points, and the reported margins over baselines are as large as roughly 7-10 points. The authors should report the mean and standard deviation over at least three random labeled-subset/seed repetitions, or otherwise provide the experimental protocol that makes single-run numbers directly comparable, so that the claimed improvements can be interpreted as statistically meaningful rather than as split noise.","section":"Tables 1 and 2"},{"comment":"The teacher update rule is written as θt ← −ωθs + (1−ω)θt, with a negative coefficient on the student weights. This is inconsistent with the text, which says the teacher is an exponential moving average of consecutive student models; the standard EMA update is θt ← ωθs + (1−ω)θt. As printed, the equation would make the teacher weights diverge from an average and cannot be what the experiments implement. Please correct the sign and confirm the exact update used in the code.","section":"Methodology, Preliminaries, Eq. (1)"},{"comment":"The Adaptive Regulation coefficients are not fully specified and are not reproducible from the paper. The constants β and ε in Eq. (7) are described qualitatively ('related to the model dropout', 'a steady parameter') but no values or ranges are given. Moreover, because τ2 = sigmoid(1/(βσ*+ε)) − 0.5 can be negative for small 1/(βσ*+ε), the product η = τ1·τ2 can be negative, which would flip the sign of the unsupervised loss; the paper does not discuss whether such negative weighting is intended or clipped. Please specify β, ε, the exact computation of σ*, and the handling of negative η, or provide code-level details.","section":"Methodology, Adaptive Regulation, Eq. (7)"},{"comment":"The limitations stated in RQ6 are reasonable, but they should be acknowledged in the main text as well: the method explores only temporal augmentation and does not combine it with modern spatial strong augmentations, and it relies solely on RGB input. These scope restrictions are not fatal, but the RGB-only point matters for the MLLM experiment in Table 5, which is a single visual-encoder replacement study rather than a general demonstration that SeFAR features suffice for multimodal fine-grained understanding.","section":"Appendix C, RQ6"}],"minor_comments":[{"comment":"The pseudo-label notation is informal: y-hat_j is written as the maximum predicted probability, while the cross-entropy H(y-hat_j, ...) requires a class label (typically a one-hot arg-max). Please write y-hat_j = argmax F_t(A_weak(x_j)) and keep the confidence threshold separate, e.g., 1(max F_t(...) > τ).","section":"Eq. (3)"},{"comment":"The claim that this is 'the first work' on semi-supervised fine-grained action recognition is qualified only by the concurrent FinePseudo work in the appendix. Please soften the claim in the abstract/introduction to 'one of the first' or 'the first open-source framework' to avoid the contradiction.","section":"Abstract and Contributions"},{"comment":"The 'Sth.-Sth.' column header is not explained in the main text; it refers to Something-Something V2, but the reader must infer the SSL protocol, label rates, and evaluation split used there. Please add a sentence or footnote describing the Something-Something V2 setup.","section":"Table 4"},{"comment":"Several important hyperparameters are given in prose only or not at all: γ1 and γ2 in Eq. (4) are said to be set to 2, but the warmup coefficient ξ uses M_n without a stated value, and the teacher prediction count U is given as 10 only later in the text. Collecting all such values in the implementation-details paragraph would improve reproducibility.","section":"Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious empirical contribution with extensive ablations and a clear application to MLLM fine-grained understanding. My main concern is the mismatch between the strong 'state-of-the-art' wording and the thin FAR baseline set, especially the absence of FinePseudo, which the authors themselves identify as the directly competing method. The additional absence of variance estimates makes the magnitude of the claimed gains hard to assess. If the authors add the direct comparison and seed statistics, or carefully downgrade the claim to one of strong performance against existing open-source baselines, the paper would be close to acceptable. The sign error in Eq. (1) and the unspecified β and ε must also be corrected for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: SeFAR is a competent, clearly written FixMatch-style framework for semi-supervised fine-grained action recognition, and as far as I can tell it is the first detailed treatment of that specific problem setting. The core ideas are sound: dual-level temporal elements (short fine-grained clips plus a longer context element), local frame reversal as strong augmentation while keeping the context in original order, and an adaptive regulation term that down-weights teacher predictions with high variance. The paper deserves credit for testing the directionality concern head-on by constructing Gym-New, a subset of direction-opposite action pairs, and for reporting consistent gains on Something-Something V2. The ablations are extensive and each component shows a clear contribution. If the empirical claims hold, this is a useful subfield-level advance.\n\nWhere it gets soft is exactly where the stress-test lands. The fine-grained baseline table contains only MemDPC, LTG, and SVFormer, all re-implemented by the authors because other methods are not reproducible; the appendix says this plainly. That is honest but it means a state-of-the-art claim rests on a thin foundation. No error bars or multi-seed runs appear anywhere, and at 5-10% label rates semi-supervised accuracy can move several points between seeds. The margins of up to 10 points need variance estimates before I'd call them real. The appendix also identifies FinePseudo, a directly competing ECCV 2025 method for the same task, and defers comparison to future work. That's candid, but it leaves the headline claim unverified.\n\nSmaller issues: beta and epsilon in the adaptive regulation equations are never given values, which is a reproducibility gap for a paper with a code link. The Gym-New subset and some design choices were selected after the authors had seen results on the same benchmarks, so there is mild selection bias. Neither is fatal.\n\nOn balance, the math is coherent, the empirical work is solid if narrow, and the writing is honest about its own limitations. I'd send this to peer review and ask for a FinePseudo comparison, error bars, and reported constants before publication. I'd also bring it to a reading group as a good case study of adapting SSL to a temporally sensitive domain.\n\nBest.","headline":"Solid FixMatch-style SSL framework for fine-grained action recognition with honest ablations, but the SOTA claim rests on only three re-implemented baselines and no variance estimates.","tokens_in":20898,"tokens_out":3968,"would_cite":true,"duration_ms":39563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SeFAR shows that reversing short fine-grained temporal chunks, while preserving context, yields state-of-the-art semi-supervised fine-grained action recognition.","keywords":["semi-supervised learning","fine-grained action recognition","temporal augmentation","teacher-student learning","pseudo-labeling","temporal perturbation","action recognition","video understanding"],"falsifier":"Build a benchmark whose classes differ only by the motion direction inside a single short clip, with the surrounding context identical across classes (for example, \"hand turns left\" vs. \"hand turns right\" with the same background and body pose). If SeFAR's accuracy on that benchmark is no better than, or worse than, a spatial-only augmentation baseline, the semantic-preservation assumption behind chunk reversal is falsified. A simpler check: on any fine-grained dataset, compare the teacher's pseudo-label accuracy on weakly augmented videos with the true-label accuracy of the same teacher on strongly augmented videos; a large gap indicates the strong augmentation has broken the label.","tokens_in":1825,"feed_emoji":"🎬","tokens_out":2614,"duration_ms":101386,"temperature":0.7,"pith_summary":"This paper tackles semi-supervised fine-grained action recognition: recognizing actions such as \"pike sole circle backward with 0.5 turn to handstand\" when only a small fraction of videos have labels. It proposes SeFAR, a teacher-student method that represents each video as a set of short fine-grained temporal chunks plus one longer context chunk, reverses only the short chunks to create a strong augmentation, and uses an adaptive weighting scheme to keep training stable when teacher predictions are uncertain. The paper claims this design beats previous semi-supervised action recognition methods on two fine-grained benchmarks, FineGym and FineDiving, at several labeling ratios and semantic granularities, and also improves results on coarse-grained UCF101 and HMDB51. If true, it would make fine-grained video understanding feasible with very little annotation, and could supply better visual features to multimodal models in specialized domains.","feed_headline":"Reversed chunks lift few-label fine-grained action recognition","feed_subtitle":"Short reversed clips plus preserved context outdo prior semi-supervised action methods on four benchmarks.","key_machinery":"The load-bearing objects are the dual-level temporal elements: from $K$ sampled frames, SeFAR builds $M$ fine-grained elements $\\{p_i\\}_{i=1}^M$, each of length $L$, together with a context element $p_{\\mathrm{context}}$ of length $2L$. The strong augmentation is $A_{\\mathrm{strong}}(\\{p_i\\}) = \\{\\overleftarrow{p_i}\\}$ while $p_{\\mathrm{context}}$ stays in its original order, forming the weak-to-strong pair for consistency regularization. The stabilizing mechanism is the Adaptive Regulation: the teacher predicts each unlabeled video $U$ times, the maximum confidence $\\mu^*$ and its standard deviation $\\sigma^*$ are extracted, and an adaptive coefficient $\\eta = \\tau_1 \\tau_2$ is computed from $\\tau_1 = \\mathrm{sigmoid}(e^{\\mu^*}-e)$ and $\\tau_2 = \\mathrm{sigmoid}(1/(\\beta\\sigma^*+\\epsilon))-0.5$ to reweight the unsupervised loss. These pieces carry the claim by converting the generic FixMatch recipe into a training loop tailored to fine-grained temporal semantics.","core_discovery":"The central claim is that consistency regularization for fine-grained actions should be built from temporal, not spatial, structure: a video is decomposed into dual-level temporal elements, and the strong augmentation reverses the short fine-grained elements while preserving the order of the context element, producing a \"moderate temporal perturbation\" that is strong enough to be a useful training signal yet weak enough to keep the action's semantics and temporal directionality intact. On top of this, SeFAR stabilizes learning by having the teacher predict each unlabeled video several times, computing the mean confidence and standard deviation of those predictions, and deriving an adaptive coefficient that up-weights confident predictions and down-weights unstable ones. The paper reports that this combination achieves state-of-the-art accuracy on FineGym and FineDiving across data scopes and label rates, outperforms prior semi-supervised methods on UCF101 and HMDB51, and yields features that improve fine-grained question answering when used as a visual encoder in an MLLM.","pith_inferences":["Beyond the paper, the same dual-level reversal recipe could be tested on other temporally sensitive semi-supervised tasks, such as surgical phase recognition or gesture segmentation, where short local motions matter but global order carries meaning.","Beyond the paper, the hand-designed sigmoid coefficients in Adaptive Regulation could be replaced by a calibrated uncertainty estimate over pseudo-labels, which would make the weighting principle dataset-agnostic rather than tuned to these benchmarks.","Beyond the paper, because direction-opposite pairs were tested only on the constructed Gym-New subset, a full-label-space analysis of forward/backward and other direction-sensitive classes would reveal whether the augmentation's semantic-preservation guarantee holds across all fine-grained categories.","Beyond the paper, the MLLM result suggests that semi-supervised FAR training could serve as a low-cost domain adapter for video question answering in specialized fields, replacing large labeled video corpora with a small labeled subset and unlabeled video."],"forward_implications":["With only 5% to 10% of labels, SeFAR outperforms prior semi-supervised action recognition methods on FineGym and FineDiving at event, set, and element granularities.","The same recipe transfers to coarse-grained datasets, beating previous SSL action recognition methods on UCF101 and HMDB51 under the same label scopes.","A SeFAR encoder trained on 5% FineGym data improves fine-grained Gym-QA accuracy over standard CLIP-style visual encoders in an MLLM, suggesting that semi-supervised FAR pretraining can substitute for large labeled video corpora in specialized domains.","Reversing only fine-grained elements outperforms spatial-only augmentation, temporal warping, full shuffle, full reverse, and speed-based perturbations, especially on direction-sensitive action pairs.","The adaptive weighting helps most at low labeling rates: on FineDiving the gain from Adaptive Regulation grows as the label rate drops from 10% to 1%."],"supporting_citations":[{"why":"Supplies the FixMatch teacher-student weak-to-strong consistency paradigm that SeFAR extends.","marker":"[Sohn et al. 2020]"},{"why":"Provides the exponential moving average teacher update used to stabilize pseudo-labels.","marker":"[Tarvainen and Valpola 2017]"},{"why":"The main baseline; contributes the EMA-teacher design for action recognition, the unlabeled mixup strategy, and the temporal-warping augmentation SeFAR compares against.","marker":"[Xing et al. 2023]"},{"why":"Provides the FineGym dataset and its event-set-element hierarchy on which the main fine-grained results are reported.","marker":"[Shao et al. 2020a]"},{"why":"Provides the FineDiving dataset used as the second fine-grained benchmark.","marker":"[Xu et al. 2022a]"},{"why":"Motivates the uncertainty-aware weighting behind the Adaptive Regulation by linking prediction uncertainty to pseudo-label reliability.","marker":"[Rizve et al. 2021]"}],"fun_headline_variants":["Temporal reversal boosts semi-supervised fine-grained action recognition","Short clip reversal teaches fine-grained actions with fewer labels","Adaptive teacher regulation stabilizes semi-supervised fine action learning","Flip short clips, stabilize teacher: semi-supervised fine actions","Time-perturbed elements advance fine-grained action SSL on few labels"],"cache_read_input_tokens":23040,"weakest_assumption_plain":"The load-bearing premise is that reversing each short fine-grained chunk while leaving the context chunk in its original order is a strong augmentation that perturbs the student's input enough to be useful without changing what the correct label is; if local motion direction is what distinguishes a class, and the preserved context cannot disambiguate it, the consistency loss would teach the student to ignore the defining cue.","fun_headline_variants_meta":{"raw":{"variants":["Temporal reversal boosts semi-supervised fine-grained action recognition","Short clip reversal teaches fine-grained actions with fewer labels","Adaptive teacher regulation stabilizes semi-supervised fine action learning","Flip short clips, stabilize teacher: semi-supervised fine actions","Time-perturbed elements advance fine-grained action SSL on few labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3475,"prompt_tokens":1020,"completion_tokens":2455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2381}},"tokens_in":636,"tokens_out":2455,"duration_ms":17229,"temperature":1.0,"reasoning_tokens":2381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:31:44.066642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a benchmark whose classes differ only by the motion direction inside a single short clip, with the surrounding context identical across classes (for example, \"hand turns left\" vs. \"hand turns right\" with the same background and body pose). If SeFAR's accuracy on that benchmark is no better than, or worse than, a spatial-only augmentation baseline, the semantic-preservation assumption behind chunk reversal is falsified. A simpler check: on any fine-grained dataset, compare the teacher's pseudo-label accuracy on weakly augmented videos with the true-label accuracy of the same teacher on strongly augmented videos; a large gap indicates the strong augmentation has broken the label.","supporting_citations":[],"review_version":1}