Pith. sign in

REVIEW 3 major objections 8 minor 300 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Align visual modalities first, add text second: staged fusion lifts action quality scoring by 21%

2026-07-09 10:51 UTC pith:JBYEAJIZ

load-bearing objection Two things: the two-stage alignment idea is reasonable and the ablations back it up, but the headline 21% gain on the authors' own dataset is confounded by clinical text that likely leaks severity information. the 3 major comments →

arxiv 2607.07438 v1 pith:JBYEAJIZ submitted 2026-07-08 cs.CV

Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

Kanglei Zhou , Ruizhi Cai , Xinning Wang , Yijian Zheng , Liyuan Wang , Jianguo Li , Xiaohui Liang This is my paper
classification cs.CV
keywords action quality assessmentmulti-modal fusioncross-modal alignmentGramian volume lossstaged alignmentclinical movement assessmentpediatric dermatomyositismulti-modal dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-modal action quality assessment fails not because we lack complementary data sources, but because heterogeneous modalities are fused in the wrong order. RGB video, optical flow, skeleton, and text descriptions occupy fundamentally different feature spaces; forcing them together simultaneously destabilizes learning and can make multi-modal models worse than unimodal ones. DualAlign addresses this by splitting alignment into two stages: first, visual modalities (RGB, flow, skeleton) are aligned to each other using a Gramian volume loss that minimizes the geometric span between their embedding vectors; second, the stabilized visual representation is aligned with textual semantics using the same geometric principle. The paper introduces MM-JDM, a clinical dataset of pediatric movement assessment with four modalities and realistic noise, imbalance, and label scarcity. On this benchmark, DualAlign achieves 85.44 average SRCC versus 64.28 for the best prior method, with consistent gains on two external sports datasets. Ablations show that single-stage Gramian alignment performs worse than no alignment at all, and that reversing the fusion order (text before visual stabilization) degrades performance by 14% SRCC, supporting the claim that visual-to-visual alignment must precede visual-to-textual alignment.

Core claim

The central finding is that the order in which modalities are aligned matters more than the alignment mechanism itself. A Gramian volume loss applied simultaneously to all four modalities (single-stage) yields 60.55 SRCC, which is worse than removing alignment entirely (63.82). The same loss decomposed into two stages—visual-visual first, visual-textual second—yields 85.44. This gap demonstrates that the benefit comes from respecting the structural hierarchy between modality types: visual modalities share spatiotemporal structure and can be coherently merged, while text operates at a semantic level that, if introduced prematurely, distorts the visual manifold before it has stabilized. The CK

What carries the argument

DualAlign uses a Gramian volume alignment loss (adapted from GRAM) applied in two stages. In stage one, RGB, optical flow, and skeleton embeddings are aligned by minimizing the determinant of their Gram matrix, with RGB as the anchor modality. In stage two, the stabilized visual representation is aligned with a CLIP-derived textual embedding using the same volume-minimization principle. Prediction uses an Equiangular Tight Frame (ETF) prototype structure for discrete grading and a coarse-to-fine strategy for continuous regression.

Load-bearing premise

The 21.16% improvement margin on MM-JDM depends on baseline methods designed for different datasets and modality configurations being fairly adapted to MM-JDM's four-modality input and 12-action grading protocol. The paper states that baselines are adapted while preserving original model designs when direct transfer is not feasible, but does not specify the exact adaptations or report which modalities each baseline used on MM-JDM. If baselines were disadvantaged by suboptimal

What would settle it

If a properly tuned single-stage alignment method (using all four modalities simultaneously with well-chosen loss weights and modality-specific encoders) matched or exceeded DualAlign's performance on MM-JDM, the core claim that staged alignment is necessary would be undermined. Alternatively, if reversing the fusion order (text first, then visual) produced equivalent results on a different dataset with different modality abstraction hierarchies, the generality of the staging principle would be questioned.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If staged alignment generalizes, then any multi-modal system combining modalities of different abstraction levels (e.g., audio + video + text) should benefit from first aligning same-type modalities before cross-type fusion, rather than aligning everything jointly.
  • The finding that single-stage Gramian alignment is worse than no alignment suggests that existing multi-modal alignment methods that treat all modalities symmetrically may be actively harming performance in settings with heterogeneous modality types.
  • The clinical dataset MM-JDM, with its realistic noise and class imbalance, provides a testbed for whether methods developed on clean sports benchmarks transfer to medical assessment—results show most prior methods degrade substantially.
  • Zero-shot evaluation of GPT-4o and Gemini-3 Pro on the same task (SRCC near zero) indicates that general-purpose large multi-modal models cannot yet replace specialized AQA systems for fine-grained movement quality scoring.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The two-stage principle may extend beyond visual-then-textual: any system with a hierarchy of modality abstraction levels could benefit from a staged alignment protocol where structurally similar modalities are consolidated first, then progressively integrated with higher-level ones.
  • The CKA analysis showing moderate rather than saturated cross-modal similarity after alignment suggests an optimal alignment target exists between independence and collapse—over-alignment may suppress discriminative modality-specific information, which has implications for how alignment quality is evaluated in multi-modal learning generally.
  • The 21.16% margin on MM-JDM versus 3-6% on established sports benchmarks raises the question of whether staged alignment provides the most benefit in noisy, small-data, clinically realistic settings where cross-modal discrepancies are most severe—this could guide where to deploy such methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes DualAlign, a two-stage multi-modal alignment framework for Action Quality Assessment (AQA). In the first stage, visual modalities (RGB, optical flow, skeleton) are aligned using an adapted Gramian volume loss. In the second stage, the stabilized visual representation is aligned with textual semantics. The authors also introduce MM-JDM, a new multi-modal AQA dataset for Juvenile Dermatomyositis muscle strength assessment, comprising 1639 samples across 12 actions with four modalities. Experiments on MM-JDM show a 21.16% SRCC improvement over the best prior method, with smaller gains on RG (3.53%) and Fis-V (5.95%). Ablation studies support the two-stage design, showing that one-stage GRAM alignment performs worse than no alignment, and that reversed fusion order degrades performance.

Significance. The paper makes two contributions: (1) a principled two-stage alignment architecture that adapts the Gramian volume loss to a progressive visual-then-textual fusion pipeline, and (2) a clinically motivated multi-modal dataset (MM-JDM) with four modalities including structured text. The ablation studies (Tab. 6, Fig. 9) are thorough and internally consistent with the central design claim: the one-stage GRAM variant (60.55 SRCC) performs worse than no alignment (63.82), and the reversed fusion order degrades SRCC by 14%. The missing-modality and label-scarcity robustness experiments (Figs. 14, 15) add practical value. The framework is evaluated against both specialized AQA baselines and zero-shot mLLMs (Fig. 6), and code is stated to be publicly available.

major comments (3)
  1. §4.2, Textual Data; Tab. 6 (w/o Text row); Fig. 17: The text descriptions on MM-JDM include clinical physical examination notes containing JDM-specific signs (e.g., Fig. 17(b): 'Gottron sign is positive'; Fig. 17(c): 'purple red edematous rash on both eyelids,' 'V-neck sign (+)'). These cutaneous manifestations are established clinical indicators of JDM disease activity, and the grading targets (CMAS-based motor function scores) also reflect disease activity. The paper's defense — 'The finalized descriptions do not contain severity-related cues that are not observable from the video' (§4.2) — does not address the actual concern: textual clinical terminology may provide a shortcut that is far easier to learn than extracting the same information from raw pixels. The ablation (Tab. 6) shows text contributes ~11 SRCC points (85.44 to 74.50), roughly half the margin over the best baseline. On
  2. §4.2, Textual Data; Tab. 6 (w/o Text row); Fig. 17 (continued): RG and Fis-V, where the physical examination field is left empty (§5.1), gains are only 3.53% and 5.95%, consistent with the hypothesis that clinical text on MM-JDM contributes severity-correlated information. To rule out this confound, the authors should run a controlled ablation on MM-JDM where the physical examination field is replaced with a neutral placeholder (as done for RG/Fis-V), keeping only action and subject descriptions. If the SRCC drop is modest, the concern is alleviated; if it is large, approximately half of the headline 21.16% margin is attributable to text content rather than the alignment framework. This is load-bearing for the central claim because the 21.16% figure is the paper's headline result.
  3. §5.1, Implementation Details: The paper states that baselines were adapted when 'direct transfer to MM-JDM is not feasible' but does not specify the exact adaptations or report the modality configurations used by each baseline on MM-JDM. Tab. 3 lists both unimodal and multi-modal baselines, but it is unclear which modalities each baseline received. For example, MLA-VL was designed for audio-visual input; did it receive audio on MM-JDM? If baselines were evaluated under different modality configurations than DualAlign's four-modality input, the 21.16% margin may be inflated. The authors should report, for each baseline in Tab. 3, the exact modality configuration used and the specific adaptations applied.
minor comments (8)
  1. §3.2, Eq. (10): The summation index K in the denominator appears to range over modality choices for the anchor, but K was previously defined as the number of modality vectors. Clarify whether K=3 (video, flow, skeleton) or whether it indexes candidate anchors.
  2. §3.3, Eq. (15): The same K notation appears in a two-modality (visual-textual) setting. If K=2 here, state this explicitly to avoid confusion with the K=3 first-stage usage.
  3. Tab. 3: Several baselines show identical SRCC values across multiple actions (e.g., 51.46 for CoRe, GDLT, HGCN, T2CR on Action 02; 37.77 for multiple methods on Action 12). This may reflect ties in ranking, but it would help to note whether these are exact ties or rounding artifacts.
  4. Fig. 2: The architecture diagram is dense and some labels are difficult to read. Consider enlarging key components or splitting into sub-figures for clarity.
  5. §5.1: The number of uniformly sampled frames (103) is mentioned in the implementation details but not in the notation section (§3.1) where T is introduced. Cross-reference for clarity.
  6. Tab. 7 caption: The table title says 'Comparison results with different backbones' but the table also includes similarity distribution plots. Consider a more descriptive caption.
  7. §5.2: The text states DualAlign outperforms the previous best by '3.4%' on RG, but Tab. 4 shows MLA-VL at 0.849 and DualAlign at 0.878, which is a 3.4% relative improvement. This should be stated as 'relative' to avoid ambiguity.
  8. Fig. 16: The diversity index formula (Eq. 20) uses q both as the number of categories and in the normalization factor. Clarify that q is the number of non-empty grade categories for each action.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The three major comments raise important points about (1) potential label leakage through clinical text on MM-JDM, (2) the need for a controlled ablation isolating the physical examination field, and (3) transparency regarding baseline modality configurations and adaptations. We address each below.

read point-by-point responses

  1. Referee: §4.2, Textual Data; Tab. 6 (w/o Text row); Fig. 17: The text descriptions on MM-JDM include clinical physical examination notes containing JDM-specific signs (e.g., Gottron sign, heliotrope rash, V-neck sign). These cutaneous manifestations are established clinical indicators of JDM disease activity, and the grading targets (CMAS-based motor function scores) also reflect disease activity. The paper's defense — 'The finalized descriptions do not contain severity-related cues that are not observable from the video' — does not address the actual concern: textual clinical terminology may provide a shortcut that is far easier to learn than extracting the same information from raw pixels. The ablation (Tab. 6) shows text contributes ~11 SRCC points (85.44 to 74.50), roughly half the margin over the best baseline.

    Authors: The referee raises a valid and important concern. We agree that our original defense in §4.2 does not fully address the shortcut-learning risk. The distinction between 'observable from video' and 'easily learnable from text' is real: even if cutaneous signs like Gottron papules are technically visible in RGB frames, a model may learn to exploit the textual mention of these signs as a far cheaper proxy for disease severity, rather than learning to detect them visually. This is a genuine confound that our current experiments do not rule out. We acknowledge this limitation honestly. revision: partial

  2. Referee: §4.2, Textual Data; Tab. 6 (w/o Text row); Fig. 17 (continued): To rule out this confound, the authors should run a controlled ablation on MM-JDM where the physical examination field is replaced with a neutral placeholder (as done for RG/Fis-V), keeping only action and subject descriptions. If the SRCC drop is modest, the concern is alleviated; if it is large, approximately half of the headline 21.16% margin is attributable to text content rather than the alignment framework. This is load-bearing for the central claim because the 21.16% figure is the paper's headline result.

    Authors: We agree that this controlled ablation is necessary and will run it for the revision. Specifically, we will train DualAlign on MM-JDM with the physical examination field replaced by a neutral placeholder (matching the RG/Fis-V protocol), retaining only action and subject descriptions. We will report the resulting SRCC alongside the full-text and no-text conditions in an updated Tab. 6. We will also add an explicit discussion of this confound in §4.2 and qualify the headline 21.16% figure accordingly. If the drop is large, we will transparently report that a portion of the MM-JDM margin is attributable to clinical text content rather than the alignment framework alone, and we will emphasize the RG and Fis-V results (where text contains no clinical examination notes) as cleaner tests of the alignment contribution. We note that on RG and Fis-V, where the physical examination field is already empty, DualAlign still achieves gains of 3.53% and 5.95% over the prior state of the art, which supports the alignment framework's contribution independent of clinical text. However, we acknowledge that the magnitude of text contribution on MM-JDM cannot be determined without the requested experiment, and we commit to running it. revision: yes

  3. Referee: §5.1, Implementation Details: The paper states that baselines were adapted when 'direct transfer to MM-JDM is not feasible' but does not specify the exact adaptations or report the modality configurations used by each baseline on MM-JDM. Tab. 3 lists both unimodal and multi-modal baselines, but it is unclear which modalities each baseline received. For example, MLA-VL was designed for audio-visual input; did it receive audio on MM-JDM? If baselines were evaluated under different modality configurations than DualAlign's four-modality input, the 21.16% margin may be inflated.

    Authors: The referee is correct that this information is missing from the manuscript and is essential for interpreting the comparison. We will add a table in the revised §5.1 specifying, for each baseline in Tab. 3, the exact modality configuration used and the specific adaptations applied. To preview: unimodal baselines (CoRe, GDLT, HGCN, DAE, T2CR, CoFInAl, PHI) received RGB video only, consistent with their original designs. Multi-modal baselines received their originally designed modalities: MVLA received RGB + text, PAMFN received RGB + skeleton, RICA2 received RGB + skeleton, and MLA-VL received RGB + audio + text (audio extracted from the recording environment). MM-JDM does not include a dedicated audio modality, but ambient audio is present in the raw recordings; for MLA-VL, we used this ambient audio. All baselines used the same dataset splits, input preprocessing, and evaluation protocol. The adaptations were limited to input dimension matching and action-specific prediction heads (to accommodate MM-JDM's per-action grade ranges), with all other hyperparameters following the original implementations. We agree this transparency is important and will include it in the revision. revision: yes

Circularity Check

0 steps flagged

Minor self-citation for the prototype-based prediction mechanism; no circularity in the central two-stage alignment claim

full rationale

The paper's central methodological contribution — the two-stage alignment framework (DualAlign) with adapted Gramian volume loss — is derived from an externally cited method (GRAM, Cicchetti et al., 2024) and is validated against external benchmarks (RG, Fis-V) and internally consistent ablations (Tab. 6, Fig. 9). The GRAM loss formulation (Def. 1, Eqs. 4-9) is cited from external work and adapted, not defined in terms of the paper's own outputs. The prototype-based prediction mechanism (Eq. 1-2) is self-cited from Zhou et al. (2024a, CoFInAl), but this is a standard classification head that is not load-bearing for the alignment claim itself — it is an inference-time mechanism independent of the alignment losses. The ablation comparing two-stage vs. one-stage alignment (Fig. 9, Tab. 6) uses the authors' own one-stage variant as a baseline, which is a legitimate internal comparison rather than circular reasoning. The MM-JDM dataset is self-constructed, but results on external datasets (RG, Fis-V) provide independent validation. The text modality concern raised by the skeptic (clinical signs in text descriptions potentially confounding severity prediction) is a correctness risk about label leakage, not a circularity issue — the paper explicitly addresses this (Sec. 4.2) and the text field is left empty on external datasets. No step in the derivation chain reduces to its inputs by construction. The self-citation of CoFInAl for the prediction mechanism is minor and does not undermine the independence of the central alignment contribution. Score 2 reflects this minor self-citation that is not load-bearing for the main claim.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The free parameters (λ1, λ2, τ, D1, D2, frame count) are all set by hand without sensitivity analysis, which is a weakness but common in the field. The key domain assumptions (visual-first alignment, video as anchor, no label leakage in text) are empirically supported by ablation but not proven from first principles. The label-leakage assumption for GPT-4o-generated text is the most concerning: the text includes physical examination notes (e.g., 'Gottron sign positive,' 'rash visible') that could correlate with disease severity and thus with quality grades, despite the authors' claim. No invented entities are postulated without falsifiable handles — the dataset and method are both testable.

free parameters (6)
  • λ1 (visual alignment loss weight) = 1.0
    Set to 1.0 by hand (Sec. 5.1); stated as equal contribution since both losses are 'normalized and defined on consistent scales,' but no sensitivity analysis is provided.
  • λ2 (visual-textual alignment loss weight) = 1.0
    Set to 1.0 by hand (Sec. 5.1); same justification as λ1, no sensitivity analysis.
  • τ (temperature parameter) = 0.1
    Used in the GRAM contrastive loss (Eqs. 5-6, 10, 15); set to 0.1 without stated justification or sensitivity analysis.
  • D1 (visual feature dimension) = 1024
    Set by hand (Sec. 5.1); no justification for this specific value.
  • D2 (shared embedding dimension) = 512
    Set by hand (Sec. 5.1); no justification provided.
  • Number of uniformly sampled frames = 103
    Set by hand (Sec. 5.1); stated as consistent with prior AQA protocols but no ablation on this choice.
axioms (5)
  • domain assumption Visual modalities (RGB, flow, skeleton) share relatively homogeneous spatiotemporal structures and can be aligned before textual semantics are introduced.
    Invoked in Sec. 3.1 as the motivation for the two-stage design. Empirically supported by ablation (Fig. 9) but not proven from first principles.
  • domain assumption Video (RGB) is the natural anchor modality for visual alignment because it provides the most comprehensive spatiotemporal representation.
    Stated in Sec. 3.2: 'we designate the video data as the anchor modality, leveraging its inherent advantages as a comprehensive visual modality.' No comparison with flow or skeleton as anchor is reported.
  • domain assumption Textual descriptions generated by GPT-4o and reviewed by clinicians do not contain grade-related cues that could cause label leakage.
    Stated in Sec. 4.2: 'The finalized descriptions do not contain severity-related cues that are not observable from the video, preventing label leakage.' This is a critical assumption for the validity of the text modality contribution, but no quantitative leakage analysis is provided.
  • domain assumption The ETF prototype structure is appropriate for both discrete grading and continuous regression in AQA.
    Adopted from Zhou et al. (2024a) in Sec. 3.1; the coarse-to-fine prediction strategy for regression is stated but not independently validated against alternatives on MM-JDM.
  • domain assumption Baselines adapted to MM-JDM preserve their original model design and are fairly comparable.
    Invoked in Sec. 5.1: 'we adapt the implementation while preserving the original model design.' The exact adaptations are not specified, making fairness unverifiable.
invented entities (2)
  • MM-JDM dataset independent evidence
    purpose: Multi-modal AQA benchmark with RGB, optical flow, skeleton, and structured text for 12 clinical actions from pediatric JDM patients.
    The dataset is described in detail (Sec. 4, Tab. 2, Fig. 5) with 1639 samples, 29 subjects, and 12 actions. Code is publicly available. However, the dataset itself is only available upon request with ethical approval, limiting independent verification. The dataset serves as both a contribution and the primary evaluation benchmark, creating a self-referential evaluation loop.
  • Two-stage Gramian alignment (adapted GRAM) independent evidence
    purpose: Progressive alignment of visual modalities first, then visual-textual, using the Gramian volume minimization principle.
    The adaptation is described in Eqs. 10 and 15, and ablated against one-stage GRAM and no-alignment variants (Tab. 6). The falsifiable prediction — that two-stage outperforms one-stage — is tested and confirmed within the paper's own experiments. External validation on RG and Fis-V provides some independent grounding.

pith-pipeline@v1.1.0-glm · 41693 in / 4078 out tokens · 568250 ms · 2026-07-09T10:51:42.954910+00:00 · methodology

0 comments
read the original abstract

Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill assessment, and healthcare. However, unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings. Although multi-modal inputs provide complementary information, existing methods still face two major challenges: heterogeneous modalities often lead to cross-modal misalignment and unstable fusion, and reliable multi-modal annotation is costly, resulting in limited dataset diversity. To address these challenges, we propose DualAlign, a two-stage multi-modal fusion framework with adaptive alignment. The framework first constructs a coherent visual representation by maximizing shared structural information across RGB video, optical flow, and skeleton modalities. Textual semantics are then incorporated after visual stabilization, allowing high-level descriptions to complement rather than distort the underlying visual manifold. To evaluate the framework under realistic multi-modal conditions, we introduce MM--JDM, a movement-quality assessment dataset integrating RGB videos, optical flow, skeleton sequences, and structured text. MM--JDM naturally exhibits modality noise, class imbalance, and label scarcity, making it a challenging benchmark for studying multi-modal fusion and alignment. Extensive experiments show that DualAlign improves average correlation on MM--JDM by 21.16% over the state-of-the-art methods and achieves gains of 3.53% and 5.95% on the RG and Fis-V benchmarks, respectively. DualAlign also remains robust under missing-modality and label-scarce conditions.

Figures

Figures reproduced from arXiv: 2607.07438 by Jianguo Li, Kanglei Zhou, Liyuan Wang, Ruizhi Cai, Xiaohui Liang, Xinning Wang, Yijian Zheng.

Figure 1
Figure 1. Figure 1: Illustration of cross-modal representation discrepancies across [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The core insight of DualAlign is leveraging a two-stage fusion design with adaptive alignment to fully exploit the complementarity of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustrations of the first-stage fusion process: (a) depicts the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustrations of the second-stage fusion process: (a) depicts the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Category and sub-grade distribution of the MM–JDM dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Zero-shot evaluation of GPT-4o, Gemini-3 Pro, and DualAlign [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Progressive modality analysis from unimodal RGB input (V) [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Normalized performance comparison of one-stage and two [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Ablation results of different alignment strategies on MM– [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: CKA similarity heatmaps between Video (V), Flow (F), Skele [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Two-dimensional loss landscapes obtained by random param [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Performance under the missing label settings on the MM– [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Three samples and their predicted grade distributions on MM–JDM: the first five columns show different frames with the visualized skele [PITH_FULL_IMAGE:figures/full_fig_p020_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Representative samples and their predicted grade distributions on RG and Fis-V: the first five columns show different frames with the vi [PITH_FULL_IMAGE:figures/full_fig_p021_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

300 extracted references · 300 canonical work pages · 23 internal anchors

  1. [1]

    InfoGCN++: Learning Representation by Predicting the Future for Online Skeleton-based Action Recognition , author=

    work page

  2. [2]

    Neural Collapse: A Review on Modelling Principles and Generalization

    Neural collapse: A review on modelling principles and generalization , author=. arXiv preprint arXiv:2206.04041 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Transactions on Machine Learning Research , year=

    Neural Collapse: A Review on Modelling Principles and Generalization , author=. Transactions on Machine Learning Research , year=

    work page

  4. [4]

    A geometric analysis of neural collapse with unconstrained features , author=

    work page

  5. [5]

    2024 , volume =

    VideoPrism: A Foundational Visual Encoder for Video Understanding , author =. 2024 , volume =

    work page 2024

  6. [6]

    LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment , author=. arXiv preprint arXiv:2310.01852 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    2024 , url=

    LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment , author=. 2024 , url=

    work page 2024

  8. [8]

    Imagebind: One embedding space to bind them all , author=

    work page

  9. [9]

    Gramian Multimodal Representation Learning and Alignment

    Gramian Multimodal Representation Learning and Alignment , author=. arXiv preprint arXiv:2412.11959 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Accommodating audio modality in CLIP for multimodal processing , author=

    work page

  11. [11]

    arXiv preprint arXiv:2411.17040 , year=

    Multimodal Alignment and Fusion: A Survey , author=. arXiv preprint arXiv:2411.17040 , year=

    work page arXiv

  12. [12]

    International Journal of Computer Vision , year =

    Multimodal Alignment and Fusion: A Survey , author =. International Journal of Computer Vision , year =

    work page

  13. [13]

    Pointclip: Point cloud understanding by clip , author=

    work page

  14. [14]

    Neurocomputing , volume=

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning , author=. Neurocomputing , volume=. 2022 , publisher=

    work page 2022

  15. [15]

    Breakthroughs in statistics: methodology and distribution , pages=

    Relations between two sets of variates , author=. Breakthroughs in statistics: methodology and distribution , pages=

    work page

  16. [16]

    SIAM review , volume=

    An overview of sequence comparison: Time warps, string edits, and macromolecules , author=. SIAM review , volume=. 1983 , publisher=

    work page 1983

  17. [17]

    Learning transferable visual models from natural language supervision , author=

    work page

  18. [18]

    Clap learning audio concepts from natural language supervision , author=

    work page

  19. [19]

    Where are we now? , author=

    Juvenile dermatomyositis. Where are we now? , author=. Clinical and Experimental Rheumatology , volume=. 2022 , publisher=

    work page 2022

  20. [20]

    Nature Reviews Rheumatology , volume=

    Update on outcome assessment in myositis , author=. Nature Reviews Rheumatology , volume=. 2018 , publisher=

    work page 2018

  21. [21]

    Current rheumatology reports , volume=

    Juvenile dermatomyositis , author=. Current rheumatology reports , volume=

    work page

  22. [22]

    Multimodal learning with incomplete modalities by knowledge distillation , author=

    work page

  23. [23]

    Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient , author=

    work page

  24. [24]

    ACM Computing Surveys (CSUR) , volume=

    A survey of learning causality with data: Problems and methods , author=. ACM Computing Surveys (CSUR) , volume=. 2020 , publisher=

    work page 2020

  25. [25]

    Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric

    Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric , author=. arXiv preprint arXiv:2411.16619 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    A comprehensive survey of continual learning: theory, method and application , author=

    work page

  27. [27]

    Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality , author=

    work page

  28. [28]

    Realtime multi-person 2d pose estimation using part affinity fields , author=

    work page

  29. [29]

    Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29--July 2, 2003 Proceedings 13 , pages=

    Two-frame motion estimation based on polynomial expansion , author=. Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29--July 2, 2003 Proceedings 13 , pages=

    work page 2003

  30. [30]

    2020 , note=

    OpenMMLab Pose Estimation Toolbox and Benchmark , author=. 2020 , note=

    work page 2020

  31. [31]

    Learnable triangulation of human pose , author=

    work page

  32. [32]

    MediaPipe: A Framework for Building Perception Pipelines

    Mediapipe: A framework for building perception pipelines , author=. arXiv preprint arXiv:1906.08172 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  33. [33]

    Learning to estimate 3d hand pose from single rgb images , author=

    work page

  34. [34]

    Convolutional pose machines , author=

    work page

  35. [35]

    Deep network for the integrated 3d sensing of multiple people in natural images , author=

    work page

  36. [36]

    Depth Dataset Using Microsoft Kinect-v2 , author=

    work page

  37. [37]

    Motionagformer: Enhancing 3d human pose estimation with a transformer-gcnformer network , author=

    work page

  38. [38]

    Vitpose: Simple vision transformer baselines for human pose estimation , author=

    work page

  39. [39]

    3d human pose estimation in video with temporal convolutions and semi-supervised training , author=

    work page

  40. [40]

    ExpertAF: Expert Actionable Feedback from Video

    ExpertAF: Expert Actionable Feedback from Video , author=. arXiv preprint arXiv:2408.00672 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  41. [41]

    TechCoach: Towards Technical-Point-Aware Descriptive Action Coaching

    TechCoach: Towards Technical Keypoint-Aware Descriptive Action Coaching , author=. arXiv preprint arXiv:2411.17130 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [42]

    2024 , pages =

    Li, Yuan-Ming and Huang, Wei-Jin and Wang, An-Lan and Zeng, Ling-An and Meng, Jing-Ke and Zheng, Wei-Shi , title =. 2024 , pages =

    work page 2024

  43. [43]

    Proceedings of AI-ED , volume=

    Automated video assessment of human performance , author=. Proceedings of AI-ED , volume=

    work page

  44. [44]

    Analyzing diving: A dataset for judging action quality , author=

    work page

  45. [45]

    Estimation of skill levels in sports based on hierarchical spatio-temporal correspondences , author=

    work page

  46. [46]

    Assessing the quality of actions , author=

    work page

  47. [47]

    , author=

    Dynamical Regularity for Action Analysis. , author=

    work page

  48. [48]

    Workshop on Modeling and Monitoring of Computer Assisted Interventions 2014 , year=

    Video based assessment of OSATS using sequential motion textures , author=. Workshop on Modeling and Monitoring of Computer Assisted Interventions 2014 , year=

    work page 2014

  49. [49]

    International journal of computer assisted radiology and surgery , volume=

    Automated video-based assessment of surgical skills for training and evaluation in medical schools , author=. International journal of computer assisted radiology and surgery , volume=

    work page

  50. [50]

    The International Journal of Medical Robotics and Computer Assisted Surgery , volume=

    Automated robot-assisted surgical skill evaluation: Predictive analytics approach , author=. The International Journal of Medical Robotics and Computer Assisted Surgery , volume=. 2018 , publisher=

    work page 2018

  51. [51]

    Computer vision and image understanding , volume=

    A comparative study of pose representation and dynamics modelling for online motion quality assessment , author=. Computer vision and image understanding , volume=. 2016 , publisher=

    work page 2016

  52. [52]

    Attractor-shape for dynamical analysis of human movement: Applications in stroke rehabilitation and action recognition , author=

    work page

  53. [53]

    2005 IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance , pages=

    Behavior recognition via sparse spatio-temporal features , author=. 2005 IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance , pages=

    work page 2005

  54. [54]

    On space-time interest points , author=

    work page

  55. [55]

    Learning realistic human actions from movies , author=

    work page

  56. [56]

    A 3-dimensional sift descriptor and its application to action recognition , author=

    work page

  57. [57]

    Action recognition with improved trajectories , author=

    work page

  58. [58]

    A Short Note about Kinetics-600

    A short note about kinetics-600 , author=. arXiv preprint arXiv:1808.01340 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  59. [59]

    Learning structured output representation using deep conditional generative models , author=

    work page

  60. [60]

    Learning spatio-temporal representation with pseudo-3d residual networks , author=

    work page

  61. [61]

    Attention is All you Need , volume =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle = NeurIPS, editor =. Attention is All you Need , volume =

    work page

  62. [62]

    Learning spatiotemporal features with 3d convolutional networks , author=

    work page

  63. [63]

    Quo vadis, action recognition? a new model and the kinetics dataset , author=

    work page

  64. [64]

    The Kinetics Human Action Video Dataset

    The kinetics human action video dataset , author=. arXiv preprint arXiv:1705.06950 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  65. [65]

    Imagenet classification with deep convolutional neural networks , author=

    work page

  66. [66]

    Deep residual learning for image recognition , author=

    work page

  67. [67]

    Video swin transformer , author=

    work page

  68. [68]

    Ssd: Single shot multibox detector , author=

    work page

  69. [69]

    Mask r-cnn , author=

    work page

  70. [70]

    Auto-Encoding Variational Bayes

    Auto-encoding variational bayes , author=. arXiv preprint arXiv:1312.6114 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  71. [71]

    EgoExoLearn: A Dataset for Bridging Asynchronous Ego-and Exo-centric View of Procedural Activities in Real World , author=

    work page

  72. [72]

    2024 , doi =

    Vision-based human action quality assessment: A systematic review , journal =. 2024 , doi =

    work page 2024

  73. [73]

    Collaborative Weakly Supervised Video Correlation Learning for Procedure-Aware Instructional Video Analysis , author=

    work page

  74. [74]

    Achieving Procedure-Aware Instructional Video Correlation Learning Under Weak Supervision from a Collaborative Perspective , author=

    work page

  75. [75]

    RICA ^2 : Rubric-Informed, Calibrated Assessment of Actions , author =

    work page

  76. [76]

    Proceedings of the 28th International Conference on Intelligent User Interfaces , pages=

    IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment , author=. Proceedings of the 28th International Conference on Intelligent User Interfaces , pages=

    work page

  77. [77]

    LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment , author=

    work page

  78. [78]

    Visual-semantic Alignment Temporal Parsing for Action Quality Assessment , author=

    work page

  79. [79]

    2024 , issn =

    Dual-referenced assistive network for action quality assessment , journal =. 2024 , issn =

    work page 2024

  80. [80]

    MICCAI workshop: M2cai , pages=

    Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling , author=. MICCAI workshop: M2cai , pages=

    work page

Showing first 80 references.