{"id":"520f0a86-fbf1-4e64-9b1a-c7d9b61f4cf7","arxiv_id":"2411.10745","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A text-conditioned diffusion model over skeleton features gives a new state of the art for zero-shot skeleton-based action recognition.","lead":"This paper trains a text-conditioned diffusion model to denoise skeleton action features, then picks the action label whose text best reconstructs the fixed test noise. The method reports gains of 2.36 to 13.05 accuracy points over prior zero-shot skeleton action recognition approaches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No controlled non-diffusion baseline and test-set selection of ttest leave the central diffusion-attribution claim unsupported; a validation-tuned direct-alignment control is needed.","rationale":"The paper makes an interesting and potentially useful proposal, and the reported gains are large and mostly consistent across many splits. The method is specified in enough detail for reimplementation, and the ablations show that the diffusion loss and triplet loss each contribute within the proposed framework. However, the central claim is stronger than what the experiments establish: it explicitly contrasts diffusion-based alignment with direct alignment, but no within-framework direct-alignment baseline is provided. The reader's weakest assumption focused on the residual-ranking inference and test-set selection of ttest; I partially agree with that, but I see the missing controlled comparison as the more load-bearing issue because it concerns the causal attribution that defines the paper's contribution. The ttest-selection problem is also real and should be fixed by validation-based selection. These concerns are not accusations of fraud; they are requests for the comparisons and protocol choices that the paper's own framing demands. A single controlled experiment combining a validation-tuned timestep with a non-diffusion alignment head would resolve both issues: if the direct baseline matches TDSM, the novelty claim collapses; if TDSM clearly wins, the diffusion mechanism receives direct support. Until then, conditional acceptance is appropriate, with the additional condition that the authors provide this baseline and use a validation split for ttest. I therefore keep the reader's conditional verdict unchanged.","tokens_in":27931,"tokens_out":9521,"duration_ms":106411,"concrete_test":"Run one controlled experiment on NTU-60 55/5 and 48/12: (a) split the unseen test classes into a validation set for choosing ttest and a held-out test set; (b) train a non-diffusion direct-alignment head (e.g., a linear projection of zx and text features with the same triplet loss and the same CLIP features) under identical data, encoders, text prompts, and hyperparameters; (c) evaluate both TDSM and the direct-alignment baseline on the held-out test set using the validation-selected ttest. If the direct-alignment baseline matches or exceeds TDSM, the central diffusion-attribution claim fails; if TDSM still wins by the claimed margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that diffusion-based alignment, rather than direct alignment, is what drives the reported gains. That claim requires ruling out confounds, but the experiments never do so. All ablations (Tables 3, 4, 5, 6) vary components inside the same DiT denoiser; there is no baseline that replaces the diffusion denoiser with a direct feature-alignment head trained on the same encoders, text features, triplet loss, and hyperparameters. Without this control, the large margins over prior methods cannot be attributed to the diffusion mechanism. Second, the inference timestep ttest is selected from test-accuracy curves (Fig. 4, Sec. 5.2), so the reported numbers include test-set tuning; because ttest is a free hyperparameter and the curves vary by split, this can materially inflate the headline margins. Third, Eq. (15) ranks unseen labels by the residual of a text-conditioned function evaluated on one fixed noisy skeleton; training enforces residual margins only for random seen negatives, and no argument or diagnostic establishes that the correct unseen label systematically yields the smallest residual. Each gap weakens the central claim, and together they leave the empirical support conditional.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes TDSM, a diffusion-based method for zero-shot skeleton-based action recognition (ZSAR). Skeleton features are extracted with a frozen GCN encoder, and text is encoded with CLIP into global and local features. In training, a DiT-based denoiser is conditioned on positive and negative text prompts and is trained with a combination of a diffusion loss and a triplet diffusion loss. At inference, each unseen skeleton is noised once with a fixed Gaussian noise vector at a fixed timestep, the denoiser predicts the noise for each candidate label, and the label is selected by minimizing the residual distance ||epsilon_test - epsilon_hat_k||^2. The paper reports consistent gains over several recent methods on NTU-60, NTU-120, PKU-MMD, and Kinetics under SynSE, PURLS, and SMIE protocols, with ablations over loss terms, text feature types, total timesteps, and noise randomization.","tokens_in":28179,"tokens_out":5475,"duration_ms":56533,"significance":"The paper introduces a new formulation for ZSAR by converting cross-modal alignment into a text-conditioned denoising task, and the empirical results are strong if the protocol concerns are resolved. It also has positive methodological hygiene: the random seed is fixed, results are averaged over ten noise realizations, and the same skeleton encoders and text prompts as prior work are used for comparisons. The Kinetics results and the U-Net versus DiT comparison are useful additions. The significance is currently conditional: without a direct-alignment control, the claimed advantage of the diffusion mechanism over standard alignment is not established, and the test-set selection of the inference timestep weakens the reported margins.","major_comments":[{"comment":"The paper's central claim is that conditioning on text during the reverse diffusion process, rather than directly aligning skeleton and text latent spaces, is what produces the gains. The ablations vary only components inside the DiT denoiser, such as the loss terms, text features, total timesteps, and noise randomization. There is no control that replaces the diffusion denoiser with a direct alignment head, for example a linear projection or MLP trained on the same frozen skeleton encoder, the same CLIP text features, the same triplet-style loss, and the same hyperparameters. Without such a control, the large margins over prior methods cannot be attributed to the diffusion mechanism, and the central 'diffusion-powered alignment' claim remains unsupported.","section":"§5.4, Tables 3–6"},{"comment":"The inference timestep t_test = 25 is selected from test-accuracy trends shown in Fig. 4, and the same figure shows that accuracy varies by several points across t_test and across noise realizations. Because t_test is a free hyperparameter tuned on the test set, the headline gains in Tables 1 and 2 may be inflated. The authors should either choose t_test on a held-out validation split or report all benchmark numbers for a range of t_test values and show that the margins over the second-best methods are stable across that range. The supplementary statement in §B.4 that t_test = T/2 is empirically best is not a substitute for validation-based selection.","section":"§5.2, Fig. 4"},{"comment":"The inference rule assumes that ||epsilon_test - epsilon_hat_k||^2 is a reliable compatibility score for unseen labels. Training enforces residual margins only for random seen negatives, and no argument or diagnostic establishes that the correct unseen label systematically yields the smallest residual. The paper should provide a ranking diagnostic on unseen classes, such as the mean rank of the correct label or the separation between correct and incorrect residuals, and ideally compare Eq. (15) with alternative scoring functions, for example predicting the clean skeleton feature rather than the noise. The supplementary's own noise-sensitivity analysis in §A.1 makes this validation particularly important.","section":"§4.4, Eq. (15)"}],"minor_comments":[{"comment":"The tables report only mean accuracies without standard deviations; given that Fig. 4 shows up to about ±2.5% noise-induced variation, the 10-run averages should be accompanied by error bars or confidence intervals.","section":"Tables 1–6"},{"comment":"The word 'Non' appears in the negative-pair caption of the overview figure; this appears to be a typo for 'No' or 'Wrong', and it should be corrected.","section":"Fig. 1"},{"comment":"The claim that T = 50 is the best total timestep is not uniformly supported by the table, since T = 100 gives a higher accuracy on the NTU-60 48/12 split (56.27 versus 56.03); the authors should reconcile this observation with their conclusion.","section":"Table 5"},{"comment":"The main text says hyperparameters were set 'through empirical validation', while the supplementary says they were tuned on the NTU-60 SynSE benchmark; the main text should state the validation protocol explicitly so that the t_test tuning concern is not compounded by ambiguity about other hyperparameters.","section":"Supp. §E"},{"comment":"The comparison with BSZSL uses a method that also consumes RGB input; this should be clearly described as an additional reference point rather than a direct comparison, since the modality difference is material.","section":"Supp. Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the proposed method is a plausible new direction for ZSAR. The main risk is the missing direct-alignment control, which is required to support the diffusion-attribution claim. If the authors can add that control and fix the t_test selection protocol, the paper would become significantly stronger. I would also encourage the editor to ask for code or a detailed evaluation protocol, since the fixed-seed and 10-run averaging choices are good but not sufficient for full reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate new application of diffusion to zero-shot skeleton action recognition, with large and consistent gains across four benchmarks. The method (TDSM) conditions the reverse diffusion process on text to denoise skeleton features, then adds a triplet loss on noise residuals. That is genuinely different from the VAE and contrastive alignment used by prior ZSAR work. The paper is clearly written, the ablations show each loss term and text feature type contributes, and the supplement is candid about noise sensitivity and about the fact that the contribution is the framework rather than new components. The architecture description is detailed enough that a competent group could reimplement it.\n\nSoft spots are real but not fatal. The load-bearing claim is that diffusion-based alignment, rather than direct alignment, explains the margins. The ablations vary components inside the same DiT denoiser; there is no controlled baseline that replaces the denoiser with a direct feature-alignment head trained on the same encoders, text features, triplet loss, and hyperparameters. Without that control, the gains over prior methods cannot be cleanly attributed to the diffusion mechanism. Second, ttest=25 is selected from test-accuracy curves in Fig. 4, so the headline numbers include test-set tuning. The ±2.5 point noise sensitivity shown at that timestep is enough to matter for some of the smaller margins (2.36, 4.12 points). Third, Eq. (15) ranks unseen labels by the residual of a text-conditioned model evaluated on one fixed noisy skeleton. Training enforces residual margins only for random seen negatives, so the zero-shot ranking property is asserted rather than demonstrated. A diagnostic showing residual rankings on unseen classes, or at least a validation split for ttest, would substantially harden the claims.\n\nWho benefits: people working in ZSAR, and more broadly anyone curious whether diffusion can serve as a discriminative alignment mechanism on small, domain-specific data. The paper deserves a serious referee. I would send it out and ask for per-run variance, a validation-based choice of ttest, and ideally the direct-alignment control. If the diffusion advantage survives that control, this is a solid contribution; if it doesn't, the headline claim needs to be reframed.","headline":"New diffusion-based alignment recipe for zero-shot skeleton action recognition with big reported gains; the diffusion-specific attribution and ttest selection need hardening before I'd trust the headline margins.","tokens_in":28697,"tokens_out":3068,"would_cite":true,"duration_ms":31526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For zero-shot skeleton action recognition, text-guided denoising beats direct alignment of skeleton and text spaces.","keywords":["zero-shot action recognition","skeleton-based action recognition","diffusion models","modality alignment","text-conditioned denoising","triplet loss","skeleton-text matching","diffusion transformer"],"falsifier":"A concrete check is to take the trained model, draw many random noise samples for a held-out unseen class, and verify that the correct label's residual is systematically the smallest among all candidate labels; the prediction rule fails if it is not. A second check is to set the inference timestep on a validation split instead of the test accuracy curves and measure the resulting accuracy drop.","tokens_in":27700,"feed_emoji":"🦴","tokens_out":7852,"duration_ms":71121,"temperature":0.7,"pith_summary":"This paper tries to establish that the right way to bridge skeleton and text spaces for zero-shot action recognition is not to align them directly but to let text guide the denoising of skeleton features inside a diffusion process. The proposed framework, TDSM, trains a diffusion transformer to predict, from a noisy skeleton feature, the noise that was added, with a text prompt acting as the conditioning signal. At inference, an unseen skeleton sample is corrupted with a fixed Gaussian noise, and the label whose text condition produces the smallest residual $\\|\\epsilon_{\\mathrm{test}} - \\hat{\\epsilon}_k\\|^2$ is chosen. The claims are backed by reported accuracy gains of 2.36 to 13.05 percentage points over recent methods on NTU-60, NTU-120, PKU-MMD, and Kinetics-200/400. If the paper is right, diffusion models can serve as a general cross-modal alignment tool for zero-shot recognition even in small, domain-specific data regimes.","feed_headline":"Denoising with text guidance beats direct skeleton-text alignment","feed_subtitle":"TDSM lets text guide reverse diffusion to unify skeleton and label spaces, gaining up to 13 points in zero-shot accuracy","key_machinery":"The central object is TDSM (Triplet Diffusion for Skeleton-Text Matching), in which a diffusion transformer conditioned on both global and local text features predicts the noise added to a skeleton feature. The load-bearing identity is the one-step residual score $\\|\\epsilon_{\\mathrm{test}} - \\hat{\\epsilon}_k\\|^2$: the denoising residual is treated as a similarity score between a skeleton and a text label. The text-conditioned reverse diffusion performs the modality alignment, while the triplet diffusion loss enforces discrimination by comparing residuals of positive and negative text conditions. The framework deliberately uses the diffusion model's conditioning machinery rather than its generative path, so inference is a single forward pass at a fixed timestep.","core_discovery":"On the paper's own terms, the central discovery is that a diffusion model used purely as an alignment mechanism, rather than as a generator, can map skeleton features and text prompts into a unified latent space where unseen action labels rank correctly. The model is trained on seen classes only, with a diffusion loss $\\mathcal{L}_{\\mathrm{diff}} = \\|\\epsilon - \\hat{\\epsilon}_p\\|^2$ on the ground-truth label and a triplet diffusion loss $\\mathcal{L}_{\\mathrm{TD}} = \\max(\\|\\epsilon - \\hat{\\epsilon}_p\\|^2 - \\|\\epsilon - \\hat{\\epsilon}_n\\|^2 + \\tau, 0)$ that pulls correct skeleton-text pairs together and pushes wrong pairs apart. At inference, a fixed noise $\\epsilon_{\\mathrm{test}}$ and a fixed timestep $t_{\\mathrm{test}} = 25$ are used, and the predicted label is $\\arg\\min_k \\|\\epsilon_{\\mathrm{test}} - \\hat{\\epsilon}_k\\|^2$. The paper attributes the gain to implicit alignment: the reverse diffusion process fuses the modalities under text guidance, avoiding the direct point-wise alignment that prior VAE-based and contrastive methods rely on.","pith_inferences":["The paper does not claim this, but the same residual-ranking recipe should transfer to other skeleton-text tasks such as retrieval or few-shot classification, since the alignment mechanism is not tied to the action-label format.","The paper mentions but does not develop the refinement that predicting the clean skeleton latent instead of the noise reduces inference fluctuation by a factor of five; that is an extension toward a more stable variant.","A testable implication beyond the reported experiments is that because the fixed inference timestep was chosen from test accuracy curves, selecting it on a validation split could change the reported margins; the paper does not measure that.","A further semantic probe, not run in the paper, would be to perturb a text prompt toward a wrong action and check that the diffusion residual increases monotonically, which would confirm the unified space is genuinely semantic."],"forward_implications":["Unseen action labels can be ranked by a one-step diffusion residual, so zero-shot inference needs no iterative sampling and no generated features.","The same framework transfers across standard, extreme, and body-part-based splits on NTU-60, NTU-120, PKU-MMD, and Kinetics, with reported gains of 2.36 to 13.05 percentage points over recent methods.","The triplet diffusion loss carries the discriminative part of the gain: combining it with the diffusion loss turns roughly 53-58 percent top-1 accuracy into 56-65 percent on the harder NTU-120 splits.","A single text prompt per action suffices; on Kinetics-200 and Kinetics-400, one-prompt TDSM outperforms a seven-prompt baseline.","Random Gaussian noise during training acts as a regularizer, with fixed noise cutting accuracy by roughly ten points."],"supporting_citations":[{"why":"Supplies the forward and reverse diffusion formulation and the noise-prediction objective that TDSM adapts for alignment rather than generation.","marker":"[22]"},{"why":"Establishes latent diffusion with text conditioning, the cross-modal mechanism TDSM reuses for skeleton-text fusion.","marker":"[52]"},{"why":"Provides the diffusion transformer backbone whose modulation and self-attention blocks are used as the denoiser.","marker":"[48]"},{"why":"Supplies the pretrained text encoder that produces the global and local text features conditioning the reverse process.","marker":"[51]"},{"why":"Gives the conventional triplet loss that the triplet diffusion loss reformulates into residual-space push-pull.","marker":"[23]"},{"why":"Defines the PURLS benchmark splits, text prompts, and encoder protocol that TDSM compares against and extends.","marker":"[79]"},{"why":"Defines the SynSE benchmark splits and prompt protocol used for the standard zero-shot evaluation.","marker":"[20]"},{"why":"Defines the SMIE benchmark splits and skeleton encoder protocol used in the second evaluation suite.","marker":"[77]"}],"fun_headline_variants":["Text-guided denoising unifies skeleton and text for zero-shot recognition","Diffusion as aligner, not generator, for skeleton-text matching","Triplet diffusion loss lifts zero-shot action recognition by up to 13 points","Diffusion-driven alignment, not generation, for zero-shot skeleton action recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, for a fixed Gaussian noise and a fixed timestep chosen from test accuracy curves, the one-step denoising residual ranks unseen action labels correctly even though the diffusion model was trained only on seen classes.","fun_headline_variants_meta":{"raw":{"variants":["Text-guided denoising unifies skeleton and text for zero-shot recognition","Diffusion as aligner, not generator, for skeleton-text matching","Triplet diffusion loss lifts zero-shot action recognition by up to 13 points","Diffusion-driven alignment, not generation, for zero-shot skeleton action recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3394,"prompt_tokens":1057,"completion_tokens":2337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":2258}},"tokens_in":673,"tokens_out":2337,"duration_ms":17822,"temperature":1.0,"reasoning_tokens":2258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:21:40.767889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to take the trained model, draw many random noise samples for a held-out unseen class, and verify that the correct label's residual is systematically the smallest among all candidate labels; the prediction rule fails if it is not. A second check is to set the inference timestep on a validation split instead of the test accuracy curves and measure the resulting accuracy drop.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained text encoder that produces the global and local text features conditioning the reverse process."},{"cited_title":"Part-aware unified representation of language and skele- ton for zero-shot action recognition","cited_arxiv_id":null,"evidence_quote":"Defines the PURLS benchmark splits, text prompts, and encoder protocol that TDSM compares against and extends."},{"cited_title":"Zero-shot skeleton-based action recogni- tion via mutual information estimation and maximization","cited_arxiv_id":null,"evidence_quote":"Defines the SMIE benchmark splits and skeleton encoder protocol used in the second evaluation suite."}],"review_version":1}