{"id":"622d6b7c-6b93-4319-bd94-ffa2da785b6d","arxiv_id":"2509.05543","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DuoCLR pretrains on trimmed skeleton sequences using Shuffle-and-Warp multi-action permutations and two surrogate tasks, significantly improving action segmentation on untrimmed videos.","lead":"DuoCLR is a contrastive pretraining method for skeleton-based action segmentation that builds multi-action sequences by shuffling and warping trimmed single-action clips. It introduces two self-supervised tasks, cross-permutation contrasting and relative order reasoning, and reports large gains on untrimmed video benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported superiority may be an artifact of two evaluation protocol choices: pretraining on the source test split (Sec. 4.1) and selecting action granularity G on the target test set (Sec. 4.3).","rationale":"Agreement with reader: partial. The Warp registration in Eq. (3) is indeed under-specified, and the ablation shows it matters, but I do not think it is the most load-bearing issue: even if the alignment is fully reproducible, the current tables may not demonstrate superiority because of the evaluation protocol. The manuscript explicitly states in Sec. 4.1 that pretraining uses the entire (training plus testing) set of the source trimmed datasets, and Sec. 4.3 selects the action granularity G based on Acc on PKU2. Both choices are visible in the text but appear inconsistent with the benchmark conventions used by the comparatives. If these are corrected and the margins persist, the central claim stands; if not, the claim must be restated. This is exactly the kind of condition that a CONDITIONAL verdict should carry. I therefore keep the reader's verdict unchanged, but with a sharper condition attached to it.","tokens_in":14868,"tokens_out":13478,"duration_ms":122085,"concrete_test":"Re-run the NTU→PKU1 and NTU→PKU2 protocols with pretraining restricted to the official NTU training split (no test split) and with action granularity G fixed to a value selected on a held-out validation split of the target dataset (e.g., 10% of PKU training sequences), then compare DuoCLR to LAC and AimCLR under the same data. Report mAP@0.1 for PKU1 and mIoU/Acc for PKU2 for G=2,3,4. If the PKU1 linear margin over LAC (61.8→85.2) shrinks substantially or the PKU2 advantage over AimCLR (21.4→54.5 mIoU, 51.3→73.9 Acc) narrows, the headline claim should be restated as conditional on a chosen split and granularity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Sec. 4.2 is that DuoCLR outperforms most state-of-the-art comparatives; the evidence is the large margins in Tables 1 and 2. Two evaluation-protocol choices make those margins hard to interpret. First, Sec. 4.1 states that pretraining uses 'the entire (both training and testing) set' of each trimmed source dataset. NTU, Kinetics-400, and Toyota Smarthome have official train/test splits, and prior self-supervised pretraining methods are typically pretrained on the training split only. If comparatives such as LAC [48] and AimCLR [18] used only the training split, DuoCLR has roughly twice the source data, so the comparison does not isolate the method. Second, Sec. 4.3 treats action granularity G as a hyperparameter and selects it using 'Acc values evaluate the transferability of the resultant (CPC+ROR) network on PKU 2'; the best G is reported as about 4. The final PKU2 numbers (and the shared G used for PKU1) are therefore selected on the evaluation set, not predicted. Neither issue alone is fatal, but together they undercut the central claim as stated: the reported 85.2/74.3 mAP and 54.5/73.9 mIoU/Acc numbers may reflect test-set access rather than a generally better representation. A clean re-run is needed before the claim can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DuoCLR, a contrastive pretraining method for skeleton-based action segmentation. DuoCLR samples trimmed single-action skeleton sequences, shuffles and warps them into multi-action permutations, and trains a GCN+TCN encoder with two surrogate losses: CPC (InfoNCE on same-class action segments across permutations) and ROR (classification of the relative permutation mapping). The pretrained encoder is transferred to untrimmed segmentation datasets (PKU1, PKU2, TSU, Charades) under linear and fine-tuned protocols. The paper reports large gains over baselines and ablates the surrogate tasks, augmentation components, action granularity, and encoder structure.","tokens_in":15208,"tokens_out":5621,"duration_ms":50179,"significance":"If the reported results are reproducible, DuoCLR would be a meaningful step: it produces a sliding-window-free, transferable skeleton feature extractor for frame-level action segmentation, and it explicitly targets the trimmed-to-untrimmed gap that most prior contrastive skeleton work ignores. The paper ships a project page with code, uses multiple external benchmarks, and provides component-wise ablations (Tables 5, 7, 8). These strengths make the central idea worth pursuing. However, the headline comparisons are currently compromised by two evaluation-protocol choices—pretraining on the full source splits (Sec. 4.1) and selecting action granularity G on the target evaluation set (Sec. 4.3)—so the significance claim is not yet established at the level stated in Sec. 4.2.","major_comments":[{"comment":"Pretraining uses \"the entire (both training and testing) set\" of each trimmed source dataset. Standard self-supervised skeleton pretraining uses only the training split (e.g., AimCLR, ActCLR, Hi-TRS in Table 1); if those comparatives used only the training split, DuoCLR is trained on roughly twice the source data and additionally uses the source test split's action labels to form CPC positive pairs and ROR classes. This protocol difference alone can explain part of the large margins in Tables 1 and 2 and prevents the comparison from isolating the proposed surrogate tasks and augmentation. Please re-run the main comparisons with the same source split convention as the baselines (train-only), or report baseline numbers trained under the same full-source protocol.","section":"Sec. 4.1 (Pretraining), Tables 1-2"},{"comment":"The action granularity G is tuned by measuring \"Acc values\" of the CPC+ROR network on PKU2, and G≈4 is then used for the reported PKU2 and PKU1 results. This is model selection on the evaluation set: the headline numbers in Tables 1 and 2 for PKU2/PKU1 are not independent of the test labels. Please replace this with a validation-split or cross-validation-based selection rule, or clearly report the G values used per dataset and show that the conclusions are stable across a range of G.","section":"Sec. 4.3 (Effectiveness of Action Granularity), Fig. 6"},{"comment":"SCS [39]—the authors' own prior work—is the closest published method: it also builds multi-action sequences from trimmed skeletons and uses contrastive learning for segmentation, and it is discussed in Sec. 2. SCS does not appear in Tables 1 or 2, and the text does not explain its absence. Without a direct comparison to SCS, the claim of outperforming \"most state-of-the-art comparatives\" is incomplete, and the incremental contribution of Shuffle-and-Warp plus CPC/ROR over SCS's offline stitching is not quantified. Please add SCS to the benchmark tables under matched protocols.","section":"Related work, Tables 1-2"},{"comment":"The Warp operation is a load-bearing component: Table 7 shows that removing Warp drops CS mAP0.1 from 85.2 to 73.8. However, the rotation/translation/scale estimation is deferred to the supplementary material with only a one-line reference, and Eq. (3) assumes a single global similarity transform can align any two heterogenous skeleton sequences into one camera configuration. Since the generality of this assumption is central to the transfer claims, the estimation procedure, its failure modes, and the sequence-level concatenation details need to be specified in the main text or in a directly accessible appendix, with at least one qualitative example of alignment success and failure.","section":"Sec. 3.2, Eq. (3), Table 7"}],"minor_comments":[{"comment":"There are copyediting issues in this section: \"and and the best model\" should be \"and the best model\", \"Siding-window-free\" should be \"Sliding-window-free\", and \"suppassing\" should be \"surpassing\".","section":"Sec. 4.3"},{"comment":"No error bars or multiple-seed statistics are reported; given the large claimed margins, reporting mean and standard deviation over at least three seeds for the main tables would strengthen the empirical claims.","section":"Tables 1-5"},{"comment":"The semi-supervised protocol should define what the 5% and 10% refer to (labeled videos? labeled frames? random subsets?) and how the remaining target-domain data is used during evaluation.","section":"Table 3"},{"comment":"The data-savings numbers are presented without specifying the baseline model and protocol used to define \"compatible performance with the vanilla model trained on 100% data\"; please clarify the comparison protocol.","section":"Table 4"},{"comment":"In Eq. (1), τ is called a temperature parameter but the formula as written is just a normalized cosine similarity; please show where τ enters (e.g., inside the exponent) so that Eq. (1) matches the InfoNCE convention used in Eqs. (7)-(8).","section":"Eq. (1), Eqs. (7)-(8)"},{"comment":"The confusion matrices and similarity matrices are difficult to read at the current figure resolution; please enlarge the panels or separate them into two figures.","section":"Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The protocol concerns in Sec. 4.1 and Sec. 4.3 are the main blockers: they affect the interpretation of the headline numbers rather than the internal derivation. If the authors can re-run the key comparisons with train-only pretraining and a validation-based choice of G, and add the SCS baseline, the paper could become acceptable. The omission of SCS is likely to be seen as under-citing the closest prior work and should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on DuoCLR. The core idea is real: instead of treating each trimmed skeleton sequence as an isolated instance, the paper shuffles several sequences together, aligns them geometrically with a Warp transform, and trains two complementary surrogate tasks—Cross Permutation Contrasting (CPC) for intra-class invariants and Relative Order Reasoning (ROR) for inter-class ordering. That's a sensible direction for action segmentation, where context between actions matters. The ablations are thorough: each component (Shuffle, Warp, CPC, ROR) contributes, and the hierarchical GCN+TCN encoder is a sound choice. Code is promised on a project page.\n\nBut the central comparison table should not be taken at face value. Two protocol choices in Section 4 directly inflate the reported margins. First, pretraining uses \"the entire (both training and testing) set\" of NTU, Kinetics-400, and Toyota Smarthome. Standard practice—and almost certainly what the comparatives like LAC and AimCLR did—is to pretrain on the training split only. Doubling the source data makes the comparison unfair and does not isolate the method. Second, action granularity G is treated as a free hyperparameter and selected by evaluating the trained network on PKU2, the evaluation dataset. The claim \"the best model is achieved when G is around 4\" means the final PKU1/PKU2 numbers are test-set-tuned, not predictions. Either issue alone would need a caveat; together they undercut the superiority claim.\n\nThere's also a missing baseline that matters: the authors' own SCS (ref [39]) does temporal stitching for the same task and is cited in related work but not compared against. That omission is hard to justify in a SOTA claim. On a smaller point, the Warp parameter estimation is deferred to supplementary, and Warp is responsible for a large chunk of the gain (73.8 vs 85.2 in Table 7), so the method's robustness hinges on a component the paper doesn't fully expose. No error bars or multiple seeds are reported, which is common but still a weakness.\n\nSo what does the paper actually contribute? A new data augmentation and two-loss contrastive design that plausibly improves representation quality for segmentation. The idea deserves to be published in a cleaned-up form, but the quantitative claims need a re-run: pretrain on train splits only, select G on a validation split, compare against SCS, report variance.\n\nFor peer review: yes, it deserves a serious referee—the subfield needs pretraining schemes that reduce frame-level annotation, and this is a fresh approach. But I'd expect heavy revision before acceptance. I wouldn't cite it in its current form.\n\nBest.","headline":"Genuinely new pretraining idea for skeleton action segmentation, but the reported margins are likely inflated by test-set access and an unfair data split.","tokens_in":15688,"tokens_out":2667,"would_cite":false,"duration_ms":23082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining on trimmed skeleton clips with shuffle-and-warp augmentations transfers to action segmentation and beats fully supervised baselines.","keywords":["skeleton-based action segmentation","contrastive representation learning","self-supervised pretraining","data augmentation","shuffle and warp","temporal action segmentation","multi-label action segmentation","transfer learning"],"falsifier":"On a transfer pair with a large viewpoint gap, e.g. pretraining on NTU RGB+D and evaluating on an untrimmed dataset recorded from an overhead camera, compare DuoCLR with and without Warp using the same linear-evaluation protocol; if mAP@0.1 drops toward the Shuffle-only value of 73.8 or below as viewpoint mismatch grows, then the single rigid alignment used by Warp is the weak link.","tokens_in":14681,"feed_emoji":"🦴","tokens_out":6778,"duration_ms":57494,"temperature":0.7,"pith_summary":"DuoCLR claims that a contrastive model pretrained on trimmed single-action skeleton sequences can learn a frame-level feature representation that transfers directly to untrimmed action segmentation, bypassing the expensive frame annotations usually needed for that task. The paper argues that prior contrastive methods fall short at segmentation because they learn isolated sequence-wise representations and ignore the action contexts that appear when multiple actions are chained together. To fix this, it generates multi-action permutations by shuffling trimmed sequences and warping them into a common camera configuration, then trains two surrogate tasks: Cross Permutation Contrasting (CPC), which pulls together representations of the same action across different contexts, and Relative Order Reasoning (ROR), which predicts the relative order mapping between two permutations. On the NTU-to-PKU1 transfer, linear evaluation mAP@0.1 rises from 56.0 for the supervised baseline to 85.2, and fine-tuning reaches 94.4 CS / 96.8 CV. A sympathetic reader would take the paper's central contribution to be a pretraining recipe that turns abundant trimmed skeleton data into a reusable feature extractor for segmentation.","feed_headline":"Pretraining on single actions lifts skeleton segmentation mAP to 85","feed_subtitle":"Shuffle-and-warp augmentations with dual surrogate tasks transfer trimmed clips to untrimmed videos without sliding windows","key_machinery":"Shuffle and Warp is the load-bearing augmentation: trimmed single-action skeleton sequences are sampled, permuted, and concatenated into multi-action sequences, with each incoming sequence warped by an estimated rotation, translation, and scale so that the concatenation looks like a physically coherent video rather than a splice of inconsistent camera views. The two surrogate tasks then operate on these permutations. CPC treats local action encodings from different permutations as positive pairs and pushes them together through an InfoNCE loss with memory banks, so the same action becomes context-invariant. ROR builds pairwise difference features between the local projections of two permutations and classifies which of the $|P_j|!$ relative orders connects them, forcing the network to encode action order and long-range context. A temporal encoder built from dilated temporal convolutions over the graph convolutional features gives the feature extractor a large receptive field, so downstream segmentation needs no sliding windows.","core_discovery":"The central claim is that self-supervised learning for skeleton action segmentation should operate on multi-action permutations of trimmed sequences rather than on isolated sequences, and that two complementary surrogate tasks make this work. CPC enforces permutation-invariant intra-class similarity: the same action appearing in different multi-action contexts should produce the same local feature. ROR enforces permutation-aware inter-class structure: from the features of two permutations that share the same action set, the model must recover the relative ordering of the actions. Together, the two tasks regularize the feature space at both local and global scales. The paper reports that this pretraining, with a graph convolutional visual encoder and a dilated temporal encoder, transfers to untrimmed multi-class and multi-label segmentation datasets and outperforms earlier self-supervised and end-to-end supervised methods on most benchmarks, e.g. mAP@0.1 of 85.2 versus 56.0 for the linear-evaluation baseline on NTU to PKU1.","pith_inferences":["If the Warp alignment generalizes beyond the tested camera settings, the same trimmed-to-untrimmed pretraining could transfer across skeleton datasets with different coordinate conventions, a setting the paper does not evaluate.","The pairing of an invariance task (CPC) with an order-reasoning task (ROR) is a general recipe; a similar dual-surrogate design could apply to other frame-level tasks such as temporal action localization or motion forecasting.","A direct testable extension is to replace the single rigid Warp with a learned or sequence-specific alignment and measure whether the remaining gap between DuoCLR and its Shuffle-only variant shrinks.","Because the unsupervised variant still reaches 81.6 mAP@0.1, the framework may run on unlabeled trimmed video at a modest cost, which the paper notes but does not exploit beyond one experiment."],"forward_implications":["With linear evaluation only, pretrained features push NTU-to-PKU1 mAP@0.1 from 56.0 to 85.2, so the extractor is useful without fine-tuning.","Fine-tuning the pretrained extractor reaches 94.4 CS / 96.8 CV on PKU1, above the end-to-end supervised segmentation baseline, so the pretraining transfers beyond a fixed feature.","The same recipe improves composite multi-label segmentation on Toyota Smarthome Untrimmed and Charades, where frames carry multiple labels.","DuoCLR reduces frame-level label requirements by 42-63% while matching fully supervised performance on the tested datasets.","The augmentation and surrogate tasks compose with classical augmentations like Shear and Crop, raising PKU1 accuracy from 85.2 to 87.1, so the gains are not tied to one augmentation family."],"supporting_citations":[{"why":"Supplies the momentum-encoder contrastive base, extreme augmentation idea, and memory-bank implementation that DuoCLR adapts.","marker":"[18]"},{"why":"Provides the graph convolutional visual encoder whose temporal pooling DuoCLR removes to keep frame-level resolution.","marker":"[46]"},{"why":"Provides the dilated temporal convolution stack used as DuoCLR's temporal encoder for long-range context.","marker":"[15]"},{"why":"LAC is the main competing trimmed-to-untrimmed segmentation method and the sliding-window baseline DuoCLR compares against.","marker":"[48]"},{"why":"Establishes the cross-view linear-evaluation protocol and the Shear/Crop augmentations used in DuoCLR's ablations.","marker":"[24]"},{"why":"MoCo supplies the InfoNCE objective and momentum contrast formulation used for CPC.","marker":"[5]"},{"why":"MS2L is the multi-task self-supervised skeleton baseline and connects the jigsaw/order-prediction lineage that ROR extends.","marker":"[27]"}],"fun_headline_variants":["Skeleton segmentation mAP jumps to 85 with dual surrogates","Two surrogate tasks segment skeletons from single-action clips","Permuted single-action clips pre-train for untrimmed segmentation","Shuffle-and-warp pre-training lifts skeleton segmentation mAP","Contrastive learning with action permutations boosts segmentation mAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Warp step can bring any two trimmed skeleton sequences into one consistent camera configuration using a single rotation, translation, and scale, so that the concatenated sequences are physically coherent; the ablation attributes much of the gain to this alignment, yet the parameter estimation is described only in the supplementary material.","fun_headline_variants_meta":{"raw":{"variants":["Skeleton segmentation mAP jumps to 85 with dual surrogates","Two surrogate tasks segment skeletons from single-action clips","Permuted single-action clips pre-train for untrimmed segmentation","Shuffle-and-warp pre-training lifts skeleton segmentation mAP","Contrastive learning with action permutations boosts segmentation mAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2496,"prompt_tokens":969,"completion_tokens":1527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1442}},"tokens_in":585,"tokens_out":1527,"duration_ms":14394,"temperature":1.0,"reasoning_tokens":1442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:33.834098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a transfer pair with a large viewpoint gap, e.g. pretraining on NTU RGB+D and evaluating on an untrimmed dataset recorded from an overhead camera, compare DuoCLR with and without Warp using the same linear-evaluation protocol; if mAP@0.1 drops toward the Shuffle-only value of 73.8 or below as viewpoint mismatch grows, then the single rigid alignment used by Warp is the weak link.","supporting_citations":[{"cited_title":"Contrastive learning from ex- tremely augmented skeleton sequences for self-supervised action recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the momentum-encoder contrastive base, extreme augmentation idea, and memory-bank implementation that DuoCLR adapts."},{"cited_title":"Spatial tempo- ral graph convolutional networks for skeleton-based action recognition","cited_arxiv_id":null,"evidence_quote":"Provides the graph convolutional visual encoder whose temporal pooling DuoCLR removes to keep frame-level resolution."},{"cited_title":"Ms-tcn: Multi-stage tem- poral convolutional network for action segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the dilated temporal convolution stack used as DuoCLR's temporal encoder for long-range context."},{"cited_title":"Lac-latent action composition for skeleton-based ac- tion segmentation","cited_arxiv_id":null,"evidence_quote":"LAC is the main competing trimmed-to-untrimmed segmentation method and the sliding-window baseline DuoCLR compares against."},{"cited_title":"3d human action rep- resentation learning via cross-view consistency pursuit","cited_arxiv_id":null,"evidence_quote":"Establishes the cross-view linear-evaluation protocol and the Shear/Crop augmentations used in DuoCLR's ablations."},{"cited_title":"Ms2l: Multi-task self-supervised learning for skeleton based action recognition","cited_arxiv_id":null,"evidence_quote":"MS2L is the multi-task self-supervised skeleton baseline and connects the jigsaw/order-prediction lineage that ROR extends."}],"review_version":2}