{"id":"8d1ec921-c8fe-40b2-b537-f0990e2dec5f","arxiv_id":"2603.05962","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Skeleton-to-Image Encoding maps joint sequences into body-part-ordered RGB-like images so MAE and DiffMAE transfer ImageNet pretraining to self-supervised skeleton action recognition, including cross-format and multi-dataset settings.","lead":"This paper turns 3D human skeleton sequences into image-like tensors so large vision models pretrained on photos can learn skeleton features without a custom skeleton network. That reuse of ImageNet weights and a format-agnostic layout may simplify multi-dataset and multi-modal action recognition.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-noted encoding assumption and metadata mismatch.","rationale":"The manuscript is a solid empirical methods paper whose central claim—that S2I lets ImageNet-pretrained vision models perform competitive self-supervised skeleton representation learning and enables format-agnostic multi-dataset pretraining—is supported by the reported tables and ablations. The single most load-bearing condition is precisely the one the reader named: that the pseudo-image retains the kinematics that matter. Because the paper already shows gains from ImageNet initialization, random masking, and multi-stream fusion, and because cross-format transfer (NTU\to Toyota/NW-UCLA) and universal pretraining improve over single-dataset self-pretraining, that assumption is empirically buttressed rather than contradicted. No deeper inconsistency (e.g., in the MAE/DiffMAE losses of Eqs. 1–2, the joint-partition tables, or the evaluation protocols) appears. Therefore the reader's CONDITIONAL verdict—conditioned on code release and tighter “first/SOTA/heterogeneous” language—remains the appropriate stance; no adjustment is warranted. The concrete permutation test would cleanly falsify or reinforce the encoding assumption without requiring new datasets.","tokens_in":22194,"tokens_out":606,"duration_ms":5094,"concrete_test":"Reproduce the NTU-60 C-sub linear-probe and fine-tune numbers of Table 1 (DiffMAE row with both pretrains) after replacing the five-part top-down ordering with a random joint permutation before stacking/resizing; if accuracy drops by more than ~5–8 points relative to the reported 83.1/91.0, the kinematic ordering is confirmed load-bearing; if it does not, the encoding is more robust than claimed and the weakest assumption softens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption correctly isolates the central risk: that S2I (Section 3.1, Figure 2)—mapping (x,y,z) to RGB, five-part kinematic reordering, and bilinear resize to 224\times224—preserves enough articulated motion structure for ImageNet-pretrained MAE/DiffMAE masked reconstruction to transfer useful priors. The paper's own ablations (Table 1: ImageNet init + skeleton pretrain; Table 2: random 75% masking best; Table 3: multi-stream fusion) and cross-format results (Tables 8–9) provide empirical support that the encoding works on the chosen benchmarks, without an internal contradiction. No stronger load-bearing flaw (e.g., circular evaluation, unstated shared-joint leakage, or broken loss derivation) is evident in the supplied text. The title/abstract mismatch (OVOR/CLIP vs. S2I skeleton) is a prompt artifact already flagged by the reader and does not undermine the skeleton claims themselves.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes Skeleton-to-Image Encoding (S2I), which partitions skeleton joints into five semantic body parts, reorders them top-down along kinematic chains, maps (x,y,z) coordinates to RGB channels, stacks frames temporally, and resizes the resulting T\times J map to 224\times224 via linear interpolation. This image-like representation allows ImageNet-pretrained MAE and DiffMAE (ViT-B) to be used directly for self-supervised skeleton pretraining via masked reconstruction (Eqs. 1–2), followed by linear probing or fine-tuning for action recognition (Eq. 3). The method is evaluated on NTU-60/120, PKU-MMD I/II, NW-UCLA, and Toyota under linear, fine-tune, semi-supervised, transfer, cross-format, and universal multi-dataset pretraining protocols, reporting competitive or SOTA numbers (especially with 3-stream joint/bone/motion fusion) without skeleton-specific architectures.","tokens_in":22420,"tokens_out":982,"duration_ms":8380,"significance":"If the central claim holds, S2I provides a simple, format-agnostic bridge that lets the community reuse large-scale vision-pretrained models and weights for skeleton representation learning. The cross-format transfer (Table 8) and universal pretraining (Table 9) results are particularly valuable, because they address the long-standing problem of heterogeneous joint layouts across capture devices without manual joint subsetting or model redesign. The work also supplies extensive multi-protocol ablations (Tables 1–3) that isolate the contribution of ImageNet initialization, masking strategy, and multi-stream fusion, giving the community a concrete, reproducible baseline for vision-to-skeleton transfer.","major_comments":[{"comment":"Section 3.1 and Figure 2: the claim that S2I “preserves essential spatial-temporal patterns” rests on an untested assumption that bilinear resize of the T\times J map to 224\times224 (and the fixed five-part ordering) retains the articulated kinematics that matter for action. No ablation compares S2I against alternative layouts (e.g., graph-distance ordering, no body-part partition, or different interpolation) or quantifies information loss; without such evidence the transfer gains in Tables 4–9 cannot be confidently attributed to the encoding rather than to the strong MAE/DiffMAE backbones and multi-stream fusion.","section":null},{"comment":"Tables 4–5 and 8: several “SOTA” claims for 3s-S2I are obtained by late fusion of three independently trained streams (joint/bone/motion). The single-stream S2I numbers are competitive but not consistently superior to recent specialized methods (e.g., MAMP, MacDiff). The paper should either (a) report single-stream results as the primary comparison or (b) apply the same multi-stream fusion to the strongest baselines so that the contribution of S2I itself is isolated.","section":null}],"minor_comments":[{"comment":"The supplied abstract and title describe an unrelated OVOR/CLIP object-recognition framework; the actual manuscript is about skeleton-to-image encoding. This metadata mismatch must be corrected before any further review cycle.","section":null},{"comment":"Section 3.3 and Table 2: “Group masking” is never formally defined; a short equation or diagram would clarify how it differs from block masking.","section":null},{"comment":"Appendix E / Table 14: the joint-to-body-part mapping for the 13-joint Toyota skeleton collapses the spine to a single “head” joint; a brief justification of how this affects the five-part layout would help readers reproduce the universal-pretraining setting.","section":null},{"comment":"Figure 1 and Figure 4 contain placeholder or low-resolution glyphs that render as boxes; higher-resolution versions are needed for print.","section":null},{"comment":"Minor typos: “spatio-temporal” vs. “spatial-temporal” inconsistency; “DiffMAE’s consistently superior performance” (Table 1 shows only a 0.5–1.7 point edge).","section":null}],"recommendation":"major_revision","confidential_remarks":"The title/abstract mismatch appears to be a prompt-construction artifact rather than author error; once corrected the paper is a solid systems contribution. The encoding assumption is the only load-bearing risk; if the authors add a modest layout ablation the paper becomes a clear accept for a vision/action venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The manuscript we actually have is the Skeleton-to-Image (S2I) work, not the OVOR/CLIP abstract in the metadata—treat the title mismatch as prompt noise and judge the skeleton paper.\n\nWhat is new is concrete and usable: partition joints into five body parts, order them kinematically, map (x,y,z) to RGB, stack time, resize to 224×224, then run ImageNet-initialized MAE or DiffMAE with masked reconstruction. That gives a format-agnostic image that supports joint training across 13/20/25-joint layouts without redesigning the backbone. The cross-format transfer (NTU→Toyota/NW-UCLA) and universal multi-dataset pretrain tables are the parts that matter beyond another NTU leaderboard entry.\n\nThe paper does the empirical job carefully. Ablations show ImageNet init is load-bearing (big linear-probe jump), random 75% masking beats joint/temporal/group, and joint+bone+motion fusion still helps after the image conversion. Linear, fine-tune, semi-supervised, and transfer protocols on NTU-60/120, PKU I/II, NW-UCLA, and Toyota are thorough; 3s-S2I is competitive or best on several splits. Losses and evaluation are standard, not circular. Citations to prior skeleton SSL are appropriate baselines.\n\nSoft spots are real but proportional. The central assumption—that the pseudo-image preserves enough articulated motion for ViT reconstruction to transfer useful priors—is only supported by end-task numbers, not by analysis of what kinematics are destroyed by bilinear resize or RGB packing. No code or error bars. “First” and some SOTA claims sit a bit high relative to close prior numbers (MAMP, MacDiff, etc.). Free parameters (mask ratio, epochs, diffusion schedule) are normal for the genre.\n\nThis is for people working on skeleton SSL, multi-modal action, or anyone who wants a simple bridge into the vision-model ecosystem. It is not theory and does not reorganize the field, but it is clear thinking, honest ablations, and a practical idea that works on public benchmarks. I would send it to peer review; a serious editor should not desk-reject it. Engage if you care about skeleton representation or cross-format pretraining; skip if you only track dense video or language-vision.","headline":"Solid empirical skeleton SSL paper: S2I lets you run ImageNet MAE/DiffMAE on joint sequences and get competitive numbers plus a real cross-format angle; not foundational, but useful and referee-worthy.","tokens_in":23072,"tokens_out":601,"would_cite":true,"duration_ms":15791,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Skeleton sequences can be turned into images so that ImageNet-pretrained vision models learn skeleton representations without any skeleton-specific architecture.","keywords":["skeleton representation learning","skeleton-to-image encoding","vision-pretrained models","masked autoencoders","self-supervised learning","cross-format transfer","action recognition"],"falsifier":"Train the identical MAE/DiffMAE pipeline on S2I images but with randomly shuffled joint order or pure Gaussian noise instead of real coordinates; if linear-probe and fine-tune accuracies on NTU-60 collapse to chance, the encoding itself is not preserving the kinematics the method relies on.","tokens_in":23011,"feed_emoji":"🦴","tokens_out":844,"duration_ms":11417,"temperature":0.7,"pith_summary":"Existing skeleton methods train custom networks on fixed joint layouts and cannot easily reuse the powerful models already trained on images. This paper shows that a simple reformatting step—partitioning joints into five body parts, ordering them kinematically, mapping coordinates to RGB, and resizing to 224×224—produces image-like tensors that MAE and DiffMAE can process directly. Initializing from ImageNet weights and continuing with masked reconstruction on these tensors yields competitive or state-of-the-art self-supervised results on NTU-60, NTU-120 and PKU-MMD. Because the representation is format-agnostic, the same model can be pretrained on mixed datasets that use 13-, 20- or 25-joint skeletons and still transfer across them. The practical payoff is that the large visual pretraining ecosystem becomes available for skeleton action recognition without redesigning networks or forcing joint alignment.","feed_headline":"Skeletons become images so vision models learn motion","feed_subtitle":"A simple joint-to-RGB encoding lets MAE and DiffMAE hit competitive skeleton action scores without custom nets","key_machinery":"Skeleton-to-Image Encoding (S2I): the fixed pipeline that turns any T×J×3 skeleton sequence into a 3×224×224 pseudo-image so that an off-the-shelf vision transformer can be pretrained and transferred unchanged.","core_discovery":"Skeleton-to-Image Encoding (S2I) converts a sparse 3-D joint sequence into a dense image by body-part partitioning, top-down kinematic ordering, (x,y,z)→RGB mapping and bilinear resize, enabling ImageNet-pretrained MAE and DiffMAE to perform self-supervised skeleton representation learning for the first time and to support cross-format and multi-dataset pretraining without skeleton-specific architectures.","pith_inferences":["If the same encoding works for generative vision models, skeleton motion synthesis could inherit the same ImageNet priors.","The five-part partition may be replaceable by learned clustering, opening a route to entirely sensor-agnostic skeleton pretraining.","Because the output is an ordinary image, standard vision interpretability tools (attention maps, occlusion) can be applied directly to skeleton decisions."],"forward_implications":["Any future larger vision foundation model can be dropped in for skeleton pretraining without redesigning a graph or transformer backbone.","Heterogeneous skeleton datasets (different joint counts or sensors) can be pooled for a single universal pretraining run.","Multi-modal action systems can share the same visual encoder for both RGB video and skeleton streams.","Cross-dataset transfer no longer requires manual joint subset selection or interpolation.","Masked modeling recipes developed for images become immediately applicable to skeleton sequences."],"fun_headline_variants":["Two-stage segment-then-CLIP match for open-vocab multi-object recognition","OVOR skips retraining by pairing region embeds with CLIP text over any vocab","CLIP image-text similarity after segmentation beats SOTA open-vocab AP","CNN/MLP feature alignment offers CLIP-free path for open-vocabulary recognition","Streamlined OVOR framework recognizes objects via segmentation and embeddings"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That mapping joint coordinates to RGB channels, grouping them into five fixed body parts and resizing to a square image still keeps enough motion structure for a vision transformer’s masked reconstruction to transfer useful ImageNet knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage segment-then-CLIP match for open-vocab multi-object recognition","OVOR skips retraining by pairing region embeds with CLIP text over any vocab","CLIP image-text similarity after segmentation beats SOTA open-vocab AP","CNN/MLP feature alignment offers CLIP-free path for open-vocabulary recognition","Streamlined OVOR framework recognizes objects via segmentation and embeddings"]},"model":"grok-4.5","effort":"low","cost_usd":0.006936,"raw_usage":{"total_tokens":1702,"prompt_tokens":771,"num_sources_used":0,"completion_tokens":100,"cost_in_usd_ticks":69360000,"prompt_tokens_details":{"text_tokens":771,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":831,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":771,"tokens_out":100,"duration_ms":6657,"temperature":1.0,"reasoning_tokens":831,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T14:06:43.384884+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the identical MAE/DiffMAE pipeline on S2I images but with randomly shuffled joint order or pure Gaussian noise instead of real coordinates; if linear-probe and fine-tune accuracies on NTU-60 collapse to chance, the encoding itself is not preserving the kinematics the method relies on.","supporting_citations":[],"review_version":1}