{"id":"1581bf8e-7c74-46b5-9e2e-5656956cdf8c","arxiv_id":"2509.00649","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MV-SSM models the joint spatial sequence with state-space layers plus projective attention and bidirectional token scanning, improving multi-view 3D pose accuracy and generalization to unseen camera setups.","lead":"A new architecture, MV-SSM, applies state-space models (the Mamba family) to multi-view 3D human pose estimation, scanning ordered sequences of body-joint tokens rather than 2D image patches. It reports better accuracy than prior learning-based methods on standard benchmarks, with the largest gains when fewer cameras are used or camera layouts change at test time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4 Row 4's ablation removes GTBS, Mamba, and keypoint-level token encoding simultaneously, so the 5.8 AP25 drop cannot be attributed to the proposed SSM/GTBS; generalization gains are never component-ablated.","rationale":"I agree with the reader's CONDITIONAL verdict, but I identify a different load-bearing concern. The reader's weakest assumption is that the joint scanning order is unspecified, so SSM dependencies could be indexing artifacts. That is a legitimate mechanistic worry, but it is secondary: even an arbitrary fixed order is a learnable inductive bias, and it can be checked by a scan-order ablation. The more damaging issue is that the strongest evidence for the proposed components—Table 4 Row 4—confounds removal of GTBS/Mamba with removal of keypoint-level token encoding. Since the paper's headline is generalization and no ablation examines generalization, the causal role of SSM/GTBS in the headline numbers is not established. This does not make the empirical comparisons false; it makes the central attribution conditional, which matches the reader's verdict. Hence I recommend no change to the verdict, while noting that the specific load-bearing weakness is the confounded ablation plus missing generalization ablations, not primarily the scan order.","tokens_in":15885,"tokens_out":6975,"duration_ms":80279,"concrete_test":"Run a controlled ablation on the same CMU Panoptic split with at least three seeds per configuration: (A) full MV-SSM; (B) remove only GTBS, reverting to default SS2D scanning while keeping Mamba and keypoint-level appearance tokens; (C) remove only Mamba (reproduce Table 4 Row 3); (D) encode only instance-level appearance tokens while keeping GTBS+Mamba (the extra change hidden in Row 4). Report mean +/- std AP25. If B is within noise of A, or if D alone reproduces most of the 93.5 to 87.7 drop, the paper's attribution of the gain to GTBS/SSM fails. As a second step, rerun A vs B on the CMU0(3) cross-camera setting; if the cross-camera generalization gain does not depend on GTBS, the abstract's generalization claim is not mechanistically tied to the proposed components.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the PSS block (SSM + projective attention) and GTBS drive the reported accuracy and generalization gains. Table 4 is the only component-level evidence. Row 4 ('w/o GTBS + Mamba') shows the largest drop (93.5 to 87.7 AP25), but Supplementary Section 3 states that this ablation also 'modified the appearance token to encode only the instance-level information' and that 'multi-view feature fusion also degenerates to only fuse the instance level features.' So Row 4 changes at least three things at once: GTBS is removed, Mamba is removed, and the token/fusion granularity is changed from keypoint-level to instance-level. The 5.8-point drop may therefore be caused by discarding keypoint-level visual tokens, not by removing the proposed scan. Row 3, which removes only Mamba (leaving projective attention), gives just 93.5 to 92.3 AP25; without seeds or error bars, this margin is within possible run-to-run noise. Moreover, all ablations are on the in-domain CMU Panoptic split; none of them tests whether PSS/GTBS contribute to the cross-camera (Table 3) or cross-scene (Table 2) generalization improvements that are the paper's headline. Thus the mechanism is not empirically isolated as the cause of the advertised gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MV-SSM, an end-to-end multi-view 3D human pose estimation framework built around a Projective State Space (PSS) block that combines projective attention with Mamba-style state space modeling, and a Grid Token-guided Bidirectional Scanning (GTBS) scheme. The model is trained on CMU Panoptic and evaluated in-domain and on cross-camera and cross-scene generalization benchmarks (CMU Panoptic variants, Campus, Shelf). The central claim is that MV-SSM generalizes to unseen camera configurations better than the transformer-based MVGFormer, with reported gains of +10.8 AP25 on the three-camera setting, +7.0 AP25 on a challenging arrangement, and +15.3 PCP on Campus Actor 1, while matching in-domain accuracy.","tokens_in":16034,"tokens_out":3795,"duration_ms":46658,"significance":"If the claims hold, this would be a useful contribution: it is an early adaptation of state space models to multi-view 3D pose estimation, and the cross-camera generalization results are practically relevant. The paper includes external benchmark comparisons, a clear architectural description, and ablation experiments on the proposed components. The main weaknesses are that the central mechanism is not cleanly isolated by the ablations, the reported margins are not accompanied by variance estimates, and a key design choice (the joint spatial sequence ordering) is left unspecified. These issues are load-bearing for the paper's strongest claims, but they are addressable with additional experiments and clarification.","major_comments":[{"comment":"The ablation labeled 'w/o GTBS + Mamba' is confounded. The supplementary states that this variant 'modified the appearance token to encode only the instance-level information' and that 'multi-view feature fusion also degenerates to only fuse the instance level features.' Thus Row 4 changes the token granularity and fusion level while also removing GTBS and Mamba. The 93.5→87.7 AP25 drop cannot be attributed to the proposed scanning mechanism. Since Row 3, which removes only Mamba, shows a 1.2 AP25 drop, the paper's statement that GTBS and Mamba are 'integral' is not established by this table. Moreover, all ablations are on the in-domain CMU Panoptic split; no component ablation isolates the contribution of PSS/GTBS to the cross-camera and cross-scene generalization gains in Tables 2 and 3.","section":"§4.2, Table 4 Row 4; Supplementary Section 3"},{"comment":"No repeated runs, error bars, or significance tests are reported. Several comparisons to MVGFormer are small: Table 1 reports 93.5 vs. 92.3 AP25; Table 3 reports CMU1 86.0 vs. 86.8 AP25 (MV-SSM is lower on AP25) and CMU0(6) 94.6 vs. 94.7 AP25. Without variance estimates, these margins may be within run-to-run noise. Please report at least three seeds with mean and standard deviation for the central comparisons.","section":"Tables 1–4"},{"comment":"The 'joint spatial sequence' is not defined. The text says the PSS block 'scans along joint dimension rather than the original time dimension,' but does not specify the ordering of the J joints in the sequence. If the default order is simply the skeleton index order, the sequential dependencies learned by the SSM may be indexing artifacts. Please specify how the joint sequence is constructed (e.g., a fixed kinematic order, a learned order, or a spatial sorting) and ablate at least one alternative ordering to show the mechanism is not an artifact of the chosen sequence.","section":"§3.1, Figure 2"},{"comment":"For cross-dataset evaluations on Campus and Shelf, the paper reports different 3D search-space centers for each dataset ([2000,5000,1000] for Campus, [0,500,800] for Shelf), while claiming the model is evaluated 'w/o fine-tuning.' If these centers are chosen using dataset-specific knowledge, the cross-dataset generalization claim is partially weakened. Please clarify whether these values are fixed a priori, chosen by a common automatic rule, or manually tuned, and discuss sensitivity of the cross-dataset results to this hyperparameter.","section":"Supplementary Section 2.2; Table 2"}],"minor_comments":[{"comment":"The notation switches between continuous-time dynamics (h'(t)) and discrete-time recurrence (ht) without explicit discretization steps; also y(t) appears as Ch(t)+Dx(t) in Eq. (1) and Ch(t) in Eq. (2). Please align notation.","section":"Eqs. (1)–(2)"},{"comment":"The heading 'Cross-arrange' should read 'Cross-arrangement' for consistency.","section":"Table 3"},{"comment":"Row 4 label 'w/o GTBS + Mamba' is ambiguous; the supplementary clarifies that the ablation also changes token granularity. The main text should state this explicitly.","section":"Table 4"},{"comment":"The AP formula uses k = 0 to n−1 with r(k) − r(k+1), which would require an r(n) term. Please clarify the indexing or define n.","section":"Supplementary Eq. (7)"},{"comment":"Typo: 'off-the-self models' should be 'off-the-shelf models.'","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper would benefit from releasing code or model weights, as the architecture details (especially the GTBS scanning order) are nontrivial to reproduce from the text alone. The 'first to adapt visual mamba for multi-view 3D HPE' claim should also be scoped carefully with respect to concurrent work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the new thing: MV-SSM is, as far as I can tell, the first state-space model applied to multi-view 3D human pose estimation. The PSS block and the GTBS scanning are new combinations, and the paper reports real gains on the generalization tests that matter: +10.8 AP25 when dropping to three cameras, +7.0 on a rearranged camera set, and +9.2 average PCP on Campus without fine-tuning. Those are the kind of numbers that would move the field if they hold up. The in-domain result is not dramatic but it matches the current transformer baseline and beats it slightly.\n\nWhere the paper is soft: the main ablation is confounded. Row 4 of Table 4 removes GTBS and Mamba together and also changes the token encoding from keypoint-level to instance-level. So the 5.8-point drop could be from losing the keypoint-level tokens, not from losing the proposed scan. The stress-test note is right about that. Row 3, which removes Mamba alone, gives only a 1.2-point drop (92.3 vs 93.5), and with no error bars or seeds reported that is within run-to-run noise. More importantly, none of the ablations touch generalization; we don't know whether the cross-camera and cross-scene improvements actually come from the PSS/GTBS components or from something else in the pipeline. Also, the 'joint spatial sequence' order is never defined. If it is just the order of the J joints in the skeleton, the SSM may be learning a scan-order artifact. That's a mechanism-level concern, not a refutation. Minor: no code or weights released, and the abstract's 'outperforms SOTA' on Campus is only true among learning-based methods; the paper itself notes that the geometry-based Dong et al. gets 96.3 PCP. That is a fair disclosure, so I don't count it against them much.\n\nAll that said, the central claim is plausible and directionally supported. The paper is honest, the experiments are on standard benchmarks with standard protocols, and the generalization scenario is the right one to test. The appropriate verdict is 'conditional': I'd want to see error bars, a properly decomposed ablation, the definition of the joint ordering, and ideally code, before trusting the mechanism. But this is exactly what peer review is for. I'd send it to a serious referee rather than desk reject.","headline":"MV-SSM is a genuinely new SSM-based multi-view 3D pose estimator with promising cross-camera generalization numbers, but the key ablation is confounded and there are no error bars or code; still worth refereeing.","tokens_in":16754,"tokens_out":2524,"would_cite":true,"duration_ms":27230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MV-SSM claims that treating body joints as a spatial sequence—scanned bidirectionally by a state-space model on top of projective multi-view attention—lets a 3D pose estimator transfer to camera setups it never trained on, beating the trans","keywords":["3D human pose estimation","multi-view geometry","state space models","Mamba","projective attention","cross-dataset generalization","camera-arrangement generalization"],"falsifier":"Retrain MV-SSM twice with identical settings except that the joint scan order in GTBS is the fixed skeleton-index order in one run and a fixed random permutation in the other; if AP25 on the CMU Panoptic test split does not fall in the random-order run, the sequential state-space modeling is not what produces the reported gains and the contribution reduces to any-order token mixing.","tokens_in":15571,"feed_emoji":"🦴","tokens_out":9134,"duration_ms":100776,"temperature":0.7,"pith_summary":"MV-SSM is an attempt to show that state space models—sequence models originally built for text and audio—can serve as the core of multi-view 3D human pose estimation. The paper's idea is to treat the body's joints as an ordered sequence, not over time but over the joint dimension, and to scan the multi-view image features around each projected keypoint bidirectionally. Why this matters: if the reported numbers hold, a pose estimator trained on one five-camera rig keeps working—and often improves over attention-based transformers—when cameras are removed, rearranged, or replaced by different datasets, a failure mode of earlier methods. The evidence is a set of in-domain and generalization results on standard benchmarks, plus ablations showing that removing the state-space component costs 5.8 points of AP25.","feed_headline":"Scanning joints as a sequence sharpens 3D pose from multi-view cameras","feed_subtitle":"A state-space block over body joints beats transformer baselines on unseen camera counts and layouts.","key_machinery":"The PSS block is the architectural unit that carries the paper's argument. Its projective-attention branch projects the current 3D keypoint estimate into each view, samples deformable points around the projected anchor, and aggregates local image features; its state-space branch takes the J joint tokens of each person and scans them as a sequence, not along time but along the joint dimension. GTBS is the scan variant: instead of scanning all image patches, it scans the sampled projection tokens bidirectionally, so the SSM sees relevant local tokens from every view while ignoring background. Stacked PSS blocks refine the keypoint tokens and output 2D residual offsets; a differentiable triangu","core_discovery":"The paper's central claim is that a multi-view 3D pose estimator can be built by modeling the ordered list of body joints as a spatial sequence at two levels: the visual-feature level (sampled multi-view image tokens) and the person-keypoint level (explicit 3D joint positions). The PSS block does this by coupling projective attention—which gathers image features around each joint's projected location in each view—with a Mamba-style state-space layer that scans the joint tokens bidirectionally (GTBS). The paper reports that this combination reaches 93.5 AP25 and 15.7 mm MPJPE on CMU Panoptic, above MVGFormer's 92.3/16.0, and that the margin grows when cameras are removed, rearranged, or the m","pith_inferences":["A testable next step the paper leaves open is to define the joint scan order kinematically (e.g., along the skeleton chain or by limb hierarchy) and compare it with the dataset's index order; this would separate true geometric sequencing from token-mixing artifacts.","Because GTBS operates only on projected anchor tokens, the same PSS block could be reused as a refinement head in other differentiable multi-view geometric pipelines, including object keypoint or camera-pose estimation, without retraining the whole backbone.","If the cross-arrangement gains come mainly from projective attention's local sampling, then varying the sampling radius and token count while freezing the SSM would reveal how much of the generalization is geometric versus sequential.","The paper's own failure case—a person visible in only one camera—suggests a straightforward extension: feed the PSS block with synthesized or hallucinated missing-view tokens so the state-space layer can propagate evidence across joints even when one view is absent."],"forward_implications":["If the reported results hold, multi-view 3D pose estimation no longer needs to be framed exclusively as an attention problem: a linear-time state-space scan over joint tokens can carry the inter-joint modeling and matches or beats the transformer baseline in-domain.","A model trained on one five-camera rig transfers to three, four, six, and seven cameras and to different camera IDs without fine-tuning, so practitioners could train once and deploy on differently sized capture volumes.","The ablation numbers make the SSM component load-bearing: dropping GTBS and Mamba together lowers AP25 from 93.5 to 87.7 and raises MPJPE from 15.7 mm to 18.6 mm.","The explicit 3D keypoint branch matters more than the visual-token branch for accuracy (91.8 vs 92.5 AP25 when each is removed), pointing to geometry-anchored refinement as the main driver.","The biggest gain over MVGFormer appears with only three cameras (+10.8 AP25), suggesting the sequence model is most valuable when each view is information-poor."],"supporting_citations":[{"why":"Supplies the selective state-space (Mamba) sequence layer that MV-SSM adapts to scan joint tokens.","marker":"[15]"},{"why":"Provides the S4 structured-state-space equations and discretization that the paper's preliminaries rely on.","marker":"[16]"},{"why":"Contributes the differentiable algebraic triangulation module used to turn refined 2D estimates and confidence scores into 3D keypoints.","marker":"[20]"},{"why":"Supplies the CMU Panoptic training data and the in-domain benchmark used for the main comparisons.","marker":"[21]"},{"why":"The transformer baseline MVGFormer that MV-SSM compares against; also the source of token initialization, NMS, and training-split choices.","marker":"[27]"},{"why":"Contributes the SS2D bidirectional image scanning that GTBS adapts from full-image to grid-token-guided scanning.","marker":"[29]"},{"why":"The VoxelPose baseline and the standard evaluation configuration that the paper reuses for fair comparison.","marker":"[36]"},{"why":"The MvP baseline; source of projective attention, hierarchical token scheme, confidence filtering, and direct multi-view supervision.","marker":"[44]"},{"why":"Establishes the cross-camera and cross-arrangement generalization protocol used in Table 3.","marker":"[2]"},{"why":"Provides the Campus and Shelf benchmarks on which cross-dataset generalization is measured.","marker":"[3]"}],"fun_headline_variants":["State-space joint scan generalizes to unseen multi-view setups","Joint sequence model outperforms transformers on new camera layouts","Modeling joint order as state space improves multi-view 3D pose","Bidirectional state-space over joints generalizes in multi-view pose","MV-SSM sequence-modeling joints beats transformers on unseen cameras"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the joint tokens form a meaningful spatial sequence with a definite order; the paper never specifies that order, so the state-space layer's sequential dependencies could be artifacts of the skeleton's fixed index rather than learned geometric structure.","fun_headline_variants_meta":{"raw":{"variants":["State-space joint scan generalizes to unseen multi-view setups","Joint sequence model outperforms transformers on new camera layouts","Modeling joint order as state space improves multi-view 3D pose","Bidirectional state-space over joints generalizes in multi-view pose","MV-SSM sequence-modeling joints beats transformers on unseen cameras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3866,"prompt_tokens":822,"completion_tokens":3044,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2959}},"tokens_in":566,"tokens_out":3044,"duration_ms":26305,"temperature":1.0,"reasoning_tokens":2959,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:23:39.232717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain MV-SSM twice with identical settings except that the joint scan order in GTBS is the fixed skeleton-index order in one run and a fixed random permutation in the other; if AP25 on the CMU Panoptic test split does not fall in the random-order run, the sequential state-space modeling is not what produces the reported gains and the contribution reduces to any-order token mixing.","supporting_citations":[{"cited_title":"Mamba: Linear-time sequence mod- eling with selective state spaces","cited_arxiv_id":null,"evidence_quote":"Supplies the selective state-space (Mamba) sequence layer that MV-SSM adapts to scan joint tokens."},{"cited_title":"Efficiently mod- eling long sequences with structured state spaces","cited_arxiv_id":null,"evidence_quote":"Provides the S4 structured-state-space equations and discretization that the paper's preliminaries rely on."},{"cited_title":"Learnable triangulation of human pose","cited_arxiv_id":null,"evidence_quote":"Contributes the differentiable algebraic triangulation module used to turn refined 2D estimates and confidence scores into 3D keypoints."},{"cited_title":"Panoptic studio: A massively multiview system for social motion capture","cited_arxiv_id":null,"evidence_quote":"Supplies the CMU Panoptic training data and the in-domain benchmark used for the main comparisons."},{"cited_title":"Multiple view geometry transform- ers for 3d human pose estimation","cited_arxiv_id":null,"evidence_quote":"The transformer baseline MVGFormer that MV-SSM compares against; also the source of token initialization, NMS, and training-split choices."},{"cited_title":"Vmamba: Visual state space model","cited_arxiv_id":null,"evidence_quote":"Contributes the SS2D bidirectional image scanning that GTBS adapts from full-image to grid-token-guided scanning."},{"cited_title":"V oxelpose: Towards multi-camera 3d human pose estimation in wild en- vironment","cited_arxiv_id":null,"evidence_quote":"The VoxelPose baseline and the standard evaluation configuration that the paper reuses for fair comparison."},{"cited_title":"Direct multi-view multi-person 3d pose estimation","cited_arxiv_id":null,"evidence_quote":"The MvP baseline; source of projective attention, hierarchical token scheme, confidence filtering, and direct multi-view supervision."},{"cited_title":"Generalizable human pose triangulation","cited_arxiv_id":null,"evidence_quote":"Establishes the cross-camera and cross-arrangement generalization protocol used in Table 3."},{"cited_title":"3d pictorial structures for multiple human pose estimation","cited_arxiv_id":null,"evidence_quote":"Provides the Campus and Shelf benchmarks on which cross-dataset generalization is measured."}],"review_version":1}