{"id":"791d2aa3-502f-4580-9ad5-f4018fcbce22","arxiv_id":"2412.18883","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MotionMap represents multimodality in human pose forecasting as a heatmap over a learned latent space, enabling sample-efficient, ranked, and controllable prediction of multiple future motions.","lead":"MotionMap encodes possible future human poses as peaks on a heatmap, letting a model predict several likely motions at once from one observation. It also ranks those futures by confidence and can recall rare motions that other methods average away.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Codebook averaging can collapse distinct transitions into one mean forecast per quantized 2D cell, undermining the claims of variable mode count and transition recall; the paper provides no intra-cell diversity analysis.","rationale":"The reader's weakest_assumption identifies the bounded-by-training-set premise. I agree this is a real limitation, but I find a more immediate, internal gap: the codebook averaging in Sec. 4.2.1. The strongest_claim requires that distinct modes be representable and that observed transitions be recalled. Yet the mapping from heatmap cell to latent is a mean over all training latents in that cell. The paper gives no evidence that cells are pure (single transition) or that the mean latent decodes to a realistic future. If a cell mixes two distinct motions, the method outputs a single averaged forecast, so the effective number of modes is smaller than the ground-truth count, and 'recall' reduces to hitting the correct cell rather than generating the correct transitions. This is consistent with the standard-split results in Tables 2/3, where MotionMap's MMADE/MMFDE gains are absent. The proposed test (intra-cell FDE analysis) settles the issue computationally using the released code. I therefore keep the reader's conditional verdict, adding this check as an explicit condition.","tokens_in":13875,"tokens_out":13225,"duration_ms":117179,"concrete_test":"Using the released code and trained codebook from Sec. 4.2.1, collect all test-set futures Y whose latent zy quantizes to each occupied cell hy. For every cell with at least two members, decode the codebook's mean latent to a forecast sequence and compute (i) the maximum pairwise FDE between group members and (ii) the FDE between the decoded mean and each member. If any group has max pairwise FDE > 0.3 m while the decoded mean is > 0.3 m from at least one member, the cell is mixing distinct motions. Report the fraction of occupied cells that are mixed; also flag any cell whose members carry different action labels. If that fraction is non-negligible, the 'variable number of modes' and 'recalls transitions' claims are not supported at the motion level.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.2.1, all latent codes zy are t-SNE projected, scaled, and rounded to heatmap cells hy. The codebook B then stores, for each cell, the mean of every zy that quantizes to that cell (B = {hy : zy}). Fine-tuning (Sec. 4.3) trains the decoder D to consume this mean latent at inference (Sec. 4.4), so each local maximum in the predicted MotionMap decodes to a single averaged future.\n\nThis creates a load-bearing gap in the argument for the central claim. If two semantically distinct transitions for the same observation quantize to the same cell, MotionMap cannot represent both; the cell's decoded forecast is an average that may be unlike either transition. The paper does not report intra-cell statistics: how many distinct futures per occupied cell, how much variance in the decoded poses, or whether different action labels share cells. With a 128x128 grid (Sec. 4.2.1) and rich datasets like AMASS, cell collisions are plausible. The paper's own Limitations paragraph (Sec. 5.5) concedes 'lack of fine-grained motion prediction within a mode', but the issue is stronger: distinct modes, not just fine variations, can be merged by the averaging step.\n\nThe quantitative evidence is also consistent with this concern. The headline 'recall' evaluation (Sec. 5.5) constructs test ground truths from training labels and encodes them through the same t-SNE/codebook pipeline, so it measures cell activation rather than faithful transition recall. Under the standard test-split protocol (Tables 2 and 3), MotionMap does not beat BeLFusion or CoMusion on MMADE/MMFDE, and its Diversity is moderate. Without an analysis of codebook-cell purity, the claims of 'capturing a variable number of modes' and 'recalling transitions' remain unverified at the granularity of actual motions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MotionMap, a heatmap-based representation for multimodal human pose forecasting. Future pose sequences in the training set are encoded by a GRU encoder, embedded into two dimensions via t-SNE, and quantized onto a 2D grid; a codebook stores, for each occupied cell, the mean latent of all sequences falling into that cell. A heatmap predictor H is trained to map an observed pose sequence to this distribution, and at inference local maxima are selected and decoded by the autoencoder's decoder, giving a variable number of ranked, deterministic forecasts. The paper also proposes a motion-transfer procedure for building multimodal ground truths from the last three frames with skeleton-size normalization, and introduces an evaluation protocol in which test ground truths are the training-set transitions closest to each test sample. Experiments are reported on Human3.6M and AMASS, including the new protocol (Table 1) and the standard test-split protocol (Tables 2 and 3), together with qualitative results on controllability, uncertainty, and rare-mode coverage.","tokens_in":14177,"tokens_out":3943,"duration_ms":39693,"significance":"The paper addresses a real limitation of sampling-based multimodal predictors: mode coverage usually requires many stochastic samples, while MotionMap obtains modes in one feed-forward pass and additionally supplies per-mode confidence. The representation also enables controllability and heteroscedastic uncertainty, and the authors release code and a project page, which supports reproducibility. If the central claims are substantiated, MotionMap would be a useful alternative to diffusion- or flow-based multimodal forecasting. However, the paper's quantitative case currently rests on a nonstandard evaluation protocol that is partly aligned with the method's own training target, and the codebook-averaging mechanism raises an unresolved concern about whether distinct modes are preserved after quantization. The qualitative results are suggestive but do not, by themselves, establish the sample-efficiency and recall claims at the level needed for a journal publication.","major_comments":[{"comment":"The codebook construction collapses all latents that quantize to the same heatmap cell into a single mean latent, B = {hy : zy}. Because the decoder is fine-tuned to consume this mean latent and inference decodes each local maximum through the codebook, two semantically distinct futures for the same observation that land in the same cell cannot both be represented; the decoded forecast will be an average that may match neither. This directly affects the claims of variable mode count, transition recall, and diversity. The manuscript provides no intra-cell statistics, such as the number of distinct futures per occupied cell, the dispersion of the zy vectors within a cell, the decoded-pose variance for a cell, or the action-label composition of cells. With a 128x128 grid and hundreds of thousands of sequences in AMASS, such collisions are plausible and should be quantified. Please report these statistics and, if collisions are substantial, consider a codebook with multiple prototypes per cell or a higher effective resolution.","section":"Sec. 4.2.1, Sec. 4.3, Sec. 4.4"},{"comment":"The headline quantitative evaluation is partly circular. The test multimodal ground truths in Table 1 are obtained by selecting, for each test sample, the closest training-label transitions, using the same motion-transfer and skeleton-scaling procedure that is used to construct the training heatmaps. The model is therefore evaluated on how well it recalls targets that were generated from the training set through the same pipeline that defines its supervision. This favors MotionMap by construction. The standard-split results (Tables 2 and 3) do not consistently support the claimed advantage: on Human3.6M, MotionMap has ADE 0.491 and MMADE 0.505, while BeLFusion has 0.472 and 0.497 and CoMusion has 0.460 and 0.505; on AMASS, MotionMap has ADE 0.624 and MMADE 0.643, while CoMusion has 0.601 and 0.629. No error bars or significance tests are reported. Please provide multi-seed error bars, report both protocols with the same number of samples, and state explicitly which conclusions hold under the strict test-split protocol.","section":"Sec. 5.5, Table 1, Tables 2 and 3"},{"comment":"The paper's well-posedness argument relies on the assumption that possible futures for any input are bounded by the transitions present in the training set, and that unseen test futures are translatable training transitions. This premise is load-bearing: the codebook contains only training latents, so a genuinely novel test transition cannot be produced by construction. The paper notes in Fig. 14 that the AMASS train/test splits can be highly imbalanced, but it does not test how often the premise fails or how performance degrades when it does. A concrete test would be to split test sequences by their nearest-neighbor distance to the training distribution, or to evaluate on held-out action categories, and report ADE/MMADE or recall as a function of that distance. Without such an analysis, the 'well-posed' framing is an assumption rather than a validated property.","section":"Sec. 1, Sec. 5.5, Fig. 14"}],"minor_comments":[{"comment":"The sentence 'we use zy instead of zy' is ambiguous; it should read 'we use the averaged codebook latent z̄y in place of the encoder latent zy'.","section":"Sec. 4.3"},{"comment":"The Diversity metric is reported with an upward arrow in Table 1 but with a downward arrow in Tables 2 and 3; the desired direction should be stated once and used consistently.","section":"Table 1 vs Tables 2 and 3"},{"comment":"The choice of grid resolution m and the t-SNE perplexity/random seed are not reported; these choices influence cell collisions and the codebook size, so they should be documented for reproducibility.","section":"Sec. 4.2.1"},{"comment":"Figure 2 is dense and the role of the codebook during fine-tuning versus inference is hard to follow; a simplified schematic with explicit tensor shapes would improve readability.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment is fair. My main concerns are the codebook-averaging collapse and the partially circular evaluation protocol; both are fixable with additional analysis and experiments, so I do not recommend rejection. The paper has a clear contribution and public code, but the central claims of variable mode count and transition recall need stronger evidence. I would also encourage the editor to ask for error bars, since several comparative numbers are close."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the MotionMap paper. Worth your time if you work on pose forecasting or multimodal prediction. The core idea is new: instead of sampling many futures from a generative model, they project all future latents into a 2D t-SNE space, quantize to a heatmap, and train a predictor to output that heatmap. Local maxima are decoded through a codebook back to latent codes, giving a variable number of modes with confidence. That's a real representational advance. It also lets them rank modes, estimate input-dependent uncertainty, and control the forecast by selecting regions of the heatmap. They share encoder/decoder with BeLFusion, so the comparison with that baseline is clean.\n\nThe motion-transfer ground-truth definition (last three frames plus skeleton normalization) is a sensible fix to a known weakness in prior multimodal ground truth construction.\n\nThe soft spots are real. The biggest is the codebook averaging. Section 4.2.1 stores, per heatmap cell, the mean of all latents that quantize to that cell. If two semantically distinct transitions land in the same cell, the decoded output is an average that may match neither. With a 128x128 grid and AMASS, collisions are plausible. The paper's limitations paragraph concedes lack of fine-grained motion within a mode, but the issue is stronger: distinct modes can be merged entirely. They don't report intra-cell variance or any cell-purity statistics. The central claim of \"variable number of modes\" and \"recalling transitions\" is unverified at the granularity of actual motions.\n\nSecond, the headline quantitative result (Table 1) is partly circular. The method is trained to predict heatmaps built from their own motion-transfer-based multimodal ground truths, then evaluated on recalling those same transitions. Under the standard test-split protocol (Tables 2 and 3), MotionMap does not beat BeLFusion on MMADE on either dataset and only sometimes wins on MMFDE. No error bars anywhere. The rare-mode claim is qualitative only.\n\nThat said, the paper is honest about its main limitation, and the representation is worth exploring. The stress-test concern about cell collisions holds up on reading; it's not a manufactured flaw. But the method is clearly specified, the code is promised, and the qualitative results support the value of a heatmap prior.\n\nWho's this for: researchers working on multimodal prediction, especially safety-critical downstream tasks where sample efficiency and mode confidence matter. It deserves a serious referee. I'd suggest major revision: add intra-cell diversity analysis, report standard-split results with error bars, and temper the transition-recall claim to match the evaluation protocol.","headline":"A genuinely new heatmap-based representation for multimodal pose forecasting, undercut by codebook averaging and an evaluation protocol the method is trained on.","tokens_in":14788,"tokens_out":3399,"would_cite":false,"duration_ms":26582,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MotionMap represents all likely human pose futures as peaks on a heatmap, found in a single forward pass.","keywords":["human pose forecasting","multimodal motion prediction","heatmap representation","sample efficiency","uncertainty estimation","codebook","motion transfer","3D human pose datasets"],"falsifier":"Construct a test set in which the true future contains a motion transition that never appears in the training data (for instance, a direction change or action combination that was deliberately held out). Running the full pipeline on this test set and observing that the true future falls outside all predicted heatmap peaks, while a diffusion baseline with sufficient samples covers it, would falsify the central claim that training-set transitions suffice for well-posed forecasting.","tokens_in":13686,"feed_emoji":"🧍","tokens_out":5602,"duration_ms":49177,"temperature":0.7,"pith_summary":"The paper argues that human pose forecasting can be made well-posed by reframing it as learning motion transitions that already exist in the training data and translating them to unseen test sequences. On top of that reframing, it introduces MotionMap, a heatmap representation in which each local maximum is a distinct possible future motion for a given observation, with a confidence attached. The central claim is that one forward pass over this heatmap, followed by a codebook lookup, replaces the large-scale random sampling of diffusion and VAE baselines while covering more modes, including rare ones. A secondary claim is that the representation naturally supports uncertainty estimation, ranking of forecasts, and user-controllable mode selection.","feed_headline":"One heatmap maps out every likely future pose","feed_subtitle":"MotionMap reads possible future motions as peaks on a learned map, with confidence per mode and no random sampling.","key_machinery":"The central object is the MotionMap: a 2D heatmap built by t-SNE projection and quantization of autoencoder latents over all future pose sequences in the dataset. Each heatmap cell stores, through a codebook, the mean latent of all futures that fell there; a Gaussian is plotted at each cell for each multimodal ground truth. The heatmap predictor H, a GRU encoder followed by 1x1 convolutions, is trained with weighted binary cross-entropy to reproduce this map from the last three observed frames. At inference, local maxima are found deterministically, and the codebook supplies the missing latent zy that the decoder needs, so no sampling is required.","core_discovery":"The paper's central discovery is that multimodality in human pose forecasting can be encoded as a spatial distribution over a learned two-dimensional latent space of future motions, rather than by sampling many stochastic predictions. The authors first redefine multimodal ground truth using motion transfer between skeletons and the last three observed frames, so that similar motions from different individuals count as alternatives. They then project latent codes of all futures onto a 2D map, quantize it into a heatmap, and train a GRU-based heatmap predictor per input. At test time, local maxima of the predicted heatmap index a codebook of average latents, each decoded into a full future pose. Because the number of peaks is not fixed, the method outputs a variable number of modes, ranks them by confidence, and recovers rare transitions that averaging would erase. The authors show on Human3.6M and AMASS that MotionMap achieves strong multimodal accuracy with a small fixed number of forecasts.","pith_inferences":["A natural extension is to treat the heatmap itself as a shared interface, letting text descriptions or action labels query modes directly, which could unify language-conditioned and pose-conditioned motion generation.","The well-posedness reframing suggests a testable protocol: measure how performance degrades when test actions are drawn from a distribution disjoint from training; if the transition-learning assumption holds, retraining on a superset should help more than additional sampling at test time.","Since the codebook stores mean latents per cell, the method could be adapted to output per-cell variance or multiple latents per cell, enabling fine-grained intra-mode variations without losing the heatmap's mode structure.","The t-SNE and quantization pipeline is data-dependent; a learned, amortized discrete embedding might preserve rare modes better than the fixed projection, a claim the paper does not test."],"forward_implications":["The number of forecasts is decided by the input itself through the number of heatmap peaks, so both common and rare modes are represented instead of being suppressed by averaging.","Mode confidences turn the forecast set into a ranking, allowing practitioners to pick the most likely future or deliberately select a rare alternative.","Because the heatmap encodes transitions seen in training, sample efficiency at a fixed budget (e.g., 5–7 forecasts) improves over diffusion and anchor-based baselines on both datasets.","The same representation decomposes uncertainty into mode confidence and per-joint heteroscedastic uncertainty, and enables controllable generation using action labels or nearby latents."],"supporting_citations":[{"why":"Supplies the autoencoder encoder/decoder architecture and the main diffusion baseline that MotionMap is compared against, as well as the standard dataset protocol.","marker":"[1]"},{"why":"Provides the baseline sampling strategy and the evaluation metrics (ADE, FDE, MMADE, MMFDE) used throughout the paper.","marker":"[3]"},{"why":"The homoscedastic uncertainty work that MotionMap's heteroscedastic uncertainty decomposition is contrasted with in Section 5.2.","marker":"[16]"},{"why":"t-SNE is the dimensionality-reduction method used to project latent codes into the two-dimensional heatmap space.","marker":"[36]"},{"why":"The OpenTSNE implementation, which provides the transform function used to project and interpolate latents.","marker":"[37]"},{"why":"The AMASS dataset, one of the two benchmarks used to construct multimodal ground truths and evaluate forecasting.","marker":"[38]"},{"why":"The Human3.6M dataset, the other benchmark providing training and test subjects and actions for evaluation.","marker":"[39]"}],"fun_headline_variants":["Heatmap peaks map out every future pose","One heatmap forecasts multiple human futures","Pose forecasting: heatmap captures rare modes","No sampling, just heatmap for pose multimodality","MotionMap: variable pose modes from one heatmap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes every test-time future is a translation of a motion transition that already appears in the training set, so a genuinely novel transition at test time cannot be predicted or evaluated under its protocol.","fun_headline_variants_meta":{"raw":{"variants":["Heatmap peaks map out every future pose","One heatmap forecasts multiple human futures","Pose forecasting: heatmap captures rare modes","No sampling, just heatmap for pose multimodality","MotionMap: variable pose modes from one heatmap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1522,"prompt_tokens":978,"completion_tokens":544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":594,"tokens_out":544,"duration_ms":5588,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:21:27.922601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set in which the true future contains a motion transition that never appears in the training data (for instance, a direction change or action combination that was deliberately held out). Running the full pipeline on this test set and observing that the true future falls outside all predicted heatmap peaks, while a diffusion baseline with sufficient samples covers it, would falsify the central claim that training-set transitions suffice for well-posed forecasting.","supporting_citations":[{"cited_title":"Belfusion: Latent diffusion for behavior-driven human motion prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the autoencoder encoder/decoder architecture and the main diffusion baseline that MotionMap is compared against, as well as the standard dataset protocol."},{"cited_title":"Dlow: Diversifying latent flows for diverse human motion prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline sampling strategy and the evaluation metrics (ADE, FDE, MMADE, MMFDE) used throughout the paper."},{"cited_title":"Toward reliable hu- man pose forecasting with uncertainty,","cited_arxiv_id":null,"evidence_quote":"The homoscedastic uncertainty work that MotionMap's heteroscedastic uncertainty decomposition is contrasted with in Section 5.2."},{"cited_title":"Visualizing data using t-sne,","cited_arxiv_id":null,"evidence_quote":"t-SNE is the dimensionality-reduction method used to project latent codes into the two-dimensional heatmap space."},{"cited_title":"Available: https://opentsne.readthedocs.io/ en/stable/ 5,","cited_arxiv_id":null,"evidence_quote":"The OpenTSNE implementation, which provides the transform function used to project and interpolate latents."},{"cited_title":"Amass: Archive of motion capture as surface shapes,","cited_arxiv_id":null,"evidence_quote":"The AMASS dataset, one of the two benchmarks used to construct multimodal ground truths and evaluate forecasting."},{"cited_title":"Human3. 6m: Large scale datasets and predictive meth- ods for 3d human sensing in natural environments,","cited_arxiv_id":null,"evidence_quote":"The Human3.6M dataset, the other benchmark providing training and test subjects and actions for evaluation."}],"review_version":1}