{"id":"4b37cdf6-c413-421e-b05f-a4bf12224a30","arxiv_id":"2507.02565","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dual-branch optimization fitting body motion and per-video 3D Gaussian appearance jointly, guided by a diffusion proxemics prior, improves close-interaction reconstruction from monocular video.","lead":"This paper reconstructs two people interacting closely in ordinary videos by combining a learned proxemics model with a rendering-based appearance branch, so ambiguous and occluded frames can still be resolved by how the people look. The approach reports gains over prior methods on interaction benchmarks and adds a new in-the-wild interaction dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The per-video appearance branch may memorize RGB independent of pose, so the claimed depth-disambiguation from appearance in Eq. (6) is not yet shown to be a real pose constraint.","rationale":"The reader's conditional verdict already identifies the same weakest assumption: the per-video appearance branch has enough capacity to fit RGB regardless of pose, which threatens the central appearance-reasoning contribution. I agree with that assessment. The paper's other contributions—the diffusion proxemics prior, physics losses, and the WildCHI dataset—are supported by ablations and would not collapse even if the appearance branch were overfitting. However, the headline claim that appearance resolves depth ambiguity in close interactions is exactly the unsupported piece, and the provided evidence (one small MPJPE gap in Table 3 plus a qualitative figure) is not sufficient to rule out overfitting. Conditional acceptance remains the right disposition: the method is coherent and empirically competitive, but the central mechanistic claim needs a discriminating experiment. If the proposed depth-swap test shows Lapp cannot distinguish correct from reversed depth order, the paper's novelty would reduce to an incremental optimizer variant, and the verdict should be reconsidered. If it shows clear sensitivity, the appearance contribution is validated. The lack of error bars and missing code/checkpoints further support the conditional status but do not change the verdict on their own.","tokens_in":13876,"tokens_out":3065,"duration_ms":38782,"concrete_test":"Select a test video from Hi4D. After full optimization converges, freeze the motion branch and record Lapp. Then swap the two subjects' depth ordering (e.g., negate their relative z-translation while preserving 2D projections by adjusting the camera, or swap identity assignments) and re-optimize only the appearance branch under the same Lapp and Lreg. If final Lapp for the depth-reversed pose is comparable to the original, Eq. (6) is insensitive to depth ordering and the appearance branch is fitting the image rather than constraining pose. A complementary check: optimize appearance on the first 64 frames with frozen pose and evaluate Lapp on held-out frames; poor held-out rendering would indicate memorization rather than a pose-usable appearance model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 builds the central appearance contribution on Eq. (6): Lapp = Lrgb + Lssim + Llpips between the splatted two-person Gaussians and the input RGB. The appearance branch is a per-video optimizable latent tensor plus a U-Net decoding it into Gaussian UV maps. Nothing in Eq. (9) (Loffset, Lscale) or the full objective prevents this branch from fitting the observed image with pose-independent opacity, color, or offset adjustments; the U-Net is re-optimized for every video, so it can absorb photometric error without moving the SMPL body. The only quantitative evidence for pose sensitivity is a 1.6 mm MPJPE difference in Table 3 (Ours w/o Appearance 60.68 vs Ours 59.06), which is smaller than the gap between other configurations in the same table (Ours w/o Physics reaches 57.01 MPJPE) and is reported without error bars. Because the claim that appearance provides depth-ordinal cues is what differentiates this method from BUDDI and CloseInt, this is the most load-bearing assumption. The WildCHI pseudo-labeling circularity is secondary, since the method's own evaluation is not trained on WildCHI.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dual-branch optimization framework for reconstructing two-person close interactions from monocular in-the-wild video. The motion branch is a diffusion model trained on interaction datasets to encode pose and proxemics priors; the appearance branch uses per-video optimizable tensors decoded by a U-Net into Gaussian UV maps, which are splatted and compared with the input RGB image through L1, SSIM, and LPIPS losses. The full objective combines appearance, keypoint reprojection, penetration, smoothness, and regularization terms, and both branches are optimized jointly. The authors also introduce WildCHI, a dataset of 100 TikTok videos with pseudo ground-truth annotations generated by the proposed method, and show that training CloseInt on this dataset improves its performance. Experiments on Hi4D and 3DPW report state-of-the-art MPJPE values (59.1 and 64.5, respectively).","tokens_in":14101,"tokens_out":4732,"duration_ms":57525,"significance":"If the main claim holds, the paper makes a useful contribution: it provides evidence that differentiable rendering of a coarse appearance model can help disambiguate depth ordering in close human interactions, a setting where keypoints and masks are unreliable. The framework is clearly described and compared against relevant recent baselines (BUDDI, CloseInt, BEV, GroupRec), and the ablation study covers the main components. The promise of code and data is also a strength. The main caveat is that the central appearance-pose coupling is validated only indirectly, and the WildCHI pseudo-labeling is self-referential without independent verification. As a result, the significance depends on additional controlled experiments that are within reach.","major_comments":[{"comment":"The central claim that the appearance branch constrains body pose through rendering is supported only by a 1.6 mm MPJPE gap (Ours w/o Appearance 60.68 vs Ours 59.06), which is smaller than other ablation differences in the same table (e.g., Ours w/o Physics reaches 57.01) and is reported from single runs without error bars. Since the appearance branch is re-optimized per video and can adjust Gaussian opacity, color, and offsets independently of the SMPL pose, it could in principle fit the RGB image without moving the body. Please add controlled experiments that separate pose sensitivity from photometric overfitting: (i) optimize the appearance branch with fixed ground-truth poses and with fixed perturbed poses and report the final Lapp values; (ii) run the full optimization with the appearance branch frozen and compare against the full model; (iii) report means and standard deviations over at least three random seeds for Table 3. Without such evidence, the 'appearance resolves depth ambiguity' contribution is not established.","section":"Section 3.3, Eq. (6), Table 3"},{"comment":"The pseudo ground-truth annotations in WildCHI are generated by the proposed method itself, and the only validation that the dataset improves regression-based models is training CloseInt on those pseudo-GT labels. This is self-referential unless the pseudo-GT accuracy is independently quantified. Please provide (i) quantitative agreement of the pseudo-GT with manually corrected annotations on a random subset, (ii) an evaluation that transfers a model trained on WildCHI to Hi4D/3DPW, or (iii) a comparison with pseudo-GT generated by an independent method such as BUDDI. This is needed to support the dataset contribution, although it is not necessary for the validity of the Hi4D/3DPW comparisons.","section":"Section 4 and Table 2 (WildCHI)"},{"comment":"The overall objective is a weighted sum of at least six loss terms, but the loss weights are not reported. The paper also does not specify the diffusion training schedule, the mask strategy probabilities, the latent tensor dimensionality, or the U-Net architecture. Because the method is optimization-based and the reported numbers depend directly on these choices, the experiments are not reproducible from the text alone. Please include the full configuration (weights for Lapp, Lreproj, Lpen, Lsmooth, Lreg, optimizer schedules, and all hyperparameters) or point to released code with the exact settings used to produce Tables 2 and 3.","section":"Section 3.3, Eq. (10), and Section 5.4"}],"minor_comments":[{"comment":"The row label 'CLLIF' should be 'CLIFF' to match Reference [31].","section":"Table 2"},{"comment":"The phrase 'proxemixs prior' contains a typo and should read 'proxemics prior'.","section":"Section 5.1"},{"comment":"The 'interaction error' metric is mentioned but never defined; unless it is taken from CloseInt [16], please provide the exact formula or a precise reference.","section":"Section 5.2"},{"comment":"The word 'differentialable' is a typo, and the notation na and nb for face normals should be defined explicitly.","section":"Section 3.3, Eq. (7)"},{"comment":"The identity channel d in the Gaussian UV map is used to restrict keypoint supervision to the rendered region of each individual, but the mechanism is only described vaguely; please clarify how the two-person Gaussian splatting attributes pixels to a specific person's rendered region.","section":"Section 3.3, appearance branch"}],"recommendation":"major_revision","confidential_remarks":"The paper is generally well written and the benchmark results are strong if the appearance ablation is meaningful. My main technical concern is the latent overfitting risk in the appearance branch, which should be addressed with the controlled experiments described in the major comments. The WildCHI pseudo-GT circularity is a secondary issue that affects the dataset contribution but not the main Hi4D/3DPW comparison. The authors' own prior work CloseInt is used as a baseline and appears to be treated fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, this is honest, well-crafted empirical work. The dual-branch combination of a diffusion proxemics prior and a per-video 3D Gaussian appearance branch is genuinely new, and the WildCHI dataset is a real resource even if pseudo-labeled. Second, the load-bearing claim—that the appearance branch constrains pose through the rendering loss in Eq. (6)—is not established. I think the stress-test note is right: the per-video U-Net plus optimizable tensors have enough capacity to fit the RGB image without moving the SMPL body, and the only quantitative support is a 1.6 mm MPJPE gap in Table 3, reported without error bars. Worse, the same table shows Ours w/o Physics beating the full method on MPJPE (57.01 vs 59.06), so the appearance branch's contribution looks fragile relative to other components.\n\nWhat the paper does well: the proxemics diffusion prior with mask strategies is a sensible way to inject interaction priors into optimization, the dual-branch optimization is clearly described, and the ablation study is thorough—each component is removed and the effects are reported, including penetration depth and interaction error. The authors are also candid about limitations: coarse appearance, two-person only, and the WildCHI pseudo-GT being manually filtered. That openness is worth crediting.\n\nThe soft spots are real but addressable. No error bars or significance tests anywhere, so the modest gains over BUDDI and CloseInt could be noise. The appearance-memorization concern needs a direct test: freeze the pose, optimize the appearance branch alone, and show the rendered image tracks pose changes. The WildCHI circularity is mild—they don't train on it for their own benchmark numbers—but the claim that the dataset can improve regression methods is demonstrated with only one baseline, CloseInt. Code and checkpoints are promised but not yet verified available.\n\nWho is this for? Anyone working on optimization-based 3D human reconstruction, especially close interactions. The idea of using RGB appearance as a soft constraint is worth discussing, and the dataset may be useful for training. It deserves a serious referee: the method is novel, the experiments are standard, and the concerns are fixable rather than fatal. My recommendation: send it to peer review, but ask for a pose-locked appearance ablation, error bars, and clarification of code availability before acceptance.","headline":"Solid, honest empirical package for close two-person reconstruction, but the paper's central appearance-reasoning claim is not yet proven; the rest of the pipeline likely carries the results.","tokens_in":14690,"tokens_out":1803,"would_cite":true,"duration_ms":23540,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reconstructing close human interactions from monocular video by jointly optimizing motion and appearance.","keywords":["3D human pose estimation","close human interaction","dual-branch optimization","diffusion proxemics prior","3D Gaussian splatting","appearance reasoning","in-the-wild reconstruction","pseudo ground-truth dataset"],"falsifier":"Take a close-interaction video in which the two people wear identical clothing, and render the two reconstructed bodies with swapped appearance textures; if the method still recovers correct depth ordering and pose accuracy, the appearance branch is not the source of the disambiguation, and the central claim would be unsupported.","tokens_in":13621,"feed_emoji":"🫂","tokens_out":8021,"duration_ms":80066,"temperature":0.7,"pith_summary":"Close two-person interactions are hard to reconstruct from a single video because occluded bodies create depth ambiguities that even modern segmentation and keypoint models fail to resolve. This paper proposes a dual-branch optimization framework that reconstructs the two people's body motions and, at the same time, a coarse per-video 3D Gaussian appearance of each person. The core claim is that rendering the reconstructed Gaussians into the image and comparing them with the raw RGB frames gives a trustworthy depth-ordinal and alignment signal, so the optimization can decide who is in front and how the bodies touch. The motion side is constrained by a diffusion model trained on large indoor interaction datasets to provide proxemics and pose priors that transfer to outdoor scenes. On Hi4D and 3DPW the method reports the best MPJPE among the compared approaches (59.1 mm and 64.5 mm), and the authors release a 100-video in-the-wild interaction dataset with pseudo ground truth built from their framework.","feed_headline":"Appearance-based optimization reconstructs close human interactions","feed_subtitle":"Jointly optimizing a diffusion proxemics prior and 3D Gaussian appearance beats keypoint-only methods on Hi4D and 3DPW.","key_machinery":"The load-bearing object is the dual-branch optimization. The motion branch is a pre-trained diffusion model with cross-attention between the two people; during optimization its network parameters are finetuned, so the prior itself regularizes the predicted motion. The appearance branch is an optimizable latent tensor decoded by a U-Net into a 14-channel Gaussian UV map, mapped onto the SMPL body surface and rendered with 3D Gaussian Splatting; an identity channel restricts keypoint supervision to each individual's rendered region. The two branches are optimized jointly against an appearance loss (L1, SSIM, LPIPS), a keypoint reprojection loss, a mesh-penetration distance field, and a smoothness term. This rendering-and-compare loop is the machinery that turns raw RGB into depth-ordinal constraints on pose.","core_discovery":"The paper claims that jointly optimizing a diffusion-based motion prior and a per-video 3D Gaussian appearance model, under appearance, proxemics, and physics constraints, recovers accurate 3D close interactions from monocular in-the-wild videos, and that this outperforms existing single- and multi-person reconstruction approaches. The central mechanism is that the appearance branch converts raw RGB into pose constraints: splatting both people's Gaussians to the image plane lets the optimization compare rendered and original images, which disambiguates depth ordering and improves model-image alignment in occluded frames. The diffusion prior contributes by keeping the non-convex optimization away from implausible local minima, while a differentiable penetration loss keeps the bodies physically plausible.","pith_inferences":["The same dual-branch scheme could extend to crowds of more than two people, since the appearance branch only needs an aggregate foreground mask; the main obstacle is training the proxemics prior on multi-person interaction data.","At roughly 3–5 minutes per 128-frame video, the optimization is fast enough that faster Gaussian splatting renderers could turn it into an interactive annotation tool.","Adversarial appearance settings, such as mismatched or camouflage textures between the two people, should degrade depth-ordering accuracy if the appearance branch is doing real work, which gives a direct test of the mechanism.","The released pseudo ground-truth dataset could be used to train feed-forward regression models, potentially closing the loop between slow optimization quality and fast inference."],"forward_implications":["On the two benchmarks tested, the method reports the lowest MPJPE among the compared methods (59.1 mm on Hi4D and 64.5 mm on 3DPW), indicating that appearance-based constraints improve pose accuracy in addition to depth ordering.","Because the framework needs only a whole-scene foreground mask rather than per-person segmentation, it sidesteps the failure of current semantic-parsing models in close contact.","The diffusion proxemics prior, trained on indoor interaction motion data, transfers to outdoor in-the-wild videos, making optimization-based reconstruction usable outside studios.","The pseudo ground-truth annotations produced by the framework improve a regression-based interaction method when used as training data, suggesting a data-generation loop.","When the appearance branch is removed, depth ordinal relationships become incorrect even though body poses stay accurate, which isolates the appearance cue's role."],"supporting_citations":[{"why":"Provides the differentiable 3D Gaussian splatting renderer that lets the appearance branch compare rendered and original images.","marker":"[23]"},{"why":"Defines the SMPL body model whose vertices carry both pose and the UV-mapped Gaussian appearance.","marker":"[36]"},{"why":"The U-Net backbone that decodes the optimizable latent tensors into Gaussian UV maps.","marker":"[47]"},{"why":"Large-scale human-human interaction motion data used to train the diffusion proxemics prior.","marker":"[33]"},{"why":"Additional large-scale interaction data (Inter-X) used to train the proxemics prior.","marker":"[62]"},{"why":"The optimization-based baseline the paper targets; it supplies the keypoint plus proxemics fitting approach this work extends with appearance.","marker":"[40]"},{"why":"The diffusion model formulation and noise schedule used to build the proxemics prior.","marker":"[41]"},{"why":"Hi4D, the indoor close-interaction benchmark used for training and evaluation.","marker":"[65]"},{"why":"3DPW, the in-the-wild benchmark used to evaluate generalization.","marker":"[57]"},{"why":"The differentiable distance-field penetration loss used to keep reconstructed bodies physically plausible.","marker":"[55]"}],"fun_headline_variants":["Appearance and proxemics cut through occlusion for close interaction 3D","Proxemics prior plus 3D Gaussians reconstruct close interactions","Appearance cues resolve occlusion for accurate 3D close interactions","Diffusion proxemics prior improves close interaction reconstruction","Joint optimization of appearance and proxemics beats keypoint-only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the appearance branch—two optimizable tensors decoded by a per-video U-Net into Gaussian UV maps—can be optimized per video, without pretraining, so that its RGB rendering loss genuinely constrains body pose rather than merely overfitting the image regardless of pose.","fun_headline_variants_meta":{"raw":{"variants":["Appearance and proxemics cut through occlusion for close interaction 3D","Proxemics prior plus 3D Gaussians reconstruct close interactions","Appearance cues resolve occlusion for accurate 3D close interactions","Diffusion proxemics prior improves close interaction reconstruction","Joint optimization of appearance and proxemics beats keypoint-only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2421,"prompt_tokens":932,"completion_tokens":1489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1404}},"tokens_in":548,"tokens_out":1489,"duration_ms":12952,"temperature":1.0,"reasoning_tokens":1404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:25:45.972473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a close-interaction video in which the two people wear identical clothing, and render the two reconstructed bodies with swapped appearance textures; if the method still recovers correct depth ordering and pose accuracy, the appearance branch is not the source of the disambiguation, and the central claim would be unsupported.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Provides the differentiable 3D Gaussian splatting renderer that lets the appearance branch compare rendered and original images."},{"cited_title":"Smpl: A skinned multi- person linear model","cited_arxiv_id":null,"evidence_quote":"Defines the SMPL body model whose vertices carry both pose and the UV-mapped Gaussian appearance."},{"cited_title":"U-net: Convolutional networks for biomedical image segmentation","cited_arxiv_id":null,"evidence_quote":"The U-Net backbone that decodes the optimizable latent tensors into Gaussian UV maps."},{"cited_title":"Inter-X: Towards Versatile Human-Human Interaction Analysis","cited_arxiv_id":"2312.16051","evidence_quote":"Additional large-scale interaction data (Inter-X) used to train the proxemics prior."},{"cited_title":"Generative proxemics: A prior for 3d social interaction from images","cited_arxiv_id":null,"evidence_quote":"The optimization-based baseline the paper targets; it supplies the keypoint plus proxemics fitting approach this work extends with appearance."},{"cited_title":"Hi4d: 4d instance segmen- tation of close human interaction","cited_arxiv_id":null,"evidence_quote":"Hi4D, the indoor close-interaction benchmark used for training and evaluation."},{"cited_title":"Recovering accurate 3d human pose in the wild using imus and a moving camera","cited_arxiv_id":null,"evidence_quote":"3DPW, the in-the-wild benchmark used to evaluate generalization."},{"cited_title":"Capturing hands in action using discriminative salient points and physics sim- ulation","cited_arxiv_id":null,"evidence_quote":"The differentiable distance-field penetration loss used to keep reconstructed bodies physically plausible."}],"review_version":1}