{"id":"f8563c2a-56e9-48b3-b2fd-84516aaac294","arxiv_id":"2607.06691","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"CoMind releases 41 h of synchronized multi-view cooking collaboration with social-cue annotations and three ToM-oriented benchmarks on which current VLMs score poorly until fine-tuned.","lead":"CoMind is a new multimodal ego-exo dataset of 41 hours of unscripted two-person cooking, with gaze, audio, 3D scans and annotations for joint attention, socially cued actions and handovers. It supplies the first large real-world resource for benchmarking Theory-of-Mind-style social reasoning in collaborative physical tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"The three tasks are presented as an operationalization of Theory of Mind, yet no independent cognitive or behavioral validation is provided that success on them requires or measures ToM.","rationale":"The reader's strongest claim (VLM deficiency + large FT gains establishing a useful foundation) is empirically well-supported by Tables 2–4 and the released data/prompts; the spatial-grounding failures and verb-accuracy jumps are concrete and reproducible. The weakest assumption the reader already isolated—the unvalidated ToM mapping in §3.1—is precisely the softest load-bearing point for the paper's broader scientific narrative. It does not invalidate the dataset or the reported numbers, so the ACCEPT verdict stands; the concern only qualifies the interpretive leap from “models improve on these labels” to “models acquire Theory of Mind.” No stronger technical flaw (data leakage, broken metrics, non-reproducible FT) appears in the manuscript or supplement.","tokens_in":34219,"tokens_out":517,"duration_ms":24155,"concrete_test":"On the held-out test split, compute Spearman rank correlation between each model's per-sample accuracy (or IoU) on the three CoMind tasks and its accuracy on a standard multimodal ToM suite (e.g., EgoToM or MuMA-ToM items that can be video-grounded). If the correlation is near zero or negative after controlling for object-category frequency, the ToM operationalization claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central framing (Abstract, Introduction, §3.1) equates the three vision tasks—Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction—with Theory of Mind for physical collaboration. Success is defined purely by matching annotated boxes, verbs/nouns, cue types, delivery flow, initiator, and TTH. No experiment, human study, or correlation with established ToM instruments (e.g., those cited in Related Work) demonstrates that these labels capture mental-state inference rather than surface multimodal pattern matching of gaze, speech, and object trajectories. Without that link, the claim that CoMind (and the large FT gains on Qwen3-VL) advances ToM-capable socially aware AI rests on an untested equivalence; the empirical VLM numbers remain valid as a social-perception benchmark but do not automatically establish the stronger cognitive claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"CoMind presents a multimodal ego–exocentric dataset of unscripted two-person cooking collaboration (80 sessions, ~41 h dual-view / ~81 h single-view, 125 participants) with synchronized Aria egocentric video, dual GoPro exocentric views, gaze and hand tracking, audio/transcripts, camera trajectories, dense kitchen scans, and scanned object meshes. The authors define three vision benchmarks intended to operationalize Theory of Mind in physical collaboration—Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction—with manual annotations for shared objects, social cue types, verbs/nouns, delivery flow, initiator, and time-to-handover. They evaluate multiple closed- and open-weight VLMs under a shared prompting protocol with participant-disjoint train/test splits, report near-zero spatial grounding for most proprietary models, and show large gains after LoRA fine-tuning of Qwen3-VL (e.g., action-verb accuracy rising from ~0.09–0.14 to ~0.64–0.65).","tokens_in":34519,"tokens_out":1183,"duration_ms":21200,"significance":"If the resource and benchmarks hold as described, CoMind fills a clear gap relative to prior ego/exo and multi-agent datasets (Table 1): long-horizon goal-directed physical collaboration with gaze, verbal/gestural cue labels, and 3D scene/object grounding. The participant-disjoint evaluation and the documented fine-tuning gains provide concrete evidence that the training split is useful for socially conditioned perception, not only as a zero-shot stress test. The three tasks are well-specified for proactive assistance research even if one disputes the strongest ToM rhetoric. Public release of data and benchmarks is a genuine contribution to multimodal and embodied social AI.","major_comments":[{"comment":"Abstract, Introduction, and §3.1 frame the three vision tasks as a formalization of Theory of Mind for physical collaboration, yet success is defined solely by matching annotated boxes, verbs/nouns, cue types, flow, initiator, and TTH. No human study, correlation with established ToM instruments (those cited in Related Work), or ablation that isolates mental-state inference from surface multimodal pattern matching is provided. The empirical VLM numbers remain valid as a social-perception benchmark; the stronger claim that CoMind advances ToM-capable AI should be tempered to “social cue–conditioned collaborative perception,” or supported by an explicit validation argument.","section":null},{"comment":"§3.4 describes multi-stage human annotation with QA and author review, but the manuscript reports no inter-annotator agreement (e.g., IoU agreement on boxes, Cohen/Fleiss κ on cue type, initiator, delivery flow, or verb/noun labels). For a dataset paper whose central value is the annotations (Tables 2–4, Figs. 13–15, Supp. annotation guides), IAA (or at least double-annotation on a held-out subset) is load-bearing for trusting the reported metrics and the fine-tuning gains.","section":null},{"comment":"§4 evaluates only general VLMs (plus random/most-frequent priors). For Joint Attention Estimation and handover timing/localization, the literature already has specialized gaze, mutual-attention, and action-anticipation models (cited in §2). Without at least one non-VLM or modular baseline (e.g., gaze-intersection + object detector, or a standard anticipation transformer), it is hard to separate “VLMs fail at social grounding” from “the tasks are hard for any current method.” Adding such baselines would strengthen the claim of a significant performance deficiency.","section":null}],"minor_comments":[{"comment":"Abstract vs. body: the abstract lists “Action Anticipation” while §3.1 and Tables use “Socially Conditioned Object Interaction Anticipation”; align naming throughout.","section":null},{"comment":"§4: only 5 uniformly sampled frames from the 10 s context are fed to VLMs. An ablation on frame count / video input (beyond the partial Supp. Table S6) would clarify whether spatial failures are partly an input bottleneck.","section":null},{"comment":"Table 4 TTH metric uses a tight ±0.25 s window; report also mean absolute error or a coarser bin so temporal performance is easier to interpret.","section":null},{"comment":"Object hierarchy (L1–L3) and synonym handling for Cat. (L1) are important for reproducibility; point more explicitly from the main text to Supp. §S5 / Fig. S9.","section":null},{"comment":"Fig. 1 / Table 1: “81” hours ego vs. “40h 43m” single-view wording can confuse; state dual-view vs. single-view totals once in a consistent way.","section":null},{"comment":"Minor polish: arXiv id and some model version strings (Claude Opus 4.5/4.6, GPT 5.x) will age quickly—cite system cards with access dates as already partly done in references.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Solid dataset paper; the ToM branding is the main overclaim and is fixable with wording. Missing IAA is the one item I would insist on before camera-ready. Scope fits a CV/ML venue that accepts resource contributions. I would not reject over the ToM framing alone."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a real dataset paper, not a rehash. Dual Aria glasses, dual GoPros, gaze, audio, dense kitchen scans, object meshes, and ~41 hours of unscripted two-person cooking with participant-disjoint splits is a combination Ego4D / EgoExo4D / HoloAssist / LEMMA simply do not give you. The three tasks (joint attention, socially conditioned next action, pre-reach handover with initiator/flow/TTH) are carefully specified, densely annotated, and hard enough that most zero-shot VLMs collapse on spatial grounding while fine-tuned Qwen3-VL jumps a lot (verb accuracy ~0.09–0.14 → ~0.64–0.65). That empirical claim is solid and useful for anyone building assistive or collaborative agents.\n\nWhat they did well: capture protocol, sync, MPS trajectories, scan alignment, custom VIA interfaces, multi-level object hierarchy, and public release of data plus prompts. Baselines include random/most-frequent, several closed models, and open models with and without LoRA. Ablations on frames vs transcripts are in the supplement. This is the kind of resource people will actually train on.\n\nSoft spots, in proportion. The Theory-of-Mind framing is the softest part. They equate success on these vision tasks with ToM without any independent cognitive validation or correlation to classic ToM instruments. The stress-test note is right about that, but it is mostly packaging: the tasks still stand as social-perception and proactive-assistance benchmarks even if you drop the ToM language. Minor technical gaps: no inter-annotator agreement numbers, fixed 10 s context and IoU/TTH thresholds, and spatial scores remain low even after fine-tuning. None of that sinks the contribution.\n\nWho it is for: embodied vision, HRI, and multimodal people who need multi-agent social cues with 3D grounding. I would bring it to reading group, cite the dataset when I need collaborative cooking data, and send it to peer review without hesitation. Accept at a top CV/embodied venue; ask them to tone down the ToM claim or add a short caveat.","headline":"Strong multi-agent ego-exo cooking resource with social-cue labels and three hard prediction tasks; the ToM framing is marketing, not a load-bearing flaw.","tokens_in":35130,"tokens_out":546,"would_cite":true,"duration_ms":7752,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"CoMind is a multimodal cooking-collaboration dataset and three Theory-of-Mind vision tasks that show current vision-language models lack social grounding, while fine-tuning on the data substantially closes the gap.","keywords":["egocentric vision","theory of mind","human-human collaboration","joint attention","action anticipation","object handover","multimodal dataset","social cues"],"falsifier":"Train models to high accuracy on the three CoMind tasks, then test whether the same models correctly predict a held-out partner’s next need or successful assistance in a new, unscripted kitchen session whose social-cue distribution differs from the training kitchens; failure of transfer would falsify the claim that the tasks capture general collaborative ToM.","tokens_in":35133,"feed_emoji":"🤝","tokens_out":1083,"duration_ms":19800,"temperature":0.7,"pith_summary":"The paper argues that Theory of Mind—the ability to infer others’ intentions from social cues—is essential for real collaboration yet almost unmeasurable in natural settings because prior datasets and benchmarks stay textual, single-agent, or short-horizon. CoMind fills that gap with 41 hours of synchronized egocentric and exocentric video, audio, gaze, hand tracking, and 3D scans of people cooking together, plus dense annotations of shared attention, social cues, and object interactions. Three escalating tasks are defined: estimating joint attention on an object, anticipating a helper’s next socially conditioned action, and predicting a handover before any reach begins. Zero-shot vision-language models score near zero on spatial localization and only modestly on cue and category recognition; fine-tuning open-weight models on CoMind training splits produces large gains, especially on action verbs and initiator identity. The claim is that this resource is now the practical foundation for training and evaluating AI that can read social signals and assist proactively in shared physical work.","feed_headline":"VLMs nearly fail joint attention and handovers; CoMind fine-tuning fixes much of it","feed_subtitle":"41 hours of dual-person cooking video plus three ToM tasks expose social-grounding gaps and give open models a training path","key_machinery":"Three interdependent vision tasks that operationalize Theory of Mind for physical collaboration: Joint Attention Estimation (shared object, dual-view boxes, cue type), Socially Conditioned Object Interaction Anticipation (helper’s next verb-noun-box given leader cues), and Collaborative Handover Prediction (time-to-handover, delivery flow, initiator, cue, object box before any reach).","core_discovery":"Current state-of-the-art vision-language models exhibit severe deficiencies on three new social-reasoning tasks grounded in real dual-person cooking collaboration—particularly near-zero accuracy on bounding-box localization of jointly attended or soon-to-be-handed objects—while fine-tuning the same open-weight models on CoMind’s training split yields large, consistent lifts (for example action-verb accuracy rising from roughly 0.09–0.14 to 0.64–0.65), establishing the dataset as a usable training and evaluation foundation for socially aware AI.","pith_inferences":["Because the tasks fix the helper as the agent that must react to the leader, the same protocol can be reused to train robot helpers that must act without explicit commands.","The large zero-shot gap on bounding boxes versus moderate category accuracy suggests current VLMs already parse linguistic intent but lack the cross-view geometric binding needed for embodied collaboration.","Gaze-following and body-pose pseudo-labels already reconstructible from the dual views and Aria MPS data could bootstrap denser social-cue supervision without new manual annotation.","If the three tasks truly track ToM, performance curves on CoMind should correlate with independent ToM battery scores of the same models on classic false-belief or intention-inference tests."],"forward_implications":["Multimodal perception systems can be trained to detect joint attention and social cues from synchronized first- and third-person video plus gaze and speech.","Proactive assistive agents can be scored on whether they correctly anticipate a partner’s next object interaction or handover before physical motion begins.","Collaborative planning models gain temporally aligned, 3D-grounded training data for long-horizon kitchen tasks that include verbal and gestural intent.","Open-weight vision-language models can be domain-adapted for social grounding, turning near-zero spatial scores into competitive ones after fine-tuning on CoMind.","Future 3D extensions become feasible by lifting the existing 2D boxes into the provided scene and object scans for embodied spatial reasoning."],"fun_headline_variants":["VLMs near-zero on joint attention; CoMind fine-tuning lifts accuracy","CoMind exposes VLM social gaps; fine-tuning recovers joint attention","Dual cooking videos reveal VLMs fail ToM tasks until fine-tuned","Open VLMs miss handovers and gaze; CoMind data closes much of the gap","Fine-tuning on CoMind turns VLM failures on social reasoning into gains"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That success on these three hand-crafted vision tasks is a valid stand-in for Theory of Mind in real collaboration, without independent cognitive or behavioral validation that the tasks measure mental-state inference rather than pattern matching of surface cues.","fun_headline_variants_meta":{"raw":{"variants":["VLMs near-zero on joint attention; CoMind fine-tuning lifts accuracy","CoMind exposes VLM social gaps; fine-tuning recovers joint attention","Dual cooking videos reveal VLMs fail ToM tasks until fine-tuned","Open VLMs miss handovers and gaze; CoMind data closes much of the gap","Fine-tuning on CoMind turns VLM failures on social reasoning into gains"]},"model":"grok-4.5","effort":"low","cost_usd":0.006542,"raw_usage":{"total_tokens":1635,"prompt_tokens":817,"num_sources_used":0,"completion_tokens":104,"cost_in_usd_ticks":65420000,"prompt_tokens_details":{"text_tokens":817,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":714,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":817,"tokens_out":104,"duration_ms":30650,"temperature":1.0,"reasoning_tokens":714,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T23:18:23.950643+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train models to high accuracy on the three CoMind tasks, then test whether the same models correctly predict a held-out partner’s next need or successful assistance in a new, unscripted kitchen session whose social-cue distribution differs from the training kitchens; failure of transfer would falsify the claim that the tasks capture general collaborative ToM.","supporting_citations":[],"review_version":1}