{"id":"33d95ec3-936c-4fed-af2e-2fd25f11c05f","arxiv_id":"2602.03430","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A video benchmark (75 tasks, 91,581 step annotations, AND/OR task graphs) and a fine-tuned MLLM baseline whose entropy-driven planner selects parallel-thread actions, beating zero-shot closed-source models on its own metrics.","lead":"ProAct-75 is a new 75-task video benchmark—built on Ego-Exo4D, COIN, UCF-Crime, and self-recorded clips—that gives robots step-level annotations plus AND/OR task graphs for proactive assistance, maintenance, and safety. ProAct-Helper, a fine-tuned multimodal LLM, chooses actions by minimizing an entropy over parallel workflow threads, beating zero-shot closed-source models but partly by optimizing the metric it is scored on.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph fidelity is the load-bearing premise: Appendix A admits human traces violate task-graph preconditions, so the feasibility filter and thread-entropy metrics may reward graph artifacts rather than real collaboration quality.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the annotated task graphs must faithfully encode real precedence/parallel structure for the feasibility filter and the PA/SS metrics to be meaningful. The paper's own Appendix A explicitly concedes that human traces may violate graph preconditions, and the Table 3 simulation's human-response model is unvalidated. I add that the PA metric is partially circular: Eq. (9) minimizes Hmix, and PA is defined as the fraction of robot actions on a thread different from the human's most recent thread, so the headline parallel-action gain is partly by construction. This concern supports the reader's CONDITIONAL verdict: the paper should report the safeguard invocation rate, validate the human-behavior model, and provide an independent outcome metric (e.g., wall-clock task completion). It does not by itself demand rejection, because the perception gains (trigger/task/step) are not affected by the graph-fidelity issue, and the entropy-driven selection is a reasonable, clearly specified heuristic. The dataset number inconsistencies and missing release hash identified by the reader remain valid but are secondary to the core empirical claim.","tokens_in":25741,"tokens_out":7128,"duration_ms":74849,"concrete_test":"Instrument Algorithm 1's safeguard: record, for every human next-step transition in the Table 3 rollouts, whether g_{t+1} is graph-illegal and the one-step runtime alignment fires. Then recompute SS, E, ER, and PA restricted to rollouts with zero safeguard events. If more than ~10% of human next steps require relaxation, or if ProAct-Helper's SS/PA advantage over Greedy and the closed-source LLMs shrinks or disappears on the clean subset, the task graph does not faithfully model human execution and the reported gains rest on the relaxation rather than on the entropy-driven heuristic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on ProAct-75's task graphs faithfully encoding the real precedence/parallel structure of human activity. Appendix A concedes the opposite: ground-truth human step traces 'may not strictly satisfy task-graph preconditions,' forcing the Table 3 simulator to relax A_legal via a one-step runtime alignment that inserts the observed human step into the legal set (Algorithm 1). This is not a minor implementation detail. The planner's action filter A_legal_t and the thread mapping π used in Eqs. (6)-(9) are both derived from the same task graph. If the graph disagrees with actual human execution, then (i) the feasibility filter prunes actions on the basis of an invalid model, and (ii) the metrics E, ER, and PA, all computed over π-induced threads, partially measure the graph's assumed structure rather than collaboration quality. The parallel-action advantage is especially vulnerable because Eq. (9) minimizes Hmix, the same quantity used to define E/ER, and PA counts exactly the different-thread choices this objective favors; the 15.58% PA gain is therefore partly by construction. A closely related premise is Section 6.4's human-behavior model ('the human follows the annotated trajectory unless preempted by the robot, then switches to parallel branches'), an author-designed assumption that every Table 3 saved-step number inherits. Neither the graph's fidelity to human traces nor the behavioral model's realism is measured anywhere in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ProAct-75, a video benchmark for proactive response with 75 tasks, 91,581 step-level annotations, and explicit task graphs encoding AND/OR dependencies and parallel threads across assistance, maintenance, and safety-monitoring scenarios. The authors propose ProAct-Helper, an MLLM-based framework with a Hierarchical Binding Module (HBM) for trigger/task/step detection and future-action prediction, and an entropy-driven heuristic search over task-graph threads for proactive action selection. The central claim is that this combination outperforms strong closed-source models, specifically improving trigger mF1 by 6.21%, saving 0.25 more steps in online one-step decisions, and increasing parallel-action rate by 15.58%. The paper also reports ablations, cross-view generalization, latency, and qualitative analyses.","tokens_in":26030,"tokens_out":4302,"duration_ms":46934,"significance":"If validated, ProAct-75 would be a valuable resource: it is one of the few video benchmarks that explicitly provides task-graph structure for proactive decision-making, and the scale (75 tasks, 91k annotations) plus the release of code and detailed annotation protocol are strengths. The idea of using a lightweight graph-constrained entropy objective on top of an MLLM is interesting and potentially practical. The paper is also commendable for including hyperparameter sensitivity, OOD view evaluation, and failure-case analysis. However, the core claim about proactive action selection is currently supported by metrics that partly encode the planner's own objective and by a simulation whose human-behavior model is not grounded in measured data; the experimental validation therefore does not yet independently establish the claimed advantage.","major_comments":[{"comment":"The Parallel Actions (PA) metric is defined as the fraction of effective robot actions with π(a) ≠ π(h_prev), i.e., actions on a different thread from the human's most recent thread. This is exactly the behavior that the entropy objective in Eq. (9) is designed to produce, since minimizing H_mix favors actions in threads with low current human participation. Consequently, the 15.58% PA gain in Table 1 and the 33.95% PA in Table 3 are substantially built into the evaluation rather than independently measured. The paper needs an external validation of collaboration quality—for example, a user study, task-completion time in a validated environment, or a comparison against a planner optimizing a different, equally reasonable objective—to show that the entropy choice yields real improvement rather than merely optimizing the metric it is scored on.","section":"§6.1 (Evaluation Metrics) and §5.3, Eq. (9)"},{"comment":"The paper states that human execution 'may not strictly satisfy task-graph preconditions' and introduces a one-step runtime alignment that augments A_legal with the observed human step for the Table 3 simulation. This is a direct admission that the task graphs do not always match actual human behavior. Since the legal set A_legal and the thread mapping π used in Eqs. (6)–(9) are both derived from the same graph, any graph–behavior mismatch means (i) the feasibility filter may prune actually-helpful robot actions, and (ii) the metrics E, ER, PA, and SS computed over π-induced threads partly measure graph artifacts rather than collaboration quality. The authors should quantify the mismatch (e.g., the percentage of human steps that required the safeguard) and report the main results on the subset of videos where the graph is fully consistent with the human trace.","section":"Appendix A, Algorithm 1, and §3.1"},{"comment":"The Table 3 simulation assumes that 'the human follows the annotated trajectory unless preempted by the robot, then switches to parallel branches until completion.' This is an author-designed behavioral assumption that is never validated against human data. All saved-step numbers in Table 3, and the online SS results in Table 1 that similarly rely on ground-truth trajectory membership, inherit this assumption. The paper should either provide empirical evidence that this response model reflects how humans actually adapt to robot actions, or show sensitivity of the conclusions to alternative human models (e.g., a human who ignores the robot and continues on the original thread, or one who re-plans optimally). Without such analysis, the saved-steps claim cannot be distinguished from an artifact of the simulation's assumptions.","section":"§6.4, Table 3"}],"minor_comments":[{"comment":"Typo: 'F or trigger/task/step detection' should be 'For trigger/task/step detection'.","section":"§6.2"},{"comment":"The symbol λ_bind is used in Eq. (3) but the decomposition into λ_tt and λ_ts appears only in Eq. (5); consider defining these earlier.","section":"§5.1"},{"comment":"The notation A_pred is treated as a set with duplicates removed, but earlier it is introduced as a sequence; please clarify the conversion and how duplicates are handled.","section":"§6.1"},{"comment":"Greedy and ProAct-Helper both achieve SS = 9.868, but Greedy has much higher E (0.837 vs 0.662) and lower PA (28.11 vs 33.95). This difference is not fully explained; the paper discusses E/ER for closed-source LLMs but not why Greedy matches ProAct-Helper on SS.","section":"Table 3"},{"comment":"The safety-monitoring tasks lack proactive action selection and future-action prediction; the paper does not state whether Table 1 and Table 3 results are restricted to assistance/maintenance tasks. Please report the task distribution for the proactive-action-selection metrics.","section":"§6.4 and Figure 5b"},{"comment":"The two 'ProAct-Helper (plain)' rows for 3B and 7B are not clearly labeled in Table 1; adding a column or grouping would improve readability.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource and the overall system design are likely useful to the community. However, the headline claims about proactive action selection rest on evaluation metrics that are partially circular with the planner's objective and on a simulation human model that is not validated. These are fixable concerns—through independent metrics, graph-fidelity reporting, and sensitivity analysis—but they are central to the paper's contribution and require a substantive revision. I would be willing to re-review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution—the first video benchmark for proactive response with explicit AND/OR task graphs and parallel-thread structure, plus a reasonable MLLM baseline with an entropy-driven planner. The math is internally consistent, the ablations are helpful, and the headline deltas trace arithmetically. Worth a serious referee.\n\nWhat's new: ProAct-75 aggregates Ego-Exo4D, COIN, UCF-Crime, and self-collected video into 91.5k step annotations across 75 tasks, with task graphs proposed by GPT-4o/Gemini and manually reviewed. That fills an actual gap. The thread-entropy objective (Eq. 7–9) is a sensible, implementable heuristic for parallel action selection, and the HBM contrastive losses are reasonable. The cross-view OOD results show real robustness gains.\n\nSoft spots, in order. First, the parallel-action metric is too close to the objective. PA counts exactly the different-thread choices that Eq. (9) minimizes; the 15.58% gain is partly by construction. The paper should report an independent metric (wall-clock completion in a neutral simulator, or a user study). Second and more load-bearing: graph fidelity. Appendix A admits human ground-truth traces may not satisfy task-graph preconditions, so Table 3 relaxes legality for the observed human step. Since the feasibility filter, thread mapping, and E/ER/PA metrics all derive from the same graphs, systematic graph-human disagreement would make the metrics measure the graph's assumptions rather than collaboration quality. The saved-steps claim also depends entirely on Section 6.4's human-behavior model, which is author-designed, not measured. These are fixable: a graph-fidelity audit (how often do human traces violate preconditions? How sensitive are the results to those relaxations?) would go a long way. Third, the dataset numbers don't reconcile: abstract says 5,383/91,581, Figure 2 says 5,701/93,835, Section 4.1 says 1,978/6,797 for non-Ego while the table rows sum to 2,255/8,825. That undermines trust until corrected. Fourth, the benchmark artifacts are promised but not released with a commit hash; for a benchmark paper that's a significant gap.\n\nBottom line: the central idea is sound and the work is honest, but the evaluation needs independent metrics and the graph-fidelity assumption needs direct validation. I'd send it to a serious referee, and cite it for the benchmark. For a reading group, maybe—the design choices are discussable.","headline":"ProAct-75 is a genuinely new benchmark with a plausible but unproven central claim; the parallel-action gains are partly built into the metric, and the graph-fidelity premise needs direct validation.","tokens_in":26766,"tokens_out":1468,"would_cite":true,"duration_ms":14349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Proactive robot helpers should aim to reduce thread-mixing, not copy the human's next step.","keywords":["proactive response","human-robot collaboration","task graphs","AND/OR graphs","multimodal large language models","video understanding benchmark","action selection","entropy-driven planning"],"falsifier":"Take a subset of videos where the human's observed step order visibly contradicts the annotated task graph, disable the one-step runtime alignment in the Table 3 simulation, and re-measure saved steps and parallel actions. If ProAct-Helper's advantage over commonsense models shrinks or reverses when graph legality is enforced strictly, then the reported gains partially measure artifacts of the graph annotation rather than better collaboration.","tokens_in":25431,"feed_emoji":"🤖","tokens_out":6646,"duration_ms":111864,"temperature":0.7,"pith_summary":"The paper argues that proactive assistance in human-robot collaboration should be treated as structure-aware decision-making: the robot should choose the next graph-feasible action that minimizes 'thread-mixing entropy' across parallel branches, rather than imitating the human's intended next step. To support this claim it introduces ProAct-75, a benchmark of 75 tasks with 91,581 step-level annotations and explicit AND/OR task graphs covering assistance, maintenance, and safety monitoring. It also presents ProAct-Helper, a multimodal large-language-model baseline with a hierarchical binding module for perception and an entropy-driven one-step search for action selection. On this benchmark, the paper reports that ProAct-Helper outperforms strong closed-source models: trigger-detection mF1 up 6.21%, 0.25 more saved steps in online one-step decisions, and 15.58% more parallel actions. If true, the result implies that a lightweight graph heuristic plus a relatively small fine-tuned model can beat common-sense planning from much larger models, and that ProAct-75 provides a reusable yardstick for structure-aware proactive response.","feed_headline":"Proactive robots work best with task-graph planning, not imitation","feed_subtitle":"Graph-aware planning saves 0.25 steps, with 15.58 percent more parallel actions than closed models.","key_machinery":"The central object is the task graph: a directed acyclic graph whose nodes are executable steps or structural start/end nodes, with edges encoding precedence and each node typed AND or OR to say when it becomes enabled. From the graph, thread mapping π groups parallelizable branches that share reachability, and the entropy-driven heuristic (Eqs. 7–9) chooses the feasible action that minimizes thread-mixing entropy, favoring stable parallel progress over copying the human's next step. A second mechanism, the Hierarchical Binding Module (HBM), is a contrastive loss that ties trigger-to-task and task-to-step representations to improve long-tail perception. Together they turn a standard MLLM int","core_discovery":"The paper's central claim is that proactive response is best modeled as five coupled tasks—trigger detection, task detection, step detection, future action prediction, and proactive action selection—with action selection grounded in an explicit task graph. ProAct-Helper learns perception via hierarchical binding losses that align trigger-to-task and task-to-step representations, then selects the next robot action by searching the task graph: among all currently legal actions, it picks the one that minimizes the length-weighted binary entropy of human/robot participation across parallel threads. The paper reports that this entropy-driven selection yields more parallel actions and saved steps","pith_inferences":["A testable extension the paper leaves implicit: replace the one-step entropy lookahead with a multi-step or full-horizon search over the same objective; the reported gains may grow if global thread balance, not just the next step, drives the heuristic.","The 'saved steps' and parallel-action numbers are computed under an author-designed simulation of human behavior (the human follows the annotated trajectory unless preempted); real deployments would need logged human responses to verify that reduced thread-mixing entropy corresponds to perceived collaboration quality.","The entropy objective is a proxy for cognitive load; a direct human-subject study comparing thread-mixing entropy against task-switching costs or subjective workload could validate or refine the design choice.","The paper's failure analysis notes future-step hallucination; one natural follow-up is graph-feasible decoding inside the language model itself rather than filtering its predictions afterwards, which could clean up action candidates before the graph search."],"forward_implications":["A fine-tuned 3B/7B multimodal model combined with a one-step graph heuristic can outperform strong closed-source models on proactive action selection, so graph structure can substitute for raw model scale.","Structure-aware benchmarks can measure collaboration properties that ordinary video benchmarks cannot: saved steps, parallel-action rate, and thread-mixing entropy.","The approach is deployable in closed loop: planning latency is small (about 0.08 seconds per step in the reported measurements), enabling online one-step decisions from keyframe streams.","Common-sense LLM planning often defaults to waiting when the human's next step seems imminent, whereas the entropy objective establishes stable robot execution threads, as shown by the parallel-action and robot-entropy metrics.","Cross-level contrastive binding (trigger→task and task→step) consistently improves long-tail task and step recognition across backbone sizes, indicating that hierarchical supervision matters for perception."],"fun_headline_variants":["Graph-guided agents beat imitation in proactive tasks","ProAct benchmark: task graphs boost proactive robot actions","Entropy-driven search lifts proactive action selection","Proactive agents save steps with graph-aware planning","Task-graph grounding sharpens proactive response accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole argument rests on the assumption that the hand-annotated task graphs faithfully encode the real precedence and parallel structure of each task—and the paper's own appendix concedes that human ground-truth step traces may not strictly satisfy graph preconditions, requiring a one-step runtime alignment safeguard in the full-rollout simulation.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided agents beat imitation in proactive tasks","ProAct benchmark: task graphs boost proactive robot actions","Entropy-driven search lifts proactive action selection","Proactive agents save steps with graph-aware planning","Task-graph grounding sharpens proactive response accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":9.6e-05,"raw_usage":{"total_tokens":834,"prompt_tokens":729,"completion_tokens":105,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":49}},"tokens_in":473,"tokens_out":105,"duration_ms":2466,"temperature":1.0,"reasoning_tokens":49,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:59:34.820887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a subset of videos where the human's observed step order visibly contradicts the annotated task graph, disable the one-step runtime alignment in the Table 3 simulation, and re-measure saved steps and parallel actions. If ProAct-Helper's advantage over commonsense models shrinks or reverses when graph legality is enforced strictly, then the reported gains partially measure artifacts of the graph annotation rather than better collaboration.","supporting_citations":[],"review_version":1}