{"id":"b9210269-5f9b-4ff0-98e3-00e3c8621c35","arxiv_id":"2611.10498","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EmbodiedIntentBench couples mission contracts with causal execution accounting to show that high-recall perception models can fail missions while lower-recall models succeed, challenging isolated accuracy metrics.","lead":"This paper introduces EmbodiedIntentBench, a benchmark for evaluating runtime adaptation strategies in mobile robotic missions using structured mission contracts and causal resource accounting. It reveals that mission success can diverge sharply from perception accuracy because computation delays cause missed observations, and that ignoring this effect can reverse the ranking of competing strategies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Multi-phase Depot result (Mask R-CNN union 1/16 vs YOLO11s tiled 14/16) depends on the serial blocking execution model; under an asynchronous/no-skip protocol the gap may shrink, weakening the central claim.","rationale":"The reader correctly identifies the serial blocking model as the weakest assumption. However, the reader still accepts the paper, apparently satisfied that the limitation is acknowledged. In my view, the multi-phase result is the paper's central empirical showpiece; its dependence on a specific (and arguably non-representative) execution model is a load-bearing concern that warrants a conditional verdict. A CONDITIONAL verdict would require the authors to either demonstrate robustness via the proposed test or explicitly bound the effect. The reader's argument about generality to real hardware is orthogonal; the concern here is about the benchmark's internal logic, not external validity. The concrete test directly checks whether the striking 1/16 vs 14/16 contrast survives under a more permissive execution model. If it does not, the paper's strongest claim weakens. Hence a conditional acceptance is more appropriate than unconditional acceptance.","tokens_in":23403,"tokens_out":2728,"duration_ms":28036,"concrete_test":"Re-run all 16 Depot multi-phase conditions (Table 3) with the no-skip execution protocol (as defined in §4.4 and used in Fig. 6), which makes all captures available regardless of inference overlap, for both Mask R-CNN union and tiled YOLO11s. If Mask R-CNN union achieves ≥10/16 missions (or the gap to YOLO11s tiled narrows to less than 4 missions), then the headline discrepancy is primarily an artifact of the serial-blocking assumption; if the gap remains large, the result is robust to this variation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline result—that a configuration with 87.5% recall completes only 1/16 missions while one with 80.7% recall completes 14/16—rests on the serial blocking execution model (Eq. 5, §2.4). In this model, high-latency inference (e.g., Mask R-CNN union at 6.91 s) skips most captures (only ~1 in 4 processed), directly causing observation loss. The authors' own dynamic-study ablation (Fig. 6b) shows that switching to a no-skip protocol reverses the ordering of two strategies, confirming sensitivity to this assumption. However, that ablation was performed only on the dynamic-urgency study, not on the multi-phase Depot and Lake scenarios where the strongest numerical result lives. Real mobile robotic systems often use asynchronous pipelines or parallel processing that retain more captures even when inference is slow. If the no-skip protocol were applied to Depot, Mask R-CNN union might process a larger fraction of captures, potentially raising its mission success rate and shrinking the gap to tiled YOLO11s. The central conclusion—that perception accuracy alone mispredicts mission success—could still hold (latency still matters for timeliness), but the dramatic 1/16 vs 14/16 gap might vanish, undercutting the paper's most striking evidence. The authors acknowledge the limitation (§2.4, §A.4) but do not quantify its impact on the multi-phase result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents EmbodiedIntentBench, a benchmark for evaluating intent-driven runtime adaptation in mobile robotic missions. It couples structured mission contracts (coverage, temporal evidence, reporting, resource constraints) with causal, resource-accounted execution over simulated UAV workloads. Two case studies—multi-phase Lake/Depot missions and dynamic-urgency missions with varied connectivity and battery—are used to compare fixed and adaptive perception strategies. The main finding is a large discrepancy between frame-level perception accuracy and mission success: in the Depot study, a configuration with 87.5% full-bank recall completes only 1 of 16 missions, while a lower-recall configuration (80.7%) completes 14 missions. An ablation shows that ignoring execution-induced observation loss overestimates mission success and can reverse strategy rankings. Replays with Jetson AGX Orin profiles change the relative success of local vs. remote strategies; intent-driven stopping reduces modeled task energy by up to 28% without reducing mission success.","tokens_in":23894,"tokens_out":3106,"duration_ms":25192,"significance":"If the results hold, EmbodiedIntentBench provides a principled and extensible methodology for evaluating runtime computation decisions against explicit mission objectives, moving beyond isolated perception accuracy or execution efficiency. The benchmark's causal execution model captures a critical coupling between computation latency and observation availability that is often overlooked. The empirical demonstrations show that high-recall perception can lead to very poor mission outcomes due to missed observations, and that resource conditions and connectivity significantly affect strategy effectiveness. This could influence how robotic sensing systems are designed and assessed, particularly in time-critical and resource-constrained applications. The benchmark is accompanied by publicly released code and data for the dynamic study, enhancing reproducibility.","major_comments":[{"comment":"The headline result—Mask R-CNN union (87.5% recall) completes 1/16 missions vs. tiled YOLO11s (80.7% recall) completes 14/16—is demonstrated under the serial blocking execution model (Eq. 5). The authors' own causal-execution ablation (Fig. 6b) on the dynamic-urgency study shows that switching to a no-skip protocol reverses the ordering of two strategies, indicating sensitivity to this assumption. Because no comparable ablation is performed on the multi-phase Depot study, it is unclear whether the dramatic 1/16 vs. 14/16 gap would shrink or persist under alternative execution models (e.g., asynchronous pipelining). The authors acknowledge the limitation (§2.4, §A.4) but do not quantify its impact on the Depot result. Please provide an ablation for Depot (or at least a clear discussion of bounds on the gap under non-blocking execution) to ensure the central empirical claim is not overstated.","section":"§5.1, Table 3","authors_acknowledge":"The no-skip protocol is an analytical ablation rather than a realizable parallel-execution implementation.","authors_acknowledge_p":0.66},{"comment":"The quality estimates used by the adaptive policies exhibit large errors (e.g., rotated tiled YOLO11s predicted recall 0.007 vs. observed 0.785; CABiNet predicted 0.548 vs. observed 0.000). The paper acknowledges this uncertainty (§4.3), but the adaptive policies rely on these estimates for decision-making. Consequently, the finding that adaptive strategies do not outperform fixed best configurations (Sec. 5.1) may partly reflect poor quality estimation rather than an inherent limitation of adaptation. Please clarify the contribution of the estimation method to the adaptive results, and discuss whether alternative quality predictors could change the conclusions about adaptive strategies.","section":"§4.3, Fig. 4b","authors_acknowledge":"These discrep- ancies characterize the uncertainty in quality estimates available to adaptive computation-selection strategies.","authors_acknowledge_p":0.68}],"minor_comments":[{"comment":"The definition of the next eligible observation uses j = min { k > i | t_k ≥ f_i }. Please clarify whether it is possible that f_i < t_i (completion before the next capture time) and how the model handles this case (e.g., immediate processing of the next capture).","section":"§2.4, Eq. (5)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The column header 'Reserve' is ambiguous; consider renaming to 'Battery reserve satisfied' or 'Compliance' for clarity.","section":"Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The dual-scale axis for recall and missions completed is clear, but the font size in the bar labels is small when printed. Increasing the font or providing a supplementary table would improve readability.","section":"Fig. 4(a)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The term 'full-bank target recall' is used without definition; consider adding a brief parenthetical explanation (e.g., 'recall over all visible target observations in the precomputed prediction bank').","section":"Abstract, line [b6]","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"In the description of resource profiles, the sentence 'Onboard inference is modeled at 15 W' might be clearer if it states that this includes GPU/SOC power for typical embedded platforms, as later referenced.","section":"§4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption of Fig. 6 refers to '648 matched policy–condition executions per setting' in panel (a) and '72' in panel (b). The relationship between these numbers (nine policies × 72 conditions = 648) could be explicitly stated for clarity.","section":"§5.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement 'Predictions, network transfers, loading costs, and downstream budgets remain fixed' when describing latency multipliers is helpful. It may be worth adding that the energy scaling applies only to the modeled local compute energy, not to the remote server or flight energy.","section":"Appendix A.4, line [b572]","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'modeled' for remote execution and network costs throughout; it is clear from context, but a one-sentence reminder in the main text that these are not physically deployed measurements would strengthen reproducibility assurance.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper delivers a genuinely useful benchmark framework and a clear empirical point: perception accuracy alone is a poor proxy for mission success when computation delays cause observation loss. The Depot result—Mask R-CNN union at 87.5% recall completing 1 of 16 missions while tiled YOLO11s at 80.7% recall completes 14—is eye-catching and the authors back it with controlled replay, multiple ablations, and honest limitation statements. That alone makes this worth reading for anyone building runtime adapters for mobile robots.\n\nThe soft spot is the one the stress-test note flags. The dramatic gap depends on the serial blocking execution model (Eq. 5), where high-latency inferences skip most captures. The authors' own no-skip ablation in the dynamic study reverses strategy rankings, but they did not run that ablation on the Depot multi-phase scenario where the headline number lives. They acknowledge the limitation in §2.4 and §A.4, but the central evidence for the paper's most striking claim is therefore conditional on an assumption that real systems often avoid via pipelining or parallel inference. I don't think this undermines the broader conclusion—latency still matters, mission success still diverges from frame-level recall—but it does mean the 1/16 vs 14/16 ratio is not a robust fact about those perception models in realistic deployments. It is a fact about those models under one particular, consequential execution policy.\n\nA secondary concern: the quality estimators used by adaptive policies are quite poor (rotated tiled YOLO11s predicted recall 0.007 vs observed 0.785; CABiNet predicted 0.548 vs observed 0.0). That limits what we can conclude about the adaptive strategies' effectiveness, and the paper could say more about the implications.\n\nOverall: solid engineering, clean evaluation, useful framework. The central insight holds. The headline number is real but fragile, and the authors are honest about its scope. This deserves a serious peer review—the framework is publishable, and the limitations are discussable rather than fatal.","headline":"Well-built benchmark that cleanly demonstrates perception-accuracy vs. mission-success divergence, but the headline 1/16 vs 14/16 gap rests on a serial blocking execution model whose impact the authors acknowledge without quantifying in that scenario.","tokens_in":24369,"tokens_out":1659,"duration_ms":17047},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"In mobile robotic sensing, a perception configuration with 87.5% recall completed only 1 of 16 missions, while a lower-recall configuration completed 14.","keywords":["embodied robotic missions","intent-driven adaptation","runtime computing systems","mission-level benchmarking","cyber-physical systems","UAV sensing","perception latency","causal execution"],"falsifier":"Deploy the evaluated perception configurations (Mask R-CNN union vs. tiled YOLO11s) on a real UAV executing the same multi-phase mission contracts and measure mission completion rates. If the higher-recall configuration achieves equal or higher mission success than the lower-recall one under realistic serial or pipelined execution, the central claim of accuracy–success decoupling would be weakened.","tokens_in":24958,"feed_emoji":"🚁","tokens_out":3405,"duration_ms":32123,"temperature":0.7,"pith_summary":"This paper argues that mission success in mobile robotic missions cannot be predicted from perception accuracy alone, because computation delays causally reduce the number of observations that can be processed during physical motion. The authors present EmbodiedIntentBench, a benchmark that connects structured mission contracts (covering evidence, timing, reporting, and resources) with causal, resource-accounted execution over recorded UAV trajectories. Their experiments show that a high-recall but slow configuration (Mask R-CNN union) completes only 1 of 16 missions, while a faster, lower-recall configuration (tiled YOLO11s) completes 14. An ablation reveals that ignoring computation-induced observation loss overestimates mission success and reverses the ordering of runtime strategies. The benchmark demonstrates that runtime evaluation must account for how computation timing affects observation availability and cumulative mission evidence.","feed_headline":"87.5% recall fails 15/16 missions; 80.7% recall succeeds 14/16","feed_subtitle":"Computation delay blocks observation opportunities, decoupling perception accuracy from mission completion in UAV sensing.","key_machinery":"Two mechanisms carry the argument. (1) Causal, resource-accounted execution (Eq. 5): a logical execution clock advances with processing, and the next eligible observation is the first capture whose timestamp falls at or after the completion time of the current blocking processing interval. This serial model captures how latency causes observation loss. (2) Structured mission contract (Eq. 1): a JSON-based specification of goals (coverage, repeated confirmation, fresh revisits), temporal constraints (deadlines per request and phase), reporting obligations (correct and unique reports, false-detection limits), and resource limits (battery reserve, communication). Together they enable evaluation of whether accumulated evidence and reports satisfy the intended mission under physical and resource constraints.","core_discovery":"The central claim is that mission-level success in mobile robotic sensing is decoupled from frame-level perception accuracy due to the causal effect of computation latency on observation availability. In the multi-phase Depot study, the authors compare nine fixed perception configurations over 16 mission conditions. Mask R-CNN union achieves the highest full-bank target recall (0.875) but completes only 1 of 16 missions, while tiled YOLO11s (0.807 recall) completes 14 missions. The high-latency configuration (6.91 s inference) processes roughly one in four observation opportunities under a two-second capture interval, limiting the evidence available for phase completion. Further, ignoring execution-induced observation loss by switching to a no-skip counterfactual increases aggregate mission success and reverses the relative ordering of quality-aware routing and fixed remote execution, confirming that causal observation accounting is essential for fair strategy comparison.","pith_inferences":["The same decoupling between perception accuracy and mission success likely applies to other mobile robotic platforms (ground vehicles, underwater robots) where observation opportunities are shaped by physical motion and computation latency. Recasting existing benchmarks with causal execution could reveal similar overestimates of mission viability.","Although the benchmark uses a serial blocking execution model, even asynchronous pipelines with bounded buffers may exhibit observation loss if downstream stages cannot keep up with capture rate; the paper's ablation suggests that the serial model is a conservative bound that may still capture the dominant effect in constrained systems.","The fixed-trajectory methodology isolates computation effects, but in closed-loop systems, navigation decisions could compensate for or exacerbate observation loss. An extension to closed-loop control could test whether the accuracy–success decoupling persists when the UAV can alter its path to reacquire missed observations."],"forward_implications":["Runtime adaptation strategies for UAV sensing should be evaluated under causal execution rather than assuming all observations are available; otherwise mission success may be overestimated and strategy rankings may be reversed.","High-accuracy but slow perception models can be detrimental to mission success in time-sensitive tasks, even when they improve individual predictions, because they cause observation loss.","Intent-driven adaptation (stopping perception when the required number of reports is already obtained) can reduce onboard energy consumption by up to 28% without reducing mission success on selected embedded profiles.","The benchmark's diagnostic measures (phase completion, evidence pipeline breakdown, requirement-level failure attribution) make it possible to identify whether a strategy fails due to insufficient evidence, untimely execution, reporting deficiencies, or resource violations."],"supporting_citations":[{"why":"MAVBench establishes the importance of evaluating robotic computation in the context of physical execution and resource consumption, providing a baseline for the interaction between computation and vehicle dynamics.","marker":"[14]"},{"why":"Streaming Perception introduces streaming accuracy as a metric that accounts for inference latency and observations missed during processing, motivating the need for latency-aware evaluation.","marker":"[17]"},{"why":"LIBERO-Dynamic incorporates inference latency in dynamic manipulation scenarios, showing that environment evolution during computation affects task outcomes, a direct precursor to causal execution.","marker":"[21]"},{"why":"ApproxDet adapts perception configurations to content, latency requirements, and resource contention, providing a reference adaptive strategy that the paper benchmarks against.","marker":"[43]"},{"why":"DEEP evaluates perception-computation allocation through its downstream decision value under resource budgets, supporting the idea that perception quality should be judged by mission-level impact rather than isolated accuracy.","marker":"[20]"}],"fun_headline_variants":["87.5% recall fails 15/16 missions; 80.7% succeeds 14/16","Computation delay, not perception accuracy, kills missions","Top recall UAV finishes 1/16 missions; lower recall 14/16","Mission success decoupled from frame-level recall"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark assumes a serial, blocking execution model where perception processing prevents subsequent observations from being processed during that interval; if real systems use asynchronous pipelining or parallel processing that can absorb more observations, the observed tradeoffs between latency and observation loss could change.","fun_headline_variants_meta":{"raw":{"variants":["87.5% recall fails 15/16 missions; 80.7% succeeds 14/16","Computation delay, not perception accuracy, kills missions","Top recall UAV finishes 1/16 missions; lower recall 14/16","Mission success decoupled from frame-level recall"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7.8e-05,"raw_usage":{"cost":7.8315776e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.8315776e-05,"upstream_inference_prompt_cost":2.5308416e-05,"upstream_inference_completions_cost":5.300736e-05},"total_tokens":1800,"prompt_tokens":984,"completion_tokens":816,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":736}},"tokens_in":984,"tokens_out":816,"cost_source":"openrouter_usage","duration_ms":8481,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":736,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-10T22:43:56.042795+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Deploy the evaluated perception configurations (Mask R-CNN union vs. tiled YOLO11s) on a real UAV executing the same multi-phase mission contracts and measure mission completion rates. If the higher-recall configuration achieves equal or higher mission success than the lower-recall one under realistic serial or pipelined execution, the central claim of accuracy–success decoupling would be weakened.","supporting_citations":[{"cited_title":"MAVBench: Micro aerial vehicle benchmarking","cited_arxiv_id":null,"evidence_quote":"MAVBench establishes the importance of evaluating robotic computation in the context of physical execution and resource consumption, providing a baseline for the interaction between computation and vehicle dynamics."},{"cited_title":"Towards streaming perception","cited_arxiv_id":null,"evidence_quote":"Streaming Perception introduces streaming accuracy as a metric that accounts for inference latency and observations missed during processing, motivating the need for latency-aware evaluation."},{"cited_title":"LIBERO-Dynamic: Benchmarking Robotic Manipulation in Dynamic Environments","cited_arxiv_id":"2610.24579","evidence_quote":"LIBERO-Dynamic incorporates inference latency in dynamic manipulation scenarios, showing that environment evolution during computation affects task outcomes, a direct precursor to causal execution."},{"cited_title":"ApproxDet: Content and contention- aware approximate object detection for mobiles","cited_arxiv_id":null,"evidence_quote":"ApproxDet adapts perception configurations to content, latency requirements, and resource contention, providing a reference adaptive strategy that the paper benchmarks against."},{"cited_title":"The Decision Value of Perception Compute","cited_arxiv_id":"2609.35910","evidence_quote":"DEEP evaluates perception-computation allocation through its downstream decision value under resource budgets, supporting the idea that perception quality should be judged by mission-level impact rather than isolated accuracy."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}