{"id":"b89b0073-f53a-4818-a64f-bf1fb4114236","arxiv_id":"2610.38199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task VLA driven by text commands can grasp and moor underwater on real ROV hardware, reaching 93.3% success including partials in a pool and 66.7% on a shipwreck mooring task.","lead":"This paper trains robot policies by imitation on 1,616 underwater demonstrations and deploys them on a BlueROV2 in a pool, a marina, and at a WWII shipwreck in Malta. The best language-commanded multi-task model scored 93.3% on graded rollouts including partial successes, while a small single-task policy beat a much larger generalist model in the field.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Headline pool rates rest on 'gradable'-only denominators and a partial-success definition that counts wrong-color grabs; the 93.3% claim does not yet separate protocol from policy competence.","rationale":"The reader's weakest assumption identified the same load-bearing concern: the reported success rates measure the evaluation protocol as much as, or more than, policy competence, because ungradable rollouts are excluded without counts, partial successes are loosely and inconsistently defined, and labeling is done by the same team. I agree with that diagnosis and do not find a separate, stronger objection. I also give credit where the paper has independent support: the arithmetic is internally consistent (1,260 pool rollouts match 14 models times 90; 508 marine rollouts match Table II), the authors acknowledge the SmolVLA gap is likely latency and data quantity rather than architecture (b131), and they acknowledge the pool lacks realistic currents (b117). The main outstanding issue is specifically that the headline numbers, especially 93.3% and the 10-of-14 comparison, include wrong-color grabs as partial successes and exclude an unreported set of ungradable rollouts. That concern does not overturn the central direction of the paper, since many models do beat the visual-only baseline and the marine ACT results are reported per condition, but it does mean the precision of the headline claims is unsupported until the evaluation protocol is reported transparently. The proposed re-scoring check would settle whether the concern actually moves the numbers; if it does not, the protocol criticism is cosmetic, but if it does, the central claim needs to be restated with the more conservative rates.","tokens_in":14641,"tokens_out":5000,"duration_ms":36744,"concrete_test":"Using the open-sourced BLUEMBODIEDKIT rollout logs, re-score every pool and marine rollout under three denominators: gradable-only success as published; all rollouts with ungradable trials counted as failures; and correct-color-only success for grab tasks with wrong-color grabs counted as failures. Report the resulting ATTITUDE overall rate, the ACT 47.9% pooled marine rate, and the number of models beating baseline under each denominator. If any headline rate shifts by more than 5 percentage points, or if the 10-of-14 comparison changes, the reported rates are protocol-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Contribution 1, the paper's strongest claim, is that a single text-commanded multi-task VLA works on real underwater hardware (abstract; Contribution 1). The load-bearing evidence is the pool ablation's headline rate, 93.3% including partials for the ATTITUDE model (Table I; b110). That rate is computed only over rollouts the authors classed as 'gradable', with exclusion counts and criteria unreported (abstract b7; Table I caption), and every rollout is hand-labeled success/partial/failure by the same team that built the pipeline (Section IV.A). The partial-success definition compounds the problem: for the grab task, a partial is explicitly 'grabbing a rod of the wrong color' (Table I caption), while for moor and remove ring no partial definition is given. For ATTITUDE, grab success with partials is 96.7%, but correct-color success is only 33.3%; 19 of 30 rollouts grabbed the wrong-color rod and still counted toward the headline. Because Contribution 1 defines a VLA partly by correct language-command execution, counting wrong-color grabs as partial successes means the 93.3% headline and the '10 of 14 beat baseline' result are dominated by outcomes that fail the language-command component of the claim. This is not an internal contradiction, but it means the headline precision is not yet supported by the reported evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper tackles imitation learning for underwater manipulation with an ROV. It collects two real-world datasets (1,616 pool episodes in calm and human-agitated water; 748 marine episodes at the Coralita X127 wreck and Manoel Island Marina), trains 14 DynamicVLA configurations for the pool and single-task ACT and SmolVLA policies for the marine case study, and evaluates them on tasks that include grab, moor, and remove-ring. The headline pool result is that 10 of 14 models beat a visual-only baseline over 1,260 gradable rollouts, with the best model (ATTITUDE) reaching 93.3% success including partials; the marine case study reports ACT at 47.9% vs. SmolVLA at 7.9% pooled success, with the largest effect on mooring. The central claim is that this is the first multi-task, text-commanded VLA deployed on real underwater hardware, plus an open-source kit (BLUEMBODIEDKIT) for data collection, training, and deployment.","tokens_in":14929,"tokens_out":7255,"duration_ms":47554,"significance":"If the headline numbers withstand audit, this is a substantial field-robotics contribution: the dataset sizes are large for underwater manipulation, the telemetry-ablation study is systematic, the depth-controller action split is a sensible embodiment-specific adaptation, and the open-sourced pipeline is a genuine community asset. The authors are also honest about the main confounds, stating that the SmolVLA gap probably reflects latency and data quantity (Section IV.C) and that the pool lacks realistic currents (Section IV.B). The arithmetic is internally consistent (14 models x 90 rollouts = 1,260; the Table II row counts sum to 508), which is a good sign. The main weakness is not the experimental design but the evaluation protocol: the reported rates depend on unstated 'gradable' exclusions and on a partial-credit definition that counts wrong-color grabs, so the precision of the headline claims is currently not supported by the paper's own reporting.","major_comments":[{"comment":"All headline success rates, including 93.3% and 47.9%, are computed over 'gradable' rollouts, but the manuscript never defines 'gradable', states the number of excluded rollouts, or gives the exclusion criteria. If ungradable rollouts are disproportionately failures (e.g., hardware loss, camera occlusion, telemetry dropout), every reported rate is inflated. Please report the full numbers of rollouts collected vs. excluded per model, task, and site, and provide the written gradability criteria used during labeling.","section":"Abstract and Table I/II captions","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"For the grab task, a partial success is explicitly a grab of the wrong-color rod. The ATTITUDE model's grab-with-partials rate of 96.7% is composed of 10 correct-color and 19 wrong-color grabs out of 30, so the strict correct-color success rate is only 33.3%. Because Contribution 1 defines a VLA partly by correct language-command execution, counting wrong-color grabs as partial successes materially inflates the 93.3% headline and the 10-of-14 comparison. Report strict task success as the primary metric, present wrong-color grabs separately, and quantify how the 'beats baseline' count changes when partials are excluded.","section":"Table I caption and row #3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Rollout outcomes are hand-labeled as success/partial/failure by the same team that built the pipeline, and no partial definition is given for moor or remove-ring, nor any inter-annotator agreement or blind-labeling procedure. Since the paper's central empirical claims rest entirely on these labels, please specify the labeling rubric for every task, state who labeled the rollouts, and report agreement measures if more than one labeler was used.","section":"Section IV.A and Table II","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The table header 'succ. part. % % w/ part.' is ambiguous, and for the Moor and Remove Ring rows it is not clear which numbers are strict success, partial success, or success-with-partials. Redraw the table with explicit per-task columns for strict success, partial, and combined rates.","section":"Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'incredible recovery trajectories' is subjective; replace it with a quantitative description, such as the fraction of rollouts that recovered after an initial miss or the measured deviation during recovery maneuvers.","section":"Section IV.C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'world's first' claim is stronger than the evidence supplied; the paper should say 'to our knowledge, the first reported' and briefly describe the scope of the literature search used to support the novelty claim.","section":"Contribution 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The disturbance channel in Eq. (1) depends on the thruster-to-acceleration mapping Phi, but the paper does not state how Phi was estimated. Since the raw-delta models and delta-token models consume this channel, please describe the identification procedure or state the exact offline fit used.","section":"Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract says 10 models 'outperformed the baseline' without specifying whether the comparison uses strict success or success-with-partials; state the comparison basis explicitly.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The one thing to know: this is a real field-robotics paper with a real deployment — a text-commanded, multi-task VLA actually running on a BlueROV2 at a WWII wreck and a marina, day and night, plus an open-sourced data/training/deployment kit. The internal arithmetic is consistent (1,260 = 90 × 14; 508 = 432 ACT + 76 SmolVLA; the pooled rates in Table II recompute exactly). The central claim, that observation-space and action-space adaptation matters at least as much as the backbone when moving BC underwater, is directionally supported. I would send this to review, not desk-reject it.\n\nWhat is genuinely new: by the authors' own survey, prior underwater BC is single-task (AquaBot, UMI-Underwater) and prior underwater VLAs were not evaluated on hardware (USim/u0, UnderwaterVLA). So \"first multi-task VLA on a real ROV, to our knowledge\" is credible. The 14-configuration proprioception ablation is systematic, the depth-setpoint action split (letting a PID own the vertical axis) is a sensible adaptation, and the recovery-trajectory data collection is a practical touch. The authors also name their own confounds — SmolVLA's gap is likely latency and data volume (b131), not an architectural ceiling, and the pool lacks currents (b117).\n\nThe soft spots, in order of size. First, the headline. Table I defines a grab partial as grabbing a rod of the wrong color. For the best model, ATTITUDE, strict correct-color grab success is 33.3%; counted with wrong-color grabs, it is 96.7%. That is 19 of 30 rollouts grabbing the wrong color — roughly a fifth of the 93.3% overall. Since a VLA is defined here partly by executing the text command, a wrong-color grab is a command-execution failure earning partial credit. The paper is transparent — the strict column and the Fig. 8 confusion matrix are there — so this is a framing problem, not a hidden result, but the abstract's precision outruns the protocol.\n\nSecond, every rate is computed over 'gradable' rollouts, with exclusion counts and criteria unreported, and outcomes are hand-labeled by the same team that built the pipeline (Section IV.A). Partial is undefined for moor and remove. A revision can fix this by reporting exclusions, an inter-rater check, and per-condition confidence intervals.\n\nMinor: n=30 per task per model leaves \"10 of 14 beat baseline\" directional rather than decisive; the site-agnostic mooring conclusion rests on n=21 and n=19; and the Phi mapping in Eq. (1) is never estimated, so the raw-delta disturbance channel is under-specified. The stress-test arithmetic holds; its word \"dominated\" is too strong, since wrong-color grabs are about a fifth of the overall headline rather than most of it.\n\nWho gets value: anyone in underwater manipulation, VLA adaptation, or evaluation methodology for field robotics. A reading group would get a genuine hour debating what \"success\" should mean for language-conditioned policies. Recommendation: conditionally accept — the capability claim is probably real, but the protocol needs revision before the headline numbers can be trusted.","headline":"Genuine field deployment with open data and tools, internally consistent numbers, and a credible first-multi-task-VLA claim — but the 93.3% headline rests in part on wrong-color grabs counted as partials.","tokens_in":15644,"tokens_out":8599,"duration_ms":52701},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper argues that adapting what an underwater robot observes and what it commands matters more than which model you train, and shows a text-commanded multi-task policy working on a real remotely operated vehicle at a Mediterranean…","keywords":["imitation learning","behavior cloning","vision-language-action models","underwater manipulation","remotely operated vehicle","domain shift","action chunking transformer","marine robotics"],"falsifier":"Blind-regrade a random sample of the pool and field rollouts under a strict rule that a grab only counts if the commanded colour is the one grasped, and report how many rollouts were excluded as ungradable and why; if the best pool configuration's lead over the images-only baseline, or the Action Chunking Transformer's lead over the generalist policy, shrinks to within the reported confidence intervals under that rule, the central claim does not hold.","tokens_in":15964,"feed_emoji":"🌊","tokens_out":3127,"duration_ms":23739,"temperature":0.7,"pith_summary":"The question the authors set out to answer is whether the generalist vision-language-action policies that now work on land survive the move to a floating robot in the sea, where currents shove the vehicle, the camera view shifts between every action, and light and turbidity distort what the model sees. Their answer is that the model is not the main lever: what you feed it to observe and how you let it act are. They collect 1,616 teleoperated episodes in a pool, train fourteen variants of one policy architecture that differ only in their sensor inputs, and test them on real hardware, where ten variants beat a camera-only baseline and the best reaches 93.3 percent when partial successes are counted. A separate field study carries two policies to the wreck of the Coralita X127 and to a Maltese marina, where a small single-task policy substantially outperforms a much larger generalist one. The claim worth caring about is that no underwater-pretrained backbone is needed: a policy trained on terrestrial data, given the right attitude readings and a depth set-point handled by a feedback controller, can latch onto a shipwreck. If that holds, marine archaeology and conservation groups get a practical path to autonomous manipulation built from off-the-shelf parts and an open-sourced toolkit.","feed_headline":"A text-commanded robot policy works at a real shipwreck","feed_subtitle":"Fourteen model variants, one pool, two field sites: what the robot is allowed to sense mattered more than which model ran.","key_machinery":"Three pieces of adaptation carry the argument. The first is a modular observation space cut from the ROV's 19-channel MAVLink telemetry row: an ATTITUDE block (roll $\\phi$, pitch $\\theta$, yaw $\\psi$), an IMU block that adds body-frame accelerations, EKF-fused angular rates, vertical velocity and three vibration channels, and a thruster-PWM block of the six commanded motor outputs. The second is a disturbance channel: the vehicle's expected acceleration is modelled as $a(t) = G_{\\text{orient}}(t) + \\Phi u(t) + \\Delta(t)$, and the residual $\\Delta(t)$ between predicted and measured acceleration is treated as the environmental disturbance; because a single frame carries no turbulence history, a window of $\\Delta$ frames (tested from 1.2 to 6.0 seconds) is compressed through the FAST tokenizer so the policy sees a compact signature of the sloshing. The third is an action split: a six-dimensional action $a_t = [v_x, v_y, \\omega_z, z_{\\text{ref}}, c_{\\text{tilt}}, e_{\\text{ref}}]$ in which vertical motion is a depth set-point handed to a closed-loop PID controller reading the on-board pressure sensor, rather than a raw vertical command. That last choice is what the authors credit for the vehicle holding station on a sloping shipwreck deck and for the recovery manoeuvres that appear when depth is in the loop.","core_discovery":"The authors claim, to their knowledge, the first successful deployment of a multi-task vision-language-action model underwater on real hardware, where a single model executes several tasks and is told which one to execute by a text command. The evidence for the multi-task claim comes from the pool: fourteen DynamicVLA variants, differing only in which of a 19-channel telemetry stream they observe and how disturbance is represented, are graded over rollouts on three tasks (grab a rod of a commanded colour, moor on a post, remove a ring). Ten of the fourteen beat the images-only baseline once partial successes are counted, and the best configuration, plain attitude (roll, pitch, yaw) trained on both calm and turbulent episodes, reaches 93.3 percent overall at a 95.2 millisecond forward-pass latency. Crucially, more sensing was not better: adding IMU and thruster-PWM channels helped time-to-completion but not success, and the authors conclude that a compact observation space balances prediction speed against task success. The field evidence is a different comparison: a single-task Action Chunking Transformer versus a generalist SmolVLA policy, trained on site, over 508 gradable rollouts, where ACT succeeds on 47.9 percent pooled against 7.9 percent, and reaches 66.7 percent on mooring a rope at the shipwreck. The unifying claim the authors draw from both is that effective underwater deployment depends on how the action and observation spaces are adapted to the physical environment at least as much as on the choice of model, and that a backbone trained only on terrestrial data suffices.","pith_inferences":["The pool ranking and the field ranking disagree about which policy class wins, and the authors never run a multi-task model at the shipwreck; a reader should treat the 93.3 percent pool figure and the 47.9 percent field figure as measuring different things, and a single experiment running the best pool configuration on the wreck would be the natural bridge.","Because the authors report success on hand-labelled rollouts over the subset they class as gradable, the headline rates are only comparable across models if the gradable filter and the partial-success rule were applied identically to every configuration, which the paper does not show; a blind regrade of a sample of the rollouts would settle this cheaply.","If the argument that observation and action spaces dominate model choice generalises, the next useful ablation is the reverse of the one run here: fix the best observation space and vary the backbone, which would separate what the spacecraft telemetry did from what the architecture did.","The disturbance-residual channel is the paper's most transferable idea, since the same expected-versus-measured acceleration arithmetic works on any vehicle with an autopilot, and it would be interesting to see whether it helps in a strong current where the tokenised window carries more signal than it did in a pool."],"forward_implications":["A single text-commanded policy can be trained on underwater teleoperation data and then run on a tethered, thruster-driven vehicle with commodity hardware, without any underwater-pretrained vision backbone.","Reading a few attitude channels from the autopilot is worth more than adding many sensors: the smallest proprioceptive observation space tested gave the best success rate while larger IMU and thruster-PWM spaces mainly bought faster task completion.","Splitting vertical control out of the learned policy and into a depth set-point tracked by a PID controller lets a policy operate at varying depths on a sloping wreck rather than only at a fixed station.","Small single-task policies remain strong competition for generalist models in the field: the Action Chunking Transformer beat the larger generalist policy on every task family, which the authors attribute to more frequent replanning and more task-specific data rather than to architecture.","Success degrades at night and on harder target geometry, so light budget and target shape, not just model choice, set the practical operating envelope for field deployments."],"supporting_citations":[{"why":"Supplies the DynamicVLA architecture that all fourteen pool models modify with new observation and action spaces.","marker":"[22]"},{"why":"Supplies the Action Chunking Transformer, which is the single-task policy used in the marine field study and the architecture that wins there.","marker":"[2]"},{"why":"Supplies SmolVLA, the generalist baseline the Action Chunking Transformer is measured against at the wreck and marina.","marker":"[21]"},{"why":"Supplies the FAST tokenizer used to compress windows of the estimated disturbance residual into tokens the policy can consume.","marker":"[29]"},{"why":"Provides the prior underwater behaviour-cloning result the pool findings are checked against, including the report that visual degradation hurts underwater perception.","marker":"[3]"},{"why":"Provides the closest prior attempt to move land-trained manipulation policies underwater, framing the gap the multi-task text-command capability is meant to fill.","marker":"[4]"},{"why":"Supplies the LeRobot dataset format the released data collection, annotation and rollout pipeline is built on.","marker":"[30]"}],"fun_headline_variants":["Sensing choice mattered more than model choice for underwater policies","14 policies, 1,260 pool trials: the best hit 93.3% with partials","At a WWII wreck, a single-task policy beat the generalist 47.9% to 7.9%","Caveat: multi-task evidence is pool-only; field test pits ACT vs SmolVLA","Open-sourced ROV kit used for 14 policies and 508 field rollouts"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that the reported success rates measure how well the policies actually manipulate, which requires that scoring only the rollouts the authors judged gradable and counting a wrong-colour grab as a partial success did not systematically favour the better-performing configurations.","fun_headline_variants_meta":{"raw":{"variants":["Sensing choice mattered more than model choice for underwater policies","14 policies, 1,260 pool trials: the best hit 93.3% with partials","At a WWII wreck, a single-task policy beat the generalist 47.9% to 7.9%","Caveat: multi-task evidence is pool-only; field test pits ACT vs SmolVLA","Open-sourced ROV kit used for 14 policies and 508 field rollouts"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"cost":0.000454181,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000454181,"upstream_inference_prompt_cost":0.000109109,"upstream_inference_completions_cost":0.000345072},"total_tokens":3095,"prompt_tokens":1199,"completion_tokens":1896,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1791}},"tokens_in":1199,"tokens_out":1896,"cost_source":"openrouter_usage","duration_ms":12793,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1791,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-31T08:46:50.501326+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Blind-regrade a random sample of the pool and field rollouts under a strict rule that a grab only counts if the commanded colour is the one grasped, and report how many rollouts were excluded as ungradable and why; if the best pool configuration's lead over the images-only baseline, or the Action Chunking Transformer's lead over the generalist policy, shrinks to within the reported confidence intervals under that rule, the central claim does not hold.","supporting_citations":[{"cited_title":"DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation","cited_arxiv_id":"2601.22153","evidence_quote":"Supplies the DynamicVLA architecture that all fourteen pool models modify with new observation and action spaces."},{"cited_title":"Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware","cited_arxiv_id":"2304.13705","evidence_quote":"Supplies the Action Chunking Transformer, which is the single-task policy used in the marine field study and the architecture that wins there."},{"cited_title":"SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics","cited_arxiv_id":"2506.01844","evidence_quote":"Supplies SmolVLA, the generalist baseline the Action Chunking Transformer is measured against at the wreck and marina."},{"cited_title":"FAST: Efficient Action Tokenization for Vision-Language-Action Models","cited_arxiv_id":"2501.09747","evidence_quote":"Supplies the FAST tokenizer used to compress windows of the estimated disturbance residual into tokens the policy can consume."},{"cited_title":"Self-Improving Autonomous Underwater Manipulation","cited_arxiv_id":"2410.18969","evidence_quote":"Provides the prior underwater behaviour-cloning result the pool findings are checked against, including the report that visual degradation hurts underwater perception."},{"cited_title":"Umi-underwater: Learning underwater manipulation without underwater teleoperation,","cited_arxiv_id":null,"evidence_quote":"Provides the closest prior attempt to move land-trained manipulation policies underwater, framing the gap the multi-task text-command capability is meant to fill."},{"cited_title":"LeRobot: An Open-Source Library for End-to-End Robot Learning.arXiv preprint arXiv:2602.22818, 2026","cited_arxiv_id":null,"evidence_quote":"Supplies the LeRobot dataset format the released data collection, annotation and rollout pipeline is built on."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}