{"id":"9b43329d-8632-459e-820c-9d211fa0bd99","arxiv_id":"2608.13474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Task progress is linearly decodable from the residual stream of π0.5 and can serve as a label-free out-of-distribution detector for deployed robot policies.","lead":"Researchers trained linear probes on the internal activations of a vision-language-action robot model and found that the fraction of a task remaining can be read off with a simple line. The same signal can flag when a robot has drifted out of distribution, without needing labeled failure examples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Progress labels are normalized time; the paper never decouples time from task state, and the language-sensitivity probe is trained with τ=1.0 swap labels, so the semantic-progress claim is not yet supported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the progress label is normalized time, so linear decodability may reflect a time-index or kinematic-phase correlate rather than semantic task progress. The raw-observation control is not decisive because the VLA's residual stream could linearize a phase variable that raw pixels do not expose. The language-counterfactual experiment was intended to separate these readings, but because the swapped prompts are labeled τ = 1.0 during training, the observed swap sensitivity is partially manufactured. The paper's own OOD results mitigate the pure-wall-clock reading: under perturbation, predicted τ stalls while rollout time advances, which a naive time decoder would not do. However, that still leaves open the possibility that the probe tracks a learned kinematic or state-progression phase rather than task completion in a semantic sense. The limitations section is honest about the time/completion conflation and the proxy labels, which supports a conditional rather than a rejecting verdict. The missing code and data remain secondary reproducibility concerns. A loitering experiment that holds task state fixed while letting time and low-level state vary would directly settle whether the probe tracks time or task progress. If the probe stays high during loitering, the central semantic claim is substantially strengthened; if it decays, the headline claim should be reframed as decoding a time-correlated state phase. The verdict should remain conditional on this and related controls, matching the reader's assessment.","tokens_in":17722,"tokens_out":9892,"duration_ms":108576,"concrete_test":"Run a loitering experiment in VLABench: take an in-distribution Add Condiment rollout and, at several progress points, replace a block of expert actions with task-irrelevant arm motions for 5–10 replans, so simulator time advances while the task state (object positions, grasp status) is unchanged. Decode τ with the frozen layer-0 probe. If predicted τ continues to decrease with t/T during the loitering blocks, the linear signal is time or kinematic phase, not task progress; if it stays at the pre-loiter level and Vτ does not trigger, the semantic-progress reading is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.3 defines τ = 1 − t/T, so every probe is trained to decode normalized elapsed time in expert trajectories. A linear readout of any variable correlated with t (proprioceptive state, object/arm displacement, a phase variable in the residual stream) will look like \"progress.\" App. B.4 shows raw observations are not linearly decodable OOD, but that does not remove the confound: the VLA's embeddings may contain a linearly readable kinematic or time phase that raw pixels lack. The OOD stagnation result is partial evidence against a naive wall-clock decoder, but the paper never runs the decisive control: time advancing while task-relevant state is held fixed, or vice versa. The language result in Sec. 4.3 is weaker than the abstract implies: the contrastive probe is trained on swapped prompts labeled τ = 1.0 (a proxy for zero progress), so the reported swap sensitivity is at least partly produced by the training labels. It demonstrates that a language-sensitive direction can be trained into the probe, not that the representation inherently encodes semantic progress. The limitations section concedes both the time/completion conflation and the proxy labels, but the central interpretability and monitoring claims inherit the unresolved time-vs-semantics ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains linear probes on the residual-stream activations of the π0.5 vision-language-action model to decode task progress, defined in Sec. 3.3 as the normalized time remaining in an expert trajectory (τ = 1 − t/T). It reports that this feature is weakly decodable with high R² at early layers, that the signal is present after large-scale robotic pre-training and also, to a lesser extent, in the pretrained PaliGemma backbone, and that a contrastively trained probe at layer 10 separates original from noun-swapped language prompts. The paper further shows that the probe is not steerable and that a residual between predicted and expected progress can serve as an unsupervised out-of-distribution detector competitive with supervised baselines. The paper also formalizes weak decodability, strong decodability, and steerability for generative models.","tokens_in":1596,"tokens_out":4969,"duration_ms":74756,"significance":"If the claims are upheld, the paper offers a lightweight, label-free runtime monitor for deployed VLAs and a useful contribution to mechanistic interpretability of robot policies. The work has several genuine strengths: the weak-decodability claim is supported by a shuffled-label control (Fig. 2, right), the OOD detector is evaluated with cross-task and cross-perturbation splits with per-task and per-mode breakdowns (Tables 5–8), and the raw-observation control in App. B.4 demonstrates that the signal is not trivially present in pixels. The formal separation of weak decodability, strong decodability, and steerability is also a valuable conceptual contribution. The main blocker is that the paper's central interpretive claim — that the decoded quantity is semantic task progress — is not yet separated from a time-step correlate, and the language-sensitivity result is partly produced by the training labels.","major_comments":[{"comment":"The progress label is defined as normalized elapsed time (τ = 1 − t/T), so every probe is trained to decode a time-dependent quantity. The abstract and Sec. 5 conclude that VLAs encode semantic quantities like task progress, but the experiments cannot distinguish semantic completion from a low-level time or phase correlate, such as a step counter or positional feature in the residual stream. The raw-observation control in App. B.4 rules out decoding from pixels alone, not from a time-index feature inside the embedding. The OOD stagnation result in Sec. 4.5 is partial evidence against a naive wall-clock decoder, but the decisive control — advancing time while holding task-relevant state fixed, or holding time while changing state — is never run. The paper's own limitations section concedes this conflation, yet the abstract and conclusion still assert the semantic reading. This is load-bearing for the interpretability and monitoring claims, and it needs either a direct control experiment or a reframing of the claims to normalized time remaining rather than semantic task progress.","section":"Sec. 3.3, Eq. (3.3); Sec. 5; App. B.4"},{"comment":"The strong-decodability result under language counterfactuals is partly circular because the contrastive probe is trained on shadow forward passes under swapped prompts, with all swapped activations labeled τ = 1.0 as a proxy. The separation between original and swapped prompts is therefore imposed by the training objective rather than discovered in the representation. App. A.4 itself states that a naive counterfactual-augmented probe does not separate the prompts and that the within-pair hinge plus BCE anchor are required; this demonstrates that a language-sensitive direction can be trained into the probe, not that the representation inherently encodes semantic progress. The drop in in-distribution R² to 0.33 further weakens the interpretation of the readout as a clean progress signal. To support the claim of strong decodability, the probe should be evaluated on held-out language perturbations or swap types not used in contrastive training, or the claim should be explicitly restricted to a language-sensitive direction that can be trained into the probe.","section":"Sec. 4.3, App. A.4"},{"comment":"The OOD detector is presented as label-free, but it relies on an expected completion time E[T|ℓ] estimated from in-distribution rollouts and a threshold δ whose selection is not described. The AUROC comparisons in Tables 1 and 2 are threshold-independent, so the headline numbers are not affected by δ, but the practical claim that Vτ is a ready-to-deploy runtime monitor needs a calibration analysis, for example how δ is set per task and sensitivity of flagging quality to E[T|ℓ] estimation error. Without this, the deployment-ready framing in the abstract is stronger than what the experiments show.","section":"Sec. 4.5, Tables 1–2"}],"minor_comments":[{"comment":"The text says probes are trained on 400 episodes (100 per task) but the dataset has 10 tasks; 100 per task would be 1000 episodes. Please clarify the number of tasks and episodes used for training these probes.","section":"Sec. 4.2"},{"comment":"The claim that the feature is observable in the pretrained PaliGemma backbone would be easier to assess if quantitative R² or MAE values were reported for each model, along with error bars, rather than only the relative bar chart.","section":"Fig. 3"},{"comment":"The dataset notation Dprobe = {(z0_t, ..., zL_t, ζt)}^N_{i=0} mixes trajectory index i and timestep t; please make the indexing consistent.","section":"Sec. 3.1, Eq. (3.3)"},{"comment":"The hyperparameters for the contrastive probe (margin schedule m(t), anchor weight λ≈1.5, sharpening temperature 0.12) are given in prose but not listed in Table 3; adding them to the table would improve reproducibility.","section":"App. A.4"},{"comment":"Several references (e.g., [12], [16], and the manuscript itself) carry 2026 arXiv IDs; please verify that these are correct and that all cited preprints are publicly available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's own limitations section acknowledges both constraints that block the central claim: the time/completion conflation and the proxy labels for counterfactual training. The abstract and conclusion nevertheless assert a semantic interpretation. In my view, the underlying weak-decodability and OOD-detection results are sound and publishable, but the revision must either supply the missing decoupling experiments or substantially soften the claims. I would not reject on this basis, since the fix is within scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The real contribution is the OOD detector: a single linear probe on layer-0 activations, fed through a simple residual against expected progress, flags stalled rollouts and is competitive with supervised SAFE at episode level, including on unseen tasks and unseen perturbation types. That is genuinely useful and label-free. The weak-decodability result is well controlled — trajectory-level splits, a shuffled-label capacity sweep, and a raw-observation control showing the signal isn't trivially in pixels. Literature coverage is fair and the paper distinguishes itself from Buurmeijer et al. and SAFE clearly.\n\nThe paper is honest about its soft spots, and they matter. tau is defined as normalized time remaining (Eq. 3.3), so every probe is, by construction, decoding time. The language-swap experiment is the intended escape, but the contrastive probe is trained on swapped prompts labeled tau=1.0, a proxy that bakes in the sensitivity it then reports. The ID R2 for the contrastive probe drops to 0.33. The raw-observation control does not rule out a time/phase variable inside the residual stream. So the 'semantic progress' claim in the abstract is not supported. What is supported is that a time-to-completion signal — likely a phase or kinematic correlate — is linearly readable and useful for monitoring. That is still a good result, just framed too strongly.\n\nThe steerability negative is clean and worth having. The emergence story (present in PaliGemma, stronger after robot pretraining) is interesting but not deeply analyzed.\n\nMain practical complaints: no code or data, single model family, simulation only. Those are standard limitations but they limit how much anyone can build on this without reimplementation.\n\nBottom line: this deserves peer review. The OOD detector result is solid and the formal weak/strong/steerable distinctions are useful vocabulary. A referee should push for code release and a direct control that decouples time from task state — e.g., holding the scene fixed while advancing the clock, or vice versa — to see which variable the probe actually tracks. The abstract should also be brought in line with the limitations section.","headline":"A genuinely useful label-free OOD detector built on a linearly readable time-remaining signal, with an interpretability headline ('semantic progress') that outruns the evidence.","tokens_in":18564,"tokens_out":3305,"would_cite":true,"duration_ms":32969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single linear probe on a vision-language-action model's internal activations reads out how far a task has progressed, and can flag stalled rollouts without any failure labels.","keywords":["vision-language-action models","task progress","linear probing","mechanistic interpretability","out-of-distribution detection","steerability","residual stream","PaliGemma"],"falsifier":"Collect rollouts where the robot executes the same task at half speed, or pauses mid-trajectory while staying on track, and check whether the probe's predicted progress follows wall-clock time instead of completion fraction; if it follows time, the semantic-progress interpretation fails. Separately, execute a swapped instruction to actual completion and test whether a probe trained with the proxy label τ=1.0 still reads true progress.","tokens_in":17456,"feed_emoji":"🤖","tokens_out":6567,"duration_ms":58198,"temperature":0.7,"pith_summary":"This paper asks whether a deployed vision-language-action model (VLA) carries a readable signal of its own task progress, and answers yes. Probing the residual-stream activations of π0.5 with a single linear map, the authors recover the normalized time remaining in a trajectory, $\\tau = 1 - t/T$, with low error. The signal is present in the pretrained PaliGemma backbone, is sharpened by large-scale robot pre-training, and transfers to unseen tasks, and a probe trained only on in-distribution data flags out-of-distribution stalls at a level competitive with supervised detectors. A naive probe ignores language-prompt swaps, but training with counterfactual prompts restores language sensitivity; injecting the feature does not steer the policy. If correct, the result gives a lightweight, label-free runtime monitor for VLA deployments.","feed_headline":"Robot policies reveal task progress in a single linear readout","feed_subtitle":"A label-free probe on π0.5 activations flags stalled tasks and matches supervised OOD detectors.","key_machinery":"The load-bearing object is the linear probe $\\varphi(z_i)=w^\\top z_i + b$ applied to mean-pooled activations $z_i$ at a chosen transformer layer, trained under an L1 loss on labels $\\tau_t = 1 - t/T$. The paper pairs the probe with three graded definitions — weak decodability (in-distribution accuracy), strong decodability (accuracy under counterfactual input transformations), and steerability (whether injecting the probe direction changes the action distribution) — and with the residual score $V_\\tau(t)=\\tau_{\\text{pred}} - (1 - t/E[T|\\ell])$ that converts probe output into an OOD flag when it exceeds a threshold. For strong decodability, a within-pair hinge separates swapped-prompt activations from original ones by a time-dependent margin, anchored by a BCE term on absolute progress. The probe's direction $w$ is also the steering vector in the injection test, so the same learned feature supports reading, monitoring, and the controllability experiment.","core_discovery":"On the paper's own terms: task progress, defined as the normalized fraction of a trajectory remaining ($\\tau(x_t)=1-t/T$), is weakly and strongly decodable from π0.5's mean-pooled residual-stream activations but not steerable. The same linear probe, fit with an L1 loss on in-distribution demonstrations, generalizes across unseen tasks once enough tasks are included; a probe on raw observations fails out of distribution, showing the representation, not the raw inputs, carries the signal. The probe also functions as an out-of-distribution detector: comparing predicted progress against expected progress $1 - t/E[T|\\ell]$ flags stalled rollouts, matching or beating supervised baselines on unseen tasks and unseen perturbation types. Language counterfactuals reveal that a naively trained probe ignores the instruction, while a contrastive probe trained with swapped prompts labeled as zero progress separates the two prompts, at the cost of lower in-distribution $R^2$.","pith_inferences":["The same probing protocol could be applied to other VLA families to test whether linear progress decodability is a general property of imitation-trained policies or specific to the π0.x lineage.","Because Vτ is a scalar residual, it could be wrapped in conformal prediction to convert the OOD flag into a failure guarantee with statistical coverage; the paper does not do this.","A speed-varied or pause-injected rollout would separate the time-index reading from the semantic-progress reading; the paper's raw-observation control does not fully rule out a time feature inside the residual stream.","The contrastive labels (swapped prompt ⇒ τ=1.0) are proxies; generating true counterfactual rollouts for swapped instructions would let the probe be validated without that labeling assumption."],"forward_implications":["A VLA can be instrumented at runtime with a single linear probe: no OOD labels, no separate failure model, just a threshold on the residual between predicted and expected progress.","The same probe doubles as a language-grounding diagnostic: a probe that ignores prompt swaps reveals where fine-tuning has degraded instruction following.","Progress is observable but not linearly controllable; injecting the readout direction does not change action outputs, so monitoring and steering are different capabilities.","The signal's presence before robot fine-tuning and its sharpening by pre-training means progress readout transfers across the π0.x family without per-task probe retraining.","When aggregated per episode, the progress residual beats supervised baselines on unseen perturbation types, making it a candidate for deployment monitoring where OOD examples cannot be enumerated in advance."],"supporting_citations":[{"why":"Supplies the π0.5 model whose residual-stream activations are probed throughout.","marker":"[2]"},{"why":"Supplies the PaliGemma vision-language backbone, the substrate from which activations are extracted.","marker":"[11]"},{"why":"Provides the shuffled-label control-task methodology used to confirm that weak decodability is not a probe-capacity artifact.","marker":"[31]"},{"why":"Provides the VLABench dataset of ten manipulation tasks used to train probes and run OOD evaluations.","marker":"[32]"},{"why":"Supplies the supervised SAFE detection baselines that the progress-probe OOD detector is compared against.","marker":"[20]"},{"why":"Supplies the Mahalanobis-distance baseline for out-of-distribution detection.","marker":"[27]"},{"why":"Defines feature observability and controllability that the paper's weak decodability, strong decodability, and steerability refine.","marker":"[12]"}],"fun_headline_variants":["Linear probe decodes task progress from robot policy","Task progress readable in robot policy activations","VLA probe flags stalled tasks without labels","One linear readout monitors robot task progress","Probe finds task progress in robot policy internals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper labels task progress by normalized elapsed time in successful demonstrations, and the counterfactual training treats any swapped prompt as zero progress, so the claim that the decoded quantity is semantic progress rather than a time index rests on those labeling choices.","fun_headline_variants_meta":{"raw":{"variants":["Linear probe decodes task progress from robot policy","Task progress readable in robot policy activations","VLA probe flags stalled tasks without labels","One linear readout monitors robot task progress","Probe finds task progress in robot policy internals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000471,"raw_usage":{"total_tokens":2336,"prompt_tokens":930,"completion_tokens":1406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1337}},"tokens_in":546,"tokens_out":1406,"duration_ms":9572,"temperature":1.0,"reasoning_tokens":1337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:13:24.726947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect rollouts where the robot executes the same task at half speed, or pauses mid-trajectory while staying on track, and check whether the probe's predicted progress follows wall-clock time instead of completion fraction; if it follows time, the semantic-progress interpretation fails. Separately, execute a swapped instruction to actual completion and test whether a probe trained with the proxy label τ=1.0 still reads true progress.","supporting_citations":[{"cited_title":"Zhang, Z","cited_arxiv_id":null,"evidence_quote":"Provides the VLABench dataset of ten manipulation tasks used to train probes and run OOD evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Mahalanobis-distance baseline for out-of-distribution detection."}],"review_version":1}