Computer Vision and Pattern Recognition · arXiv:2611.06918 v3 · submitted 4 Nov 2026
From Pixels to Actions–Towards Native Vision-Language-Action Policies
The paper says
“Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards.”
more of the abstract
Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 5 Nov 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuinely useful depth-readout study on an encoder-free VLA; the within-family results are solid, and the cross-family 'native beats encoder' claim is real but under-matched. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §5.1, Table 2; Abstract ([b6], [b32])
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.93):
“Although differences in pretraining and model capacity preclude attributing the gains solely to encoder removal, these results establish the native backbone as a strong and transferable foundation for action learning.”
Pith says: Objection 1 of 3 · major
The abstract's claims that native architectures 'outperform encoder-based counterparts under controlled comparisons' and that 'additional scale alone cannot recover it' are load-bearing but outrun the evidence. The only direct native-vs-encoder comparisons are cross-family: NEO-2B differs from Qwen3-VL-2B and Gemma-3-4B in pretraining corpus and objective, tokenization, positional embeddings, attention pattern, and capacity, and the authors themselves state at [b32] that 'differences in pretraining and model capacity preclude attributing the gains solely to encoder removal.'
more of objection 1
The scale claim rests entirely on Gemma-3-4B (4B, 84.5% frozen) trailing NEO-2B (2B, 90.9% frozen); that is exactly the comparison the acknowledgment says cannot bear causal weight, so 'additional scale alone cannot recover it' is not established as stated. Please either add a matched-interface control (an encoder-based variant of NEO trained with the identical three-stage recipe, or an encoder-free variant of an established VLM family) or rewrite [b6] to report the comparison-level fact with the confound stated in the same sentence.
The paper says, at §5.1, Table 3
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The encoder-access control changes two variables at once: the depth from which the expert reads and the alignment status of the features it reads. The Qwen vision-tower features come from a separately pretrained encoder without language or state conditioning, so the drop from 79.5% to 64.9% under a frozen backbone shows that unaligned intermediate features are not directly usable by …
more of objection 2
… the expert; it does not test whether early aligned features would help, and it cannot serve as an interface-level test of encoder-free versus encoder-based design. The Finding paragraph ('not simply in accessing earlier visual features, but in how encoder-free pretraining shapes them into control-ready representations') already interprets Table 3 correctly, but the surrounding text invites the reader to treat it as a readout-depth control. Please state explicitly that Table 3 is a feature-alignment control and that the readout-depth evidence comes from Table 4 and Figure 4 within the NEO family.
The paper says, at §5.2, §6 ([b9], [b36])
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The general principle in [b9]/[b36] — that effective control depends on which stages of visual computation remain accessible to action, and that the fixed encoder output is suboptimal — is supported only by ablations of a single backbone family (NEO-2B) with a single pretraining recipe.
more of objection 3
Table 4 and Figure 4 are internally clean, but a within-family depth result cannot by itself license a cross-architecture law about VLA interfaces. I recommend scoping the conclusion to the NEO family (for example, 'within the native NEO family, readout depth is a genuine policy variable') and presenting the cross-family results in Table 2 as evidence that the native backbone transfers well to action learning, rather than as confirmation of the depth mechanism.
5 minor objections
The paper says, at Tables 2 and 7
Pith says: minor objection 1 of 5
The same system is reported at 95.3% average LIBERO in Table 2 (full FT, step 30k, 500 episodes per suite) and 97.4% in Table 7 (step 27.5k, 2,000 episodes per suite); the discrepancy and the differing protocols are unexplained and should be reconciled in the text.
The paper says, at Table 11
Pith says: minor objection 2 of 5
The reported request-to-chunk latency of 261.9 ms is smaller than the compiled policy forward pass of 282.5 ms listed immediately above it, which is only possible if the two measurements make different assumptions; please reconcile or annotate the table.
show 3 more
The paper says, at §5.2, Figure 4
Pith says: minor objection 3 of 5
The intervention analysis is performed on 'diagnostic skip-28 models,' but the text does not state whether these are frozen or fine-tuned; since the readout-depth conclusion differs between regimes (Table 4), this information matters for interpretation.
The paper says, at Table 3 and §5.1
Pith says: minor objection 4 of 5
The choice of 13 expert layers for the encoder-access control, rather than the 28 layers used in the layer-map ablations, and the identity of the vision-tower layers that are read, are not specified in the main text; the same applies to the appendix-referenced pretraining configurations, which are not present in the submitted manuscript.
The paper says, at Introduction, contributions
Pith says: minor objection 5 of 5
The phrase 'early visual representations are pretty effective for control' in the second contribution bullet is informal for a journal article; suggest 'consistently effective' or similar.
What would settle it
Construct a matched pair from one VLM family: take Qwen3-VL-2B and make an encoder-free variant by replacing its vision tower with a NEO-style patch embedder initialized into the LLM embedding space, while keeping the encoder-based original as the control. Pretrain both on the same OXE corpus for the same number of steps with the same 28-layer action expert and the same skip-28 layer map, then evaluate frozen-backbone LIBERO success over multiple seeds. If the encoder-free variant does not exceed the encoder-based variant by a margin comparable to the 11.4-point gap in Table 2, the 'encoder-free interface' effect asserted by [b6] and [b9] is not established. The reverse check, inserting a frozen SigLIP encoder in front of NEO and rerunning the comparison, would serve the same purpose.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 5 Nov 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things. First, the paper's real contribution is the systematic depth-readout evidence, not the encoder-free framework itself: layer-map ablations with matched expert capacity (Table 4), an encoder-access control on Qwen (Table 3), and KV-swap/masking interventions with a matched-noise control (Figure 4) converge on the claim that early visual layers carry action-relevant signal. Second, the headline claim that native architectures outperform encoder-based ones is weaker than the abstract suggests, and the authors mostly know it.
The within-family depth result holds up. Skip-28 beating end-28 on a frozen backbone (90.9 vs 86.7) is a clean comparison: same backbone, same 28-layer expert, same corpus, only the layer map changes. The first-14/last-14 comparison is muddier because it also shortens the backbone, and the paper says so. The intervention plots are a nice addition; the matched-noise control addresses the obvious confound that swaps are just bigger perturbations than nothing.
Where the paper strains is the native-versus-encoder conclusion. Table 2 compares NEO-2B against Qwen3-VL-2B and Gemma-3-4B across pretraining corpus, objective, tokenization, and capacity, so the gap cannot be attributed to encoder removal alone. The authors concede this at [b32], which is honest, but the abstract's 'additional scale alone cannot recover it' [b6] is still too strong for a single cross-family comparison. Table 3 narrows the gap by giving Qwen direct access to intermediate vision-tower features and showing it does not help, but those features come from an unaligned, separately pretrained tower, so it is not a matched test of the interface. The stress-test note has this right: encoder removal is never manipulated while holding everything else fixed. The depth principle is supported within one family; the general law about fixed encoder endpoints is not.
Smaller issues: no error bars or significance tests on any headline number, the stage-3 'in-house action data' is undisclosed so the final weights cannot be reproduced, and the action-only baseline in Table 6 anchors a co-training recommendation but has no training recipe. None of these are load-bearing flaws, but a referee should ask for them.
This paper deserves a serious referee. The depth-readout study is carefully done on external benchmarks, the limitations are unusually explicit, and the robustness results on LIBERO-plus and RoboTwin 2.0 are worth taking seriously. I would send it out, and I would push either for a matched-scale native/encoder pair or for language that stops short of claiming scale alone cannot close the gap.
The strongest objection
For [b9] and the abstract's claim that 'additional scale alone cannot recover it' [b6] to hold, the visual interface must be an independent causal variable: an encoder-free policy should beat an encoder-based policy when matched in everything except the interface. The paper's direct evidence never realizes this condition. In Table 2, NEO-2B differs from Qwen3-VL-2B and Gemma-3-4B in pretraining corpus and objective, tokenization, positional embedding, attention pattern, and capacity, not only in the absence of an encoder; the authors concede at [b32] that gains cannot be attributed solely to encoder removal. The Qwen encoder-access control in Table 3 routes expert layers to intermediate vision-tower features, but those features come from a separately pretrained tower without language or state conditioning, so the control narrows but does not isolate the claimed variable. The remaining controlled evidence, Table 4 layer maps and Figure 4 interventions, is strong for the claim that early readout helps within the authors' own native NEO-2B family, but a single family with one pretraining recipe cannot support a general 'fixed encoder output is suboptimal' principle across VLA designs. The central assertion is not internally contradicted; it is under-supported exactly where the paper generalizes from within-family ablations to a cross-family law.
In good faith, the paper is unusually transparent: it reports the layer-map ablations, the matched-noise control in Figure 4A, and explicitly acknowledges at [b32] that pretraining and capacity differences preclude attributing gains solely to encoder removal. Those internal controls credibly establish that readout depth matters within the authors' NEO-2B architecture. The problem is that the central claim [b9] and the abstract's scale claim [b6] are stated as general facts about encoder-free versus encoder-based VLA design, while the only head-to-head evidence is cross-family. The reader's weakest_assumption already identifies this confound; this stress pass agrees and sharpens it: Table 3's control does not create a matched interface contrast, because Qwen's vision-tower intermediate features are not language-conditioned and cannot reproduce what NEO early shared layers provide. A matched encoder-free versus encoder-based pair within one VLM family would settle whether the interface itself is responsible. Such a test is expensive but feasible, and without it the appropriate verdict remains CONDITIONAL: the depth-readout effect within NEO is supported, but its generalization to all VLAs is not yet established. No adjustment to the reader's verdict is needed.
The referee's report
If the depth-readout thesis holds, the paper makes a genuinely useful contribution. It identifies visual readout depth as an explicit, tunable policy variable, provides an open native-VLA system with strong open-model results on LIBERO-plus (89.7) and RoboTwin 2.0 Clean2Rand (69.4), and validates on a real robot. The within-family evidence is methodologically careful: the key layer-map contrast (Table 4, skip-28 vs end-28 frozen, 90.9% vs 86.7%) matches expert capacity and backbone depth; the intervention study (Figure 4A) includes a matched-noise control; and Table 3 is an honest negative result showing that unaligned intermediate encoder features do not transfer. The authors disclose the main confound ([b32]) and explicitly label cross-paper LIBERO comparisons as benchmark context rather than controlled comparisons ([b201]-[b202]).
Pith's reading of the paper (a machine's summary, not the authors')
The paper sets out to establish that a robot manipulation policy should not be limited to the single visual feature vector produced by a pretrained vision encoder. It introduces NEO-VLA, a structurally encoder-free architecture in which raw pixels are patchified by a lightweight convolutional embedder and processed jointly with language throughout the backbone, with a flow-matching action expert that predicts continuous trajectories while reading cached attention keys and values from selected backbone layers. With the backbone frozen and only the expert trained, the native design reaches 90.9% average LIBERO success, versus 79.5% for Qwen3-VL-2B and 84.5% for Gemma-3-4B under the same expert and robot-training corpus; full fine-tuning narrows but does not erase the gap. Layer-map ablations and key–value swap interventions show action-critical visual information emerges early, often peaking before the final representation. If this is right, the depth at which a policy reads visual computation is a genuine design variable, and the default fixed-encoder interface leaves control-relevant signal unused.
What carries the argument. The load-bearing mechanism is the layer map $M=(m_1,\dots,m_d)$, which assigns each layer of the action expert to the cached attention keys and values of a particular backbone layer; comparing first-28, skip-28, end-28, and full-40 maps isolates readout depth under matched expert capacity. The native visual interface is a two-stage convolutional patch embedder with overall stride 32 that turns pixels directly into backbone tokens, so image patches and instruction text co-evolve through a shared 40-layer transformer with no separately pretrained vision encoder, image codebook, or autoencoder. The action expert is a narrow flow-matching network that predicts a continuous velocity field over 30-step action chunks, attending jointly to its own action tokens and to the cached keys and values from its mapped backbone layers. The same attachment points support the interventions, in which one layer's cached keys and values are swapped with another observation or one input span is masked, revealing distinct depth profiles for appearance, language, and geometry signals.
Core claim. The central claim is that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output. Three lines of evidence carry the argument: with the backbone frozen, the encoder-free NEO-VLA reaches 90.9% average LIBERO success against 79.5% for Qwen3-VL-2B and 84.5% for the larger Gemma-3-4B under the same action expert and robot corpus; a layer map that keeps early layers accessible (skip-28) beats a late-only map (end-28) by 90.9% versus 86.7% when frozen and 95.3% versus 92.5% after full fine-tuning; and swapping early-layer cached keys and values between observations perturbs predicted translation more than late-layer swaps, with a matched-noise control indicating the effect depends on content, not just disturbance size. The paper also claims that directly exposing an encoder-based model to intermediate vision-tower features does not recover the advantage, lowering frozen performance from 79.5% to 64.9%, so how the visual representations are formed and aligned matters as much as early access. The authors state that pretraining and capacity differences across backbones preclude attributing the gains solely to encoder removal.
The premise it rests on. The argument depends on the compared models being similar enough that the frozen-backbone gap comes from removing the encoder rather than from their different pretraining, objectives, or sizes; the authors themselves say those differences prevent attributing the gains solely to encoder removal, and the depth result is inferred from ablations on one backbone family.
If the paper is right (Pith's inferences, not the authors')
Readout depth becomes a tunable architectural choice: action experts can be attached to early, late, or distributed backbone layers, and frozen-backbone deployment is a realistic regime because the native representations already support control.
Policies that keep early visual layers accessible should be especially stable under camera and appearance changes, while spatial-layout and task-goal changes remain the harder failure mode, matching the paper's LIBERO-plus and LIBERO-Pro profile.
Preserving multimodal understanding during action post-training is part of policy robustness: in the paper, action-only fine-tuning drops RoboTwin clean-to-randomized success from 69.4% to 10.7% and destroys the model's ability to produce text.
Exposing an encoder-based model to intermediate vision-tower features is not a substitute for native representation formation, so future comparisons should control how visual features are produced, not only which layers the policy reads.
Aggregate LIBERO results are saturating, so benchmarking should move to perturbation-specific suites such as LIBERO-plus and RoboTwin 2.0 to reveal differences in robustness.
A cleaner isolating experiment than any in the paper would be a pretraining-matched native-versus-encoder pair, holding corpus, objective, and scale fixed; that pair alone would decide how much of the frozen gap is an interface effect.
Since the layer map is hand-set, a natural extension is to learn or search the readout depths per task or embodiment; the intervention profiles suggest optimal depth is task-dependent.
The early-layer swap result predicts that real-robot policies with early readout should stay accurate under camera-intrinsic and viewpoint changes, because their control signal is tied to raw geometry rather than distilled semantics.
The co-training result implies a reporting convention worth adopting: manipulation benchmarks should be paired with a language-understanding score, since action-only post-training can look fine on task success while silently destroying generalization.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-05
The authors' main claim, as the classifier picked it out 1.00
“Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.”
How the main claim is stated: hedged possibility 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.28 · human participants 0.04 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.05 · compares with prior results 0.94 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. | background 1.00 | 0.05 | 0.02 | asserted 0.99 |
| Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. | background 0.69 | 0.07 | 0.03 | asserted 0.86 |
| This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? | gap 0.83 | 0.05 | 0.04 | unclear 0.62 |
| We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. | contribution 0.96 | 0.86 | 0.05 | asserted 0.99 |
| Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. | method 0.98 | 0.82 | 0.07 | asserted 1.00 |
| This simple change reveals a consistent pattern. | result 0.74 | 0.53 | 0.03 | asserted 0.91 |
| Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. | result 1.00 | 0.93 | 0.23 | asserted 1.00 |
| More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. | result 1.00 | 0.94 | 0.12 | asserted 0.99 |
| Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. | result 1.00 | 0.92 | 0.17 | asserted 1.00 |
| Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output. | interpretation 1.00 | 0.67 | 0.92 | hedged possibility 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-05: 1 request, 7,431 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment