{"id":"3000b20d-c907-446b-bc31-04ff97614a73","arxiv_id":"2608.07870","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"V-Simba, a visual RL architecture combining layer normalization, weight decay, and a distributional critic, matches or outperforms complex baselines on 29 continuous control tasks while using less compute.","lead":"This paper introduces V-Simba, a neural network architecture for reinforcement learning agents that learn from camera images, and shows it matches or beats more complex algorithms on 29 robotic control tasks. It matters because it suggests that simple architectural changes, rather than new learning algorithms, can make image-based robot learning faster and cheaper.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark comparisons change the RL algorithm (SAC plus distributional critic and reward normalization) at the same time as the architecture, so the 'architecture alone' claim is not cleanly isolated in the headline results.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the main comparisons change the base RL algorithm at the same time as the architecture. This is the right concern because the paper's central contribution is that 'robust architectural design alone' produces state-of-the-art visual continuous control. That claim requires holding the learning algorithm fixed across the comparison, but the benchmark plots in Section 5.2 compare DrQ-v2 (DDPG, MSE critic) against V-Simba (SAC, distributional critic, reward normalization), so the observed differences cannot be unambiguously attributed to the architecture. The only fixed-algorithm evidence is the single-environment Dog Stand analysis in Figure 2 and Appendix Figures 9-10, which is suggestive but far too narrow to support the 29-task claim. This is not a disagreement with the field's consensus or an internal inconsistency; it is an attribution problem: the experiments as presented do not isolate the quantity the headline claims. The ablation table varies components within V-Simba's own SAC base and therefore does not resolve the conflation with DrQ-v2's algorithm. The concern is addressable with controlled benchmark experiments, so a conditional verdict is appropriate rather than rejection. If the fixed-algorithm comparison reproduces the gains, the architecture claim would be substantially strengthened; if not, the paper's main attribution should be revised toward a combined architectural-plus-algorithmic contribution.","tokens_in":20959,"tokens_out":5153,"duration_ms":59813,"concrete_test":"Extend the Appendix 7.1/7.3 architecture comparison from Dog Stand to all 18 DMC Medium/Hard tasks, plus representative Adroit and Meta-World tasks. For each task, train both DrQ-v2's architecture and V-Simba under two fixed base algorithms: (A) DDPG with augmentation and DrQ-v2-style MSE critic, and (B) SAC with V-Simba's distributional critic and reward normalization, using identical hyperparameters and 5 seeds. Report per-task deltas and bootstrap intervals. If V-Simba's architecture-only margin is near zero or within noise where the headline plots show large gains, the benchmark results are driven by the algorithm change rather than by the architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that V-Simba is built on SAC with data augmentation and adds LayerNorm, L2 weight regularization, a distributional critic, and reward normalization. Yet Section 5.2's DMC Medium paragraph says 'our base algorithm, DrQ-v2, falls behind... simply replacing DrQ-v2's neural network with our proposed architecture,' which is misleading: DrQ-v2 is DDPG with an MSE critic, while V-Simba uses SAC, a categorical critic, and reward scaling. The benchmark curves in Figures 5-8 therefore vary the learning algorithm and the architecture simultaneously. The controlled comparisons in Figure 2 and Appendix Figures 9-10 do hold the algorithm fixed, but only on the single Dog Stand environment, so they cannot support the 29-task 'architectural design alone' claim in the abstract and contribution list. The central attribution of the headline gains to architecture is thus unsupported without a fixed-algorithm benchmark comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces V-Simba, an architecture for visual continuous control built on top of SAC with data augmentation. The architecture adds layer normalization, residual connections, pointwise convolutions, early downsampling, weight decay, a distributional (categorical) critic, and reward normalization. The authors claim that V-Simba matches or outperforms state-of-the-art methods across DMC, Adroit, and Meta-World (29 tasks) with a single hyperparameter set, while being more computationally efficient than DrQ-v2, and they provide diagnostic metrics (loss sharpness, dormant ratio, feature diversity, norm growth) and an ablation study to support the design. The paper includes controlled architecture comparisons on Dog Stand under both DDPG and SAC.","tokens_in":21199,"tokens_out":3446,"duration_ms":38314,"significance":"If the results hold, the paper would demonstrate that a carefully designed convolutional architecture can yield large sample-efficiency gains in visual continuous control without complex algorithmic add-ons, offering a simple, reusable component for the community. The paper ships public code and provides an unusually detailed stability analysis of common visual RL encoders, which is a useful contribution regardless of the headline claim. The ablation study gives explicit evidence that each component contributes to the final performance. However, the central claim that 'architectural design alone' drives the benchmark gains is not cleanly supported by the main experiments because the algorithm changes simultaneously with the architecture.","major_comments":[{"comment":"The headline benchmark comparisons vary the RL algorithm and the architecture at the same time. V-Simba is built on SAC (Section 4), while DrQ-v2 uses DDPG, and V-Simba further adds a categorical distributional critic and reward normalization (Section 4.1). The sentence 'simply replacing DrQ-v2’s neural network with our proposed architecture' is therefore misleading, because the learning algorithm also changes. The controlled algorithm-fixed comparisons in Figure 2 and Appendix 7.3 provide supporting evidence on Dog Stand, but only on a single environment. As a result, the abstract and Contribution 3 claim that 'architectural design alone' achieves superior performance across 29 tasks is not established by the main benchmark results; the claim needs either a fixed-algorithm comparison across a wider task set or a more careful wording that attributes the gains to the combined architectural and algorithmic package.","section":"Section 5.2 (DMC Medium paragraph) and Figures 5–8"},{"comment":"The paper describes the distributional critic and reward normalization as part of the V-Simba architecture, but these are algorithmic components rather than architectural changes in the usual sense. Contribution 3 states that 'robust architectural design alone can achieve superior sample and compute efficiency without introducing complex algorithmic add-ons,' which overstates the purity of the architectural contribution. The categorical critic and reward scaling are value-learning modifications, not neural-network layout choices, and the abstract’s emphasis on architecture should be reconciled with this.","section":"Contribution 2/3 and Section 4.1"},{"comment":"The abstract claims V-Simba 'matches or outperforms' state-of-the-art methods, but the DMC Hard paragraph states only that V-Simba 'performs competitively with MR.Q,' which is a weaker claim. The paper should either present quantitative aggregates showing that V-Simba matches or exceeds all baselines across the full benchmark, or soften the abstract to 'is competitive with' where appropriate.","section":"Abstract and Section 5.2 (DMC Hard)"}],"minor_comments":[{"comment":"The phrase 'our base algorithm, DrQ-v2' is incorrect: the base algorithm is SAC, while DrQ-v2 is a DDPG-based method. Please rephrase to avoid confusion.","section":"Section 5.2"},{"comment":"'We first opt LayerNorm' should read 'We first adopt LayerNorm'.","section":"Section 4.1"},{"comment":"'highlighted base on their relative percentile difference' contains a typo: 'base' should be 'based'.","section":"Table 1 caption"},{"comment":"The Burda et al. exploration paper appears twice (2018a and 2018b) with identical content; please consolidate.","section":"References"},{"comment":"The definition of the dormant ratio is typeset in a garbled way; please reformat the equation for readability.","section":"Appendix 7.2"},{"comment":"The caption says both architectures are evaluated on SAC, but the surrounding text also discusses DrQ-v2's default DDPG. Please clarify in the caption that this is a controlled comparison holding the algorithm fixed.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' prior Simba and SimbaV2 papers for the core design principles, and the self-citation pattern is noticeable. That alone is not disqualifying, since the paper tests against external baselines. The main concern is the mismatch between the 'architecture alone' claim and the actual experimental design. I would ask the authors to either provide fixed-algorithm comparisons across a broader set of tasks or revise the claim to describe the full V-Simba method (architecture plus value-learning modifications) rather than pure architecture. If the authors can supply such evidence, the paper would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about visual RL architectures. The practical claim — a cheap network swap that stabilizes training and beats DrQ-v2 at lower compute — is probably real. The recipe (LayerNorm, weight decay, categorical critic, pointwise convolutions, early downsampling) is assembled in a new configuration for pixels, and the diagnostic analysis of sharpness, dormant ratio, and feature diversity is a nice addition. The 29-task evaluation across DMC, Adroit, and Meta-World shows strong curves, and the compute savings over DrQ-v2 are real. The code is promised, which is good.\n\nThe soft spot is exactly what the stress-test note flags. In the main benchmark comparisons, the architecture and the learning algorithm change simultaneously: DrQ-v2 is DDPG with an MSE critic, while V-Simba uses SAC, a categorical critic, and reward normalization. The Section 5.2 sentence claiming you can get these gains by “simply replacing DrQ-v2’s neural network” is misleading — more than the network changes. The controlled architecture-only comparisons in Figure 2 and the appendix run only on Dog Stand, so they cannot support the 29-task “architectural design alone” claim in the abstract. This is addressable: run the fixed-algorithm comparison across the full suite, or soften the claim.\n\nThe ablation table is weaker than it looks. Most component removals drop the aggregate score by 0.02–0.06 with three seeds and overlapping error bars; calling these “visible drops” oversells them. The direction is consistent, so it is a minor issue. Separately, the abstract says “matches or outperforms,” but the DMC Hard text says “competitive with MR.Q” — the headline slightly outruns the data.\n\nOverall, the paper is honest in its limitations section and the core idea is valuable. I would send it to review, with the fixed-algorithm benchmark comparison as a condition for acceptance. A reader working in visual RL should cite it, but with a caveat about the isolation issue.","headline":"Useful architecture paper, but the 'architecture alone' claim is not cleanly isolated across the main benchmarks because the algorithm changes (DDPG to SAC, distributional critic) at the same time.","tokens_in":649,"tokens_out":839,"would_cite":true,"duration_ms":39598,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a norm-stabilized network architecture (V-Simba) alone can match or beat leading visual continuous-control methods on 29 tasks, while training faster than DrQ-v2.","keywords":["deep reinforcement learning","visual continuous control","neural architecture design","layer normalization","distributional critic","sample efficiency","weight regularization","plasticity loss"],"falsifier":"Run a factorial comparison on a subset of DMC tasks: V-Simba's architecture with DrQ-v2's algorithm (DDPG, MSE critic) versus DrQ-v2's architecture with V-Simba's algorithm (SAC, categorical critic, reward normalization). If the architecture-only swap reproduces most of the performance gap at 1 million steps, the central claim holds; if the algorithm-only swap does, it fails. A second check: for every task where V-Simba wins, compute the top Hessian eigenvalue of both trained critics; the proposed mechanism predicts V-Simba's landscape is consistently flatter.","tokens_in":20793,"feed_emoji":"🧠","tokens_out":9604,"duration_ms":90094,"temperature":0.7,"pith_summary":"This paper asks whether the architectural design principles that recently improved state-based RL can be transferred to pixel-based continuous control, where progress has come mostly from algorithms rather than network design. It claims the answer is yes: a redesigned convolutional network called V-Simba, built on SAC with data augmentation, matches or outperforms leading visual RL methods on 29 tasks across DMC, Adroit, and Meta-World using a single hyperparameter set, while training in less wall-clock time than DrQ-v2 and with fewer parameters. The design idea is that visual RL's standard shallow encoder becomes unstable — sharp loss landscapes, dormant units, feature collapse, and growing feature, parameter, and gradient norms — and that constraining those norms with layer normalization, $\\ell^2$ weight decay, and a distributional critic restores stable, efficient learning. If true, the result matters because it offers a cheap, algorithm-agnostic performance boost: swap the network, keep the rest.","feed_headline":"Architecture alone matches top visual-RL methods","feed_subtitle":"V-Simba adds normalization and cheap convolutions to a simple SAC agent, beating complex baselines at 1M steps.","key_machinery":"The load-bearing machinery is a set of components that jointly constrain feature, parameter, and gradient norms. Layer normalization is placed at the input and after each stage, exploiting the scale-invariance property $\\mathrm{Norm}(cWx) = \\mathrm{Norm}(Wx)$, which makes gradients scale as $1/c$ with weight magnitude and therefore demands explicit control of weight growth via $\\ell^2$ decay. A categorical distributional critic with KL divergence and reward normalization keeps Q-value targets bounded and the loss landscape smooth. Wrapped around these are compute-saving choices — an early stride-2 convolution, 2×2 max-pooling, and pointwise 1×1 convolutions in inverted-bottleneck residual blocks — that keep the normalized network fast enough for 84×84 visual observations. These components work together to hold effective norms within a narrow band, which the paper links to smoother loss landscapes and higher final returns.","core_discovery":"The paper identifies that the standard visual-RL architecture popularized by DrQ-v2 — a shallow convolutional encoder with a single normalization layer — develops severe training instabilities: the loss landscape becomes sharp, a growing fraction of neurons go dormant, features collapse toward low diversity, and feature, parameter, and gradient norms grow without bound. It proposes V-Simba, which counters these instabilities by placing layer normalization at the input and after every block, adding $\\ell^2$ weight regularization, and replacing the mean-squared-error critic with a categorical distributional critic trained by KL divergence under reward normalization. To make this extra machinery affordable on high-resolution pixel inputs, V-Simba applies an early stride-2 convolution, uses 2×2 max-pooling for downsampling, and builds its convolutional blocks from lightweight pointwise (1×1) convolutions in an inverted-bottleneck residual layout. The paper reports that this architecture, without any new exploration strategy, world model, or auxiliary task, matches or outperforms DrQ-v2, DrM, TACO, TD-MPC2, MR.Q, and A-LIX across 18 DMC tasks, 3 sparse Adroit tasks, and 8 Meta-World tasks at 1 million environment steps, and that it reduces training time from 5.4 to 4.8 hours per 1M DMC steps.","pith_inferences":["The headline comparisons mix an architecture change with a base-algorithm change (SAC + distributional critic + reward normalization versus DDPG + MSE critic), so the cleanest reading is that architecture drives a large share of the gain, while the full margin may also depend on the algorithmic switches.","The same norm-control recipe could be tested on other high-dimensional observation regimes, such as multi-camera or partially observable robotic inputs, where the standard CNN encoder likely exhibits similar instability.","Since the paper notes sharpness aligns most consistently with performance ordering, a direct testable extension is whether a sharpness-reduction method on the standard DrQ-v2 architecture would close a substantial part of the gap without architectural change.","A natural stress test for the generality claim is to attach V-Simba's encoder to model-based methods such as TD-MPC2 or to offline datasets, where representation stability matters differently."],"forward_implications":["Adopting V-Simba as a drop-in replacement for DrQ-v2's encoder and heads should improve sample efficiency for any visual continuous-control algorithm that currently uses the standard shallow CNN.","Because V-Simba uses a single hyperparameter set across 29 tasks, its design reduces the per-task tuning burden that many algorithmic baselines require.","The diagnostic metrics — sharpness, dormant ratio, feature diversity, and norm growth — offer a concrete checklist for evaluating future visual RL architectures, even though no single metric is predictive alone.","V-Simba's lower parameter count and training time mean that the same norm-control approach can be scaled to wider or deeper networks without incurring the computational cost that previously made such scaling prohibitive for pixel inputs.","The success of transferring Simba-style principles from state-based to visual RL suggests that further state-based architectural advances may also carry over to pixel domains."],"supporting_citations":[{"why":"Defines DrQ-v2, the standard visual-RL architecture and algorithm that V-Simba replaces and the main baseline for comparisons.","marker":"Yarats et al. (2021a)"},{"why":"Supplies the Simba design guidelines from state-based RL, which V-Simba adapts to pixel inputs.","marker":"Lee et al. (2024b)"},{"why":"Provides the SAC algorithm that V-Simba is built on.","marker":"Haarnoja et al. (2018)"},{"why":"Introduces the categorical distributional critic whose KL loss stabilizes V-Simba's value learning.","marker":"Bellemare et al. (2017)"},{"why":"Provides the reward normalization and shift-and-norm tricks used to keep gradient scales stable.","marker":"Lee et al. (2025)"},{"why":"Grounds the scale-invariance argument linking weight growth to shrinking gradients under LayerNorm.","marker":"Lyle et al. (2024a)"},{"why":"DrM, a strong baseline; also the source of dormant-ratio analysis and Meta-World results.","marker":"Xu et al. (2023)"},{"why":"MR.Q, a strong baseline with its own normalized architecture, used for benchmark and stability comparisons.","marker":"Fujimoto et al. (2025)"}],"fun_headline_variants":["Simple architecture matches top visual-RL methods","V-Simba: Normalization and cheap convs rival SOTA","Architecture alone rivals complex visual-RL baselines","Cheap architectural changes match visual-RL SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance gains are attributed to the architecture itself, but the headline benchmark comparisons change the base algorithm together with the network, so the claim that architecture alone is responsible rests on the assumption that the concurrent switch to SAC with a distributional critic and reward normalization is not driving most of the improvement.","fun_headline_variants_meta":{"raw":{"variants":["Simple architecture matches top visual-RL methods","V-Simba: Normalization and cheap convs rival SOTA","Architecture alone rivals complex visual-RL baselines","Cheap architectural changes match visual-RL SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3074,"prompt_tokens":1032,"completion_tokens":2042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1977}},"tokens_in":648,"tokens_out":2042,"duration_ms":18044,"temperature":1.0,"reasoning_tokens":1977,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:44:02.388730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a factorial comparison on a subset of DMC tasks: V-Simba's architecture with DrQ-v2's algorithm (DDPG, MSE critic) versus DrQ-v2's architecture with V-Simba's algorithm (SAC, categorical critic, reward normalization). If the architecture-only swap reproduces most of the performance gap at 1 million steps, the central claim holds; if the algorithm-only swap does, it fails. A second check: for every task where V-Simba wins, compute the top Hessian eigenvalue of both trained critics; the proposed mechanism predicts V-Simba's landscape is consistently flatter.","supporting_citations":[],"review_version":1}