Pith. sign in

REVIEW 4 major objections 6 minor 63 references

Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The Aftab architecture—a five-layer CNN with Hadamard multiplicative pooling and a distributional-ensemble-dueling head—lifts a replay-free, target-network-free PQN agent to an IQM Human-Normalized Score of 6.479 on Atari-57, a 0.86…

desk verdict Useful Atari-57 architecture study; the Procgen generalization claim is contradicted by the paper's own Table S5 and should be recalculated or dropped. read the letter →

arxiv 2608.07335 v1 pith:LDILW7NA submitted 2026-08-07 cs.LG cs.AI

Taha Shieenavaz , Shabnam Zareshahraki , Loris Nanni This is my paper · ORCID
classification cs.LGcs.AI
keywords DeepreinforcementlearningParallelizedQ-Network(PQN)Buffer-freeCNNencodersHadamaxrepresentationDistributionalDuelingnetworkarchitectureProcgengeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the encoder, not the optimizer, is the main performance lever in buffer-free, parallelized Q-learning. In the Parallelized Q-Network (PQN) framework, which removes replay buffers and target networks, the authors replace the standard three-layer CNN with deeper and multiplicative architectures and find that a five-layer CNN with Hadamard max-pooling, topped by a distributional-ensemble-dueling value head, reaches an IQM Human-Normalized Score of 6.479 on Atari-57. That is a 0.86 Probability of Improvement over the PQN baseline and holds with weight decay set to zero, so the stability comes from structure and normalization rather than regularization. On Procgen Hard the same model scores 0.418 IQM normalized versus 0.382 for PQN, which the paper reads as out-of-distribution resilience. A sympathetic reader would care because it suggests a cheap, memory-efficient path to strong Atari performance that does not rely on the standard large-buffer machinery.

What carries the argument

The load-bearing object is the Gamma-Hadamax-Valid encoder combined with the Aftab value head. Gamma-Hadamax-Valid is a five-block CNN in which each block computes two LayerNorm-normalized projections, combines them with a Hadamard (element-wise) product, applies GELU, and downsamples by explicit max-pooling instead of strided convolution; this decoupling expands the effective receptive field from 39×39 to 70×70. The Aftab head is a distributional ensemble dueling head: ten independently initialized dueling networks, each outputting a 51-atom categorical distribution over returns, trained with the Two-Hot Gaussian (HL-Gauss) cross-entropy loss, with per-episode head sampling approximating Thompson sampling and mean-aggregation at evaluation. This machinery is what keeps unregularized online TD updates stable and drives temporally extended exploration.

What would settle it

Recompute the Procgen comparison using the mean over all 16 environments, or normalize each game against fixed external references (e.g., a random policy and a standard baseline) instead of the two models in this suite; if Aftab's aggregate then falls to or below PQN's, the claimed out-of-distribution generalization advantage is refuted. As a second check, ablate the Hadamard product in Gamma-Hadamax-Valid: if IQM HNS on Atari-57 does not drop materially, the 70×70 receptive-field mechanism is not the driver.

Watch

Extended reading notes

Core claim

The central claim is that a specific composite architecture, called Aftab, establishes a new efficient reference point for model-free, buffer-free value-based reinforcement learning. The paper walks through three phases: among eight parameter-constrained CNN encoders, the five-layer Gamma topology (channels 32→48→64, 3×3 kernels, total 1.84M parameters) matches the best four-layer Alpha (3.481 vs 3.536 IQM HNS) at lower cost; integrating Gamma with Hadamax multiplicative pooling (Gamma-Hadamax-Valid) raises IQM HNS to 5.343 with the same 1.84M parameters; and replacing the scalar head with a distributional ensemble dueling head yields Aftab at 6.479 IQM HNS (95% CI [5.890, 7.210]) with a 0.86 Probability of Improvement over PQN. The same final model scores 0.418 IQM Procgen Normalized Score on Procgen Hard versus 0.382 for PQN. The paper presents these results as demonstrating that structural inductive bias—depth, multiplicative feature interaction, and multi-head value estimation—can substitute for replay buffers, target networks, and ℓ2 regularization in off-policy TD learning.

Load-bearing premise

The out-of-distribution generalization claim rests on a Procgen normalization defined only against the two models in this study, with an IQM that drops the lowest-quarter environments—including the five games where Aftab scores zero—so the 0.418-versus-0.382 gap is a relative, filtered ranking rather than an absolute measure.

Editorial extensions

If this is right

  • A replay-free agent can exceed professional-human-normalized performance on Atari-57 with a 1.84M-parameter encoder, so memory-constrained and high-throughput setups need not sacrifice score for buffering.
  • Encoder topology and value-head structure, not raw parameters, are the primary drivers of sample efficiency in this regime; the 23.8M-parameter shallow Eta variant scores below the 1.84M-parameter Gamma.
  • Hadamard-style multiplicative pooling with decoupled max-pooling is a transferable design pattern for value-based RL backbones, roughly doubling Gamma's IQM HNS at the same parameter budget.
  • Distributional, ensemble, and dueling heads—normally associated with replay-based Rainbow—can be combined stably in an online, buffer-free setting, which widens the design space for low-memory agents.
  • The measured 0.418 vs 0.382 gap on Procgen Hard is offered as evidence that the architecture generalizes to procedurally generated, non-stationary visual domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the receptive-field account predicts a falsifiable ablation—replacing the Hadamard product with a plain sum, or restoring strided convolutions to shrink the field from 70×70 to 39×39 while keeping parameters fixed, should substantially reduce IQM HNS if the mechanism is real.
  • Extension: the zero-weight-decay stability is observed, not proven; the paper gives no TD-Jacobian bound for Aftab, so a formal stability analysis of the unregularized updates would be needed before relying on this recipe beyond the tested seeds.
  • Extension: the Procgen Normalized Score of Eq. 3 is defined with min and max taken over the two models in the suite, so 0.418 vs 0.382 is a relative ranking rescaled to [0,1]; a fair out-of-distribution comparison would require fixed reference scores from a third baseline.
  • Extension: the same encoder redesign could be applied to other buffer-free algorithms, such as meta-learned rule discovery, to test whether the depth-plus-multiplicative-interaction benefit transfers beyond PQN-style TD learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Aftab, a buffer-free PQN-style agent whose contribution is an empirical study of CNN encoder topologies and value-head designs. Phase 1 compares eight parameter-constrained CNNs and selects Gamma as a balanced backbone; Phase 2 integrates Hadamax blocks; Phase 3 adds distributional, ensemble, and dueling heads, yielding a composite architecture with a reported Atari-57 IQM HNS of 6.479 and a 0.86 Probability of Improvement over PQN. The paper also reports a Procgen Hard evaluation and claims an IQM Procgen Normalized Score of 0.418 versus 0.382 for PQN, interpreted as evidence of out-of-distribution generalization.

Significance. If the Atari-57 results are taken at face value, the paper provides a useful and unusually well-controlled architectural benchmark: the parameter/FLOP accounting is explicit, the training budget is fixed at 200M frames, results are averaged over four seeds, the statistical tests are non-parametric with multiple-comparison correction, and the authors state that raw logs are open-sourced. The Atari protocol follows standard practice, and the Phase 1/2/3 ablation structure is a genuine contribution to the buffer-free RL literature. However, the Procgen generalization claim, which is one of the two headline results in the abstract, is not supported by the paper's own data, and the model-selection protocol raises a real optimism risk for the Atari numbers. The paper is worth repairing, but it needs substantive technical revision before the claims can be accepted.

major comments (4)
  1. [§5.2, Eq. (3), Table S5] The reported Procgen PNS values cannot be reconstructed from the paper's own supplementary table under any stated reading of Eq. (3). With only two agents, a per-environment min/max normalization collapses PNS to a binary win/loss indicator; Table S5 gives Aftab 7 wins and 9 losses, so the IQM over 16 games is 3/8 = 0.375 for Aftab and 5/8 = 0.625 for PQN, not the headline 0.418/0.382. If instead the min and max are taken globally across all 32 raw scores (minimum -0.415, maximum 31.491), Aftab's IQM is approximately 0.13 and PQN's is approximately 0.09, again not the reported values. The five zero-score Aftab environments (Bossfight, Caveflyer, Climber, Coinrun, Jumper) are mostly removed by the IQM's lower 25% trim, so the 'confirm out-of-distribution generalization' conclusion in the abstract and Section 8 is an artifact of the aggregation plus the relative normalization. The authors should either redefine PNS so that it is not a two-agent relative ranking, report the correct numbers, or remove the Procgen superiority claim.
  2. [§5.1, §6.1–§6.3] The final Atari-57 IQM is reported on the same benchmark used to select the Gamma encoder, the Gamma-Hadamax-Valid variant, and the final regression heads. The robustness check in §6.1 uses random validation subsets drawn from Atari-57, but it still selects on the target distribution, and §6.2 and §6.3 appear to select the Hadamax and head variants using full-Atari comparisons (e.g., the p<0.001 improvement of Gamma-Hadamax-Valid over Gamma, and the p=0.014 improvement of Aftab over the scalar baseline). This selection-on-test risk is load-bearing for the 6.479 IQM and 0.86 Probability of Improvement headline. The paper should provide a held-out game split, a pre-registered selection rule, or an explicit correction acknowledging that the reported numbers are selection-tuned rather than independent test-set scores.
  3. [Tables 9, 11, 12; Tables 8 and S3] The Phase 3 ablation tables are internally inconsistent. Table 9 reports IQM HNS values of 5.343, 5.812, 5.940, and 6.479 for Gamma-Hadamax-Valid, Distributional Dueling, Ensemble Dueling, and Aftab, respectively, while Table 11 reports an IQM of 6.031 for Distributional Dueling and Table 12 reports 5.592 for Ensemble Dueling. The Phase 2 tables also disagree: Table 8 gives Gamma-Hadamax-Valid IQM 5.383, while Table S3 gives 5.254, and per-game values differ (e.g., Alien 3.137 in Table 8 versus 3.143 in Table S3). In addition, the 'Gamma' column in Tables 11/12 reproduces the standard Gamma scores from Phase 1, not the Gamma-Hadamax-Valid baseline described in the Phase 3 text. These discrepancies must be reconciled because the Phase 3 head ablation is a central contribution and the reader cannot determine which set of numbers is authoritative.
  4. [§3.2, §4.5, §7.3] The stability discussion is appropriately cautious about removing ℓ2 regularization, and the paper explicitly disclaims a formal convergence guarantee. This is a strength, not a defect. However, the paper should state more clearly that the empirical stability claim is limited to the four evaluated seeds and the fixed hyperparameter setting, and that the 'deadly triad' discussion in §7.1 is observational rather than a proof. As written, the transition from empirical stability to the phrase 'circumvents these nonlinear temporal-difference optimization traps' in §7.1 is too strong relative to the evidence presented.
minor comments (6)
  1. [§3.5] The text says 'As illustrated in Figure X' but no such figure is included; please insert the figure or remove the reference.
  2. [§5.2 and §6.3] The Procgen raw-score table is referred to as 'Table S4' in §6.3 but is actually Table S5; the pointer in §5.2 to 'Tables 3.S4 and 4.S4' is also non-standard and should be corrected.
  3. [§7.1] The parenthetical 'Bigfish (7.233 vs. 31.491) and Dodgeball (0.192 vs. 12.596)' does not state which number belongs to Aftab and which to PQN; reading it against Table S5 is confusing and the sentence should be rewritten with explicit model names.
  4. [§4.4 and Table 4] The phrase '50 million environmental steps per parallel environment' plus 'equates to 200 million observed frames' is arithmetically ambiguous; with 128 parallel environments the total step count would be far larger than 200M. Please state the total training budget unambiguously.
  5. [Figure 4 caption] The caption mentions a 'frozen target network parameters θ−', but PQN removes target networks; the figure appears to describe a DQN-style procedure and should be reconciled with the buffer-free setting or relabeled as a generic illustration.
  6. [Tables 8 and S3] The naming of Hadamax variants is inconsistent across tables ('Gamma-Valid Hadamax', 'Gamma-Hadamax-Valid', 'Hadamax-Gamma-Valid'); please unify the terminology throughout the manuscript and supplementary material.

Circularity Check

1 steps flagged · score 5.0 of 10

The Procgen normalized-score claim is self-referential: Eq. (3) defines PNS using the two models being compared, and the reported IQM values are not reproducible from the paper's own Table S5.

  1. self definitional [Section 5.2, Eq. (3); Section 6.3, Table S5]
    "Because Procgen lacks official human baseline standards to compute a standard Human-Normalized Score, we define and calculate the Procgen Normalized Score (PNS). The PNS is computed via a Min-Max Normalization of raw agent scores against the minimum and maximum performance limits observed across our experimental suite: PNS=(Agent Score−Score_min)/(Score_max−Score_min) (3) ... the final Aftab configuration achieved an aggregate IQM Procgen Normalized Score (PNS) of 0.418 ... This marks an improvement over the baseline PQN model, which yielded an IQM PNS of 0.382 ..."

    On Procgen the paper ran exactly two models, PQN and Aftab, as shown by the two data columns of Table S5. Therefore Score_min and Score_max in Eq. (3), whether taken per game or globally across the suite, are just the raw scores of the two models being compared. Each PNS value is a relative rescaling of the pairwise comparison rather than an externally anchored generalization measure, so the claimed 0.418 versus 0.382 gap is a restatement of the two-agent raw-score ordering. Even that ordering is not supported by the paper's own table: Aftab scores exactly 0 in Bossfight, Caveflyer, Climber, Coinrun, and Jumper, while PQN is positive in all five, and Aftab wins only 7 of the 16 games. No reading of Eq.

full rationale

The Atari-57 portion of the paper is not circular in the load-bearing sense: the Human-Normalized Score of Eq. (1) is anchored to fixed human and random reference scores, the reported IQM and Probability of Improvement are computed from the paper's own per-game tables, and the architectural selections are justified by external baselines (PQN, Hadamax) rather than by self-citations. The Phase 2 receptive-field explanation is a deterministic consequence of the stated kernel and stride configurations, not an imported uniqueness theorem. The paper also honestly disclaims formal convergence guarantees in Section 7.3, which reduces concern about hidden derivations. However, the Procgen out-of-distribution claim is partially self-referential. Equation (3) defines the Procgen Normalized Score using Score_min and Score_max 'observed across our experimental suite'; with only two agents in the suite, this makes PNS a two-agent relative indicator. Moreover, Table S5 does not reproduce the reported 0.418 versus 0.382 IQM values under any straightforward application of the stated formula, and the five zero-score Aftab environments are exactly the ones trimmed by the IQM's lower-quartile removal. Because this Procgen result is one of the paper's two headline claims and is used to conclude out-of-distribution generalization, the circularity score is raised to 5 rather than left at 0; the Atari claims remain independently anchored and keep the paper from being entirely circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard RL assumptions, the stability of unregularized PQN, the validity of the chosen benchmarks, and the appropriateness of selecting architectures on the evaluation benchmark. No new physical or mathematical entities are introduced. The Procgen PNS normalization is the most fragile piece because it is self-referential to the two compared models.

assumptions (4)
  • ad hoc to paper Layer Normalization plus the Gamma topology maintains stable TD learning when L2 weight decay is set to zero.
    Section 4.5 explicitly removes weight decay and states that no formal convergence guarantee or TD Jacobian bound is provided; the paper assumes empirical stability across seeds.
  • domain assumption The Atari-57 and Procgen Hard protocols are valid external benchmarks for the central claims.
    Sections 5.1 and 5.2 adopt standard Atari normalization and define a custom Procgen normalization; the latter is not externally anchored.
  • ad hoc to paper Selecting Gamma and the final regression heads using validation splits on Atari-57 does not materially overfit the final reported Atari-57 scores.
    Section 6.1 describes a 10-split robustness check but still selects the encoder on the same benchmark family that produces the headline IQM HNS, so the final numbers are not fully held out.
  • ad hoc to paper The Procgen PNS normalization, using min and max of the two compared models, is a meaningful cross-agent generalization metric.
    Eq. (3) defines PNS using Score_min and Score_max observed across the experimental suite, which with only two models makes the score a relative ranking rather than an absolute generalization measure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks." pith.science (2026). https://pith.science/paper/LDILW7NA

@misc{pith2026260807335,
  author       = {Pith},
  title        = {Pith review of: Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDILW7NA}},
  note         = {Machine review of arXiv:2608.07335}
}
read the original abstract

Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm achieves stable off-policy learning without relying on computationally expensive replay buffers or target networks. However, the representational capacity and parameter efficiency of visual encoders operating in these buffer-free settings remain underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks for PQN. We design and rigorously evaluate eight distinct CNN topologies, optimizing for sample efficiency under strict parameter constraints. Furthermore, we study the impact of representation and value estimation enhancements by integrating the Hadamax encoding paradigm and advanced Q-learning extensions, including distributional, ensemble, and dueling heads. Extensive experiments on the Atari-57 benchmark demonstrate that our proposed composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.479, establishing a 0.86 Probability of Improvement over the standard PQN baseline. Additionally, structural resilience evaluations on the highly non-stationary Procgen Hard benchmark confirm out-of-distribution generalization, with Aftab yielding an IQM Procgen Normalized Score of 0.418 compared to the baseline's 0.382. Ultimately, this work establishes an efficient, probabilistically superior structural reference for model-free reinforcement learning, all while preserving the simplicity and memory efficiency of unbuffered, parallelized optimization. The complete Aftab framework, including all model definitions, training configurations, and raw experimental logs, is open-sourced and available on our GitHub repository: https://github.com/tahashieenavaz/aftab

Figures

Figures reproduced from arXiv: 2608.07335 by the authors.

Figure 1
Figure 1. (a) Nature DQN highlights the standard workflow: a single actor populates a Replay Buffer, and a frozen Target Network is required to calculate stable learning targets. (b) PQN demonstrates the streamlined approach: multiple Vectorized Environments provide synchronous batches directly to the Learner Q-Network, eliminating the Target Network and Replay Buffer. 3.3. Hadamax Encoder The Hadamax (Hadamard Max-Pooling) e… view at source ↗
Figure 2
Figure 2. Evolution of Convolutional Processing Blocks. (a) The standard unnormalized [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Dueling Network Architecture. The shared convolutional feature extractor bifurcates [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The "Regression as Classification" Optimization Pipeline. The diagram illustrates [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Topological Evolution of Evaluated Encoders. A structural comparison of the core [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Phase 1 Statistical Significance Matrices. Significance heatmaps generated via pairwise Wilcoxon Signed-Rank Tests across the Atari-57 suite for the baseline convo￾lutional architectures. To maintain statistical integrity across the 36 pairwise comparisons, we applied …
Figure 7
Figure 7. Figure 7: Probability of Improvement (Encoders). A pairwise probability matrix, [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: Phase 2 Statistical Significance Matrices. Significance heatmaps generated via pairwise Wilcoxon Signed-Rank Tests for the Hadamax integration. To maintain statistical validity, the Holm-Bonferroni correction was applied to control the family-wise error rate at α = 0.0…
Figure 9
Figure 9. Figure 9: Probability of Improvement (Hadamax Integration). Pairwise probability matrix [PITH_FULL_IMAGE:figures/full_fig_p040_9.png]
Figure 10
Figure 10. Figure 10: Probability of Improvement (Final Regression Architectures). Pairwise comparison [PITH_FULL_IMAGE:figures/full_fig_p050_10.png]
Figure 11
Figure 11. Figure 11: Global Performance (IQM HNS). The HNS serves as a standardized metric where 0.0 represents random play and 1.0 represents professional human performance. The 5- layer Alpha and Gamma variants outperform the original DQN encoder. Comparison between full training (left)…
Figure 12
Figure 12. Figure 12: Wilcoxon Signed Rank Test. A significance heatmap generated via pairwise Wilcoxon Signed-Rank Tests conducted across the Atari-57 suite for Phase 1. To maintain sta￾tistical integrity across 36 comparisons, we applied the Holm-Bonferroni correction to control the fami…
Figure 13
Figure 13. Figure 13: Phase 2 Global Performance (IQM HNS). The integration of Hadamard max-pooling alters representational capacity. The Gamma-Hadamax V1 and V2 variants diverge from the baseline Hadamax architecture, demonstrating that multiplicative feature interactions pair effectively…
Figure 14
Figure 14. Figure 14: Phase 3 Global Performance (IQM HNS). The ablation of advanced value￾estimation heads operating on the Gamma-Hadamax-Valid backbone. Aftab framework yields the highest performance among the evaluated models, reaching 6.479 IQM. The zoomed frame (right) illustrates the…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 27 canonical work pages

  1. [1]

    14th International Conference on Frontiers in Handwriting Recognition (ICFHR) , pages=

    Real-time segmentation of on-line handwritten arabic script , author=. 14th International Conference on Frontiers in Handwriting Recognition (ICFHR) , pages=. 2014 , organization=

    work page 2014

  2. [2]

    EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine

    Weng, Jiayi and Huang, Min Lin and Huang, Shengyi and Bo, Hao and Makoviichuk, Denys , title =. arXiv preprint arXiv:2206.10558 , year =

    work page Pith review arXiv

  3. [3]

    arXiv preprint arXiv:1801.00690 , year =

    Tassa, Yuval and Doron, Yotam and Muldal, Alistair and Erez, Tom and Li, Yazhe and Casas, Diego de Las and Budden, David and Abdolmaleki, Abbas and Merel, Josh and Lefrancq, Andrew and Lillicrap, Timothy and Riedmiller, Martin , title =. arXiv preprint arXiv:1801.00690 , year =

    arXiv

  4. [4]

    6th International Conference of Soft Computing and Pattern Recognition (SoCPaR) , pages=

    Fast classification of handwritten on-line Arabic characters , author=. 6th International Conference of Soft Computing and Pattern Recognition (SoCPaR) , pages=. 2014 , organization=

    work page 2014

  5. [5]

    arXiv preprint arXiv:1804.09028 , year=

    Estimate and Replace: A Novel Approach to Integrating Deep Neural Networks with Existing Applications , author=. arXiv preprint arXiv:1804.09028 , year=

    arXiv

  6. [6]

    Nature , volume =

    Human-level control through deep reinforcement learning , author =. Nature , volume =. 2015 , publisher =

    work page 2015

  7. [7]

    arXiv preprint arXiv:1312.5602 , year =

    Playing Atari with Deep Reinforcement Learning , author =. arXiv preprint arXiv:1312.5602 , year =

    arXiv

  8. [8]

    arXiv preprint arXiv:2407.04811 , year=

    Simplifying Deep Temporal Difference Learning , author=. arXiv preprint arXiv:2407.04811 , year=

    arXiv

Show all 63 references
  1. [9]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  2. [10]

    Espeholt, Lasse and Marinier, Raphael and Stanczyk, Piotr and Wang, Ke and Michalski, Marcin , booktitle=

  3. [11]

    Scaling Up Kernels in

    Ding, Xiaohan and Zhang, Xiangyu and Han, Jungong and Ding, Guiguang , booktitle=. Scaling Up Kernels in

  4. [12]

    arXiv preprint arXiv:1701.04128 , year=

    A Guide to Receptive Field Arithmetic for Convolutional Neural Networks , author=. arXiv preprint arXiv:1701.04128 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:1409.1556 , year=

    Very deep convolutional networks for large-scale image recognition , author=. arXiv preprint arXiv:1409.1556 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:1603.07285 , year=

    A guide to convolution arithmetic for deep learning , author=. arXiv preprint arXiv:1603.07285 , year=

    arXiv

  7. [15]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Rethinking the inception architecture for computer vision , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  8. [16]

    International Conference on Machine Learning (ICML) , pages=

    Dueling network architectures for deep reinforcement learning , author=. International Conference on Machine Learning (ICML) , pages=

  9. [17]

    International Conference on Learning Representations (ICLR) , year=

    On the Variance of the Adaptive Learning Rate and Beyond , author=. International Conference on Learning Representations (ICLR) , year=

  10. [18]

    Proceedings of the IEEE 12th International Conference on Computer Vision (ICCV) , year =

    Jarrett, Kevin and Kavukcuoglu, Koray and Ranzato, Marc'Aurelio and LeCun, Yann , title =. Proceedings of the IEEE 12th International Conference on Computer Vision (ICCV) , year =

  11. [19]

    , title =

    Nair, Vinod and Hinton, Geoffrey E. , title =. Proceedings of the 27th International Conference on Machine Learning (ICML) , year =

  12. [20]

    arXiv preprint arXiv:1607.06450 , year=

    Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

    arXiv

  13. [21]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  14. [22]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Volodymyr and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and Legg, Shane and Kavukcuoglu, Koray , title =. Proceedings of the 35th International Conference on Mac...

  15. [23]

    and Yang, Zhao and François-Lavet, Vincent , title =

    Kooi, Jacob E. and Yang, Zhao and François-Lavet, Vincent , title =. arXiv preprint arXiv:2505.15345 , year =. doi:10.48550/arXiv.2505.15345 , url =

    doi arXiv

  16. [24]

    arXiv preprint arXiv:2504.13112 , archivePrefix =

    Hadamard product in deep learning: Introduction, advances and challenges , author =. arXiv preprint arXiv:2504.13112 , archivePrefix =. 2025 , month =. 2504.13112 , primaryClass =

    2025 arXiv

  17. [25]

    arXiv preprint arXiv:1606.08415 , year =

    Gaussian Error Linear Units (GELUs) , author =. arXiv preprint arXiv:1606.08415 , year =

    arXiv

  18. [26]

    2018 , publisher =

    Reinforcement Learning: An Introduction , author =. 2018 , publisher =

    2018

  19. [27]

    Proceedings of the IEEE , volume =

    Gradient-based learning applied to document recognition , author =. Proceedings of the IEEE , volume =. 1998 , publisher =

    1998

  20. [28]

    Journal of Artificial Intelligence Research , volume =

    The Arcade Learning Environment: An Evaluation Platform for General Agents , author =. Journal of Artificial Intelligence Research , volume =. 2013 , doi =

    2013

  21. [29]

    Machine Learning , volume =

    Learning to predict by the methods of temporal differences , author =. Machine Learning , volume =. 1988 , publisher =. doi:10.1007/BF00115009 , url =

    1988 doi

  22. [30]

    International Conference on Machine Learning , pages =

    A Distributional Perspective on Reinforcement Learning , author =. International Conference on Machine Learning , pages =. 2017 , publisher =

    2017

  23. [31]

    Proceedings of the 40th International Conference on Machine Learning , series =

    Atari-5: Distilling the Arcade Learning Environment Down to Five Games , author =. Proceedings of the 40th International Conference on Machine Learning , series =. 2023 , publisher =

    2023

  24. [32]

    and Hessel, Matteo and Zintgraf, Luisa and Singh, Satinder and van Hasselt, Hado and Silver, David , title =

    Oh, Junhyuk and Farquhar, Greg and Kemaev, Iurii and Calian, Dan A. and Hessel, Matteo and Zintgraf, Luisa and Singh, Satinder and van Hasselt, Hado and Silver, David , title =. Nature , volume =. 2025 , doi =

    2025

  25. [33]

    Nature , volume =

    Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian and Chen, Yutian and Lillicrap, Timothy and Hui, Fan and Sifre, Laurent and van den Dries...

    2017 doi

  26. [34]

    arXiv preprint arXiv:1707.07012 , year =

    Learning Transferable Architectures for Scalable Image Recognition , author =. arXiv preprint arXiv:1707.07012 , year =

    arXiv

  27. [35]

    arXiv preprint arXiv:1509.02971 , year =

    Continuous control with deep reinforcement learning , author =. arXiv preprint arXiv:1509.02971 , year =

    arXiv

  28. [36]

    arXiv preprint arXiv:2410.09754 , year =

    Simba: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning , author =. arXiv preprint arXiv:2410.09754 , year =

    arXiv

  29. [37]

    arXiv preprint arXiv:1902.05605 , year =

    CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity , author =. arXiv preprint arXiv:1902.05605 , year =

    1902 arXiv

  30. [38]

    arXiv preprint arXiv:2101.05982 , year =

    Randomized Ensembled Double Q-learning: Learning Fast Without a Model , author =. arXiv preprint arXiv:2101.05982 , year =

    arXiv

  31. [39]

    arXiv preprint arXiv:2007.05929 , year =

    Data-Efficient Reinforcement Learning with Self-Predictive Representations , author =. arXiv preprint arXiv:2007.05929 , year =

    2007 arXiv

  32. [40]

    and Agarwal, Rishabh and Castro, Pablo Samuel , booktitle =

    Schwarzer, Max and Obando Ceron, Johan Samir and Courville, Aaron and Bellemare, Marc G. and Agarwal, Rishabh and Castro, Pablo Samuel , booktitle =. Bigger, Better, Faster: Human-level. 2023 , publisher =

    2023

  33. [41]

    Advances in Neural Information Processing Systems , year =

    Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control , author =. Advances in Neural Information Processing Systems , year =

  34. [42]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Deep Reinforcement Learning with Double Q-Learning , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2016 , url =

    2016

  35. [43]

    James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander

  36. [44]

    Advances in Neural Information Processing Systems 32 , editor =

    PyTorch: An Imperative Style, High-Performance Deep Learning Library , author =. Advances in Neural Information Processing Systems 32 , editor =. 2019 , publisher =

    2019

  37. [45]

    Journal of Machine Learning Research , volume=

    Statistical comparisons of classifiers over multiple data sets , author=. Journal of Machine Learning Research , volume=

  38. [46]

    Biometrics Bulletin , volume=

    Individual comparisons by ranking methods , author=. Biometrics Bulletin , volume=. 1945 , publisher=

    1945

  39. [47]

    Scandinavian Journal of Statistics , year =

    Holm, Sture , title =. Scandinavian Journal of Statistics , year =

  40. [48]

    and Gensler, H

    Aickin, M. and Gensler, H. , title =. American Journal of Public Health , year =

  41. [49]

    Journal of Thoracic Disease , year =

    Chen, Shi-Yi and Feng, Zhe and Yi, Xiaolian , title =. Journal of Thoracic Disease , year =

  42. [50]

    Proceedings of the Twelfth International Conference on Machine Learning (ICML) , year =

    Baird, Leemon , title =. Proceedings of the Twelfth International Conference on Machine Learning (ICML) , year =

  43. [51]

    Backprop-Free Reinforcement Learning with Linear Concept Bottleneck Models , booktitle =

    Fellows, Matthew and Lyle, Clare and Hua, Gerasimos and Luketina, Jelena and Gal, Yarin and Whiteson, Shimon and Rockt. Backprop-Free Reinforcement Learning with Linear Concept Bottleneck Models , booktitle =

  44. [52]

    The Annals of Mathematical Statistics , pages =

    Robbins, Herbert and Monro, Sutton , title =. The Annals of Mathematical Statistics , pages =

  45. [53]

    , title =

    Borkar, Vivek S. , title =

  46. [54]

    arXiv preprint arXiv:1602.04621 , year=

    Deep Exploration via Bootstrapped DQN , author=. arXiv preprint arXiv:1602.04621 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:1806.04613 , year=

    Improving Regression Performance with Distributional Losses , author=. arXiv preprint arXiv:1806.04613 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2403.03950 , year=

    Stop Regressing: Training Value Functions via Classification for Scalable Deep RL , author=. arXiv preprint arXiv:2403.03950 , year=

    arXiv

  49. [57]

    Advances in Neural Information Processing Systems , volume=

    Deep Reinforcement Learning at the Edge of the Statistical Precipice , author=. Advances in Neural Information Processing Systems , volume=

  50. [58]

    2021 , eprint=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. 2021 , eprint=

    2021

  51. [59]

    2022 , eprint=

    Generalized Data Distribution Iteration , author=. 2022 , eprint=

    2022

  52. [60]

    Mastering Atari, Go, chess and shogi by planning with a learned model , volume=

    Schrittwieser, Julian and Antonoglou, Ioannis and Hubert, Thomas and Simonyan, Karen and Sifre, Laurent and Schmitt, Simon and Guez, Arthur and Lockhart, Edward and Hassabis, Demis and Graepel, Thore and Lillicrap, Timothy and Silver, David , year=. Mastering Atari, Go, chess ...

    doi

  53. [61]

    arXiv preprint arXiv:1912.01588 , year=

    Leveraging Procedural Generation to Benchmark Reinforcement Learning , author=. arXiv preprint arXiv:1912.01588 , year=

    1912 arXiv

  54. [62]

    Distill , volume=

    Computing Receptive Fields of Convolutional Neural Networks , author=. Distill , volume=. 2019 , publisher=

    2019

  55. [63]

    2017 , eprint=

    Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , author=. 2017 , eprint=

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.