Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Improving Deep Reinforcement Learning in Minecraft with Action Advice

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that persistent action advice, delivered as a two-timestep repetition of each suggested action, lets a deep reinforcement learning agent move decisively through perceptually aliased states in Minecraft, converging faster…

desk verdict Useful extension of Newtonian Action Advice to DQN in Minecraft, but the NAA-over-FA result is confounded because the two agents get different advice formats (cardinal directions vs atomic actions), so the persistence claim is not yet established. read the letter →

arxiv 1908.01007 v1 pith:XPJTTPGP submitted 2019-08-02 cs.LG cs.AIstat.ML

Spencer Frazier , Mark Riedl This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords perceptualaliasingactionadviceinteractivemachinelearningdeepreinforcementMinecraftNewtonianFeedbackArbitrationsyntheticoracle
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep reinforcement learning agents struggle in 3D environments where many distinct states look alike, and this paper tests whether occasional human-style action advice can fix that. The authors adapt two learning-from-advice algorithms, Feedback Arbitration and Newtonian Action Advice, to a Minecraft item-delivery task in a maze deliberately filled with repeated textures, mirrored halves, and a hedge corridor. Using a synthetic teacher whose advice frequency and accuracy are controlled, they report that high-frequency advice beats low-frequency advice, that even 50%-accurate advice helps more than no advice, and that Newtonian Action Advice, which repeats each piece of advice for two extra timesteps, converges fastest and spends less training time stuck in aliased corridors. The point of the work is that a small amount of persistent human guidance can substitute for a great deal of exploration when visual features are ambiguous.

What carries the argument

The central mechanism is the friction parameter of Newtonian Action Advice: each advice item is repeated by the agent to itself for a fixed number of timesteps (set to 2 here) after it is first received, converted from a cardinal direction into orientation-relative movement commands. This persistence is the only substantive difference from Feedback Arbitration, whose confidence-based arbitration logic the agents share. The persistence converts advice into a short-term memory that carries the agent through stretches of visually identical states, which is exactly where a one-shot advisory signal would be washed out by aliasing. The original grid-world formulation of the algorithm is extended to a DQN so it can operate on pixel input in 3D.

What would settle it

Run the same Minecraft maze and protocol with actual human teachers giving advice at natural rates and with natural errors, and compare convergence curves and corridor visitation heatmaps for NAA, FA, and baseline DQN; if NAA does not converge faster and spend less time in aliased states than the others under human advice, the central claim fails. A sharper test: fix all parameters but set friction to 1, and if the NAA advantage over FA disappears, then advice persistence, not some other implementation detail, is the operative mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that Newtonian Action Advice, an algorithm that treats each action suggestion as remaining valid for a short persistence window (here, two timesteps) instead of being consumed once, lets a deep Q-network agent cross perceptually aliased regions more decisively. In a 20x20 Minecraft map with mirrored halves, repeated wall textures, and a hedge corridor, the NAA agent converges on a high-reward policy faster than the Feedback Arbitration agent and far faster than a baseline DQN, under all four combinations of high or low advice frequency and 100% or 50% oracle accuracy. Heatmaps of state visitation show the NAA agent distributes its visits more uniformly through the hedge corridor and spends less time tracing walls, whereas the baseline and FA agents linger in aliased states. Because the agents receive identical advice streams, the authors attribute the difference to persistence: advice acts as a short-term memory aid that bridges the ambiguity until the learned policy overcomes it.

Load-bearing premise

The entire comparison of advice robustness rests on a synthetic oracle with perfect environment information standing in for a human teacher; if real human advice has different timing, error structure, or state-dependent biases, the reported robustness to infrequent and inaccurate advice may not transfer.

Editorial extensions

If this is right

  • High-frequency advice is superior to low-frequency advice, and both vastly beat no advice, so enabling a human teacher to give frequent small suggestions is worthwhile even when cognitive load limits availability.
  • Even 50%-accurate advice outperforms no advice, meaning agents can tolerate noisy or partly mistaken human teachers, reducing the demand that humans be perfect.
  • The NAA agent reconverges in fewer than 40 episodes on a 90-degree-rotated version of the map, suggesting that the learned visual filters generalize beyond the specific maze layout.
  • Because FA and NAA receive the same advice streams but NAA uses advice more often and moves more uniformly through aliased regions, the persistent-advice mechanism, not the raw advice volume, drives the aliasing benefit.
  • Visitation heatmaps provide a diagnostic for aliasing trouble: agents that over-trace walls and linger in corridors are likely struggling with perceptual aliasing, while better-guided agents explore goal-adjacent areas more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The synthetic oracle's errors are random and unbiased, whereas real human teachers make structured, state-dependent mistakes; if human advice errs in systematic ways at specific junctions, the 50%-accuracy robustness result may be optimistic.
  • The persistence window resembles an external memory mechanism, so the friction parameter may be tunable against aliasing severity: a testable extension is sweeping friction over 1, 2, 4, and 8 timesteps and correlating it with corridor length or texture repetition.
  • If persistence is the true cause of the advantage, then on a low-aliasing map the NAA-FA gap should shrink; comparing the two algorithms on a map with distinctive landmarks would isolate the mechanism.
  • The claim that inaccurate advice still helps has an untested boundary: at some accuracy below 50%, repeated wrong suggestions could actively mislead, so the paper demonstrates robustness only down to the 50% level.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether interactive machine learning via action advice can mitigate perceptual aliasing in deep reinforcement learning for a 3D Minecraft environment. It adapts two existing learning-from-advice algorithms, Feedback Arbitration (FA) and Newtonian Action Advice (NAA), to a Deep Q-Network agent and evaluates them under four synthetic-oracle conditions varying advice frequency and accuracy, plus a baseline RL agent. The main claims are that advice-based agents converge faster than baseline RL, NAA converges faster than FA, NAA is more robust in high-aliasing regions, and even significantly inaccurate advice improves learning.

Significance. The problem—perceptual aliasing in 3D environments—is relevant and under-explored, and the paper makes a concrete contribution by extending NAA to a DQN setting, which is a natural adaptation of prior grid-world work. The synthetic-oracle methodology allows controlled variation of advice frequency and accuracy, which is a useful experimental design. If the findings were supported, the work would be a valuable step toward practical human-in-the-loop deep RL in visually aliased environments. However, the current empirical evidence is undermined by a confounded baseline comparison, a confound between advice persistence and advice representation, and the absence of any statistical analysis.

major comments (3)
  1. [Experiments (baseline compensation)] The baseline RL agent is allowed 1,500 actions per episode while FA and NAA are capped at 1,000 actions per episode. This confounds the comparisons in Figures 4 and 5: the baseline receives 50% more training experiences per episode and a longer horizon over which to accumulate reward. The claim that advice-based agents converge 'much faster' than baseline RL is therefore not cleanly supported. The authors should re-run the baseline with the same 1,000-action cap or provide a matched-budget comparison.
  2. [Methods: Newtonian Action Advice Agent vs. Feedback Arbitration Agent] The two advice algorithms differ in more than the persistence mechanism. The Methods describe NAA as receiving advice in the form of cardinal directions (north, south, east, or west), which the agent converts to a turn-then-move sequence, whereas FA is described only as receiving 'action advice' queued in a pending array, with no directional semantics specified. This representation confound matters: in the mirrored, center-symmetric map, a cardinal-direction signal explicitly disambiguates orientation in exactly the wall-facing and corridor states where the heatmaps in Figure 7 show the largest NAA–FA differences. The paper attributes NAA's faster convergence and better aliasing behavior to advice persistence, but without an ablation that holds the advice representation fixed while toggling persistence (e.g., FA with cardinal-direction advice, or NAA with atomic-action advice), this attribution is not justified.
  3. [Results (Figures 4–7)] No error bars, confidence intervals, or statistical tests accompany the reported learning curves, advice counts, or heatmap differences. The statement 'No significant difference was observed in per-episode mean advice count for high-frequency (HF) conditions (M_NAA=73, M_FA=76)' uses the term 'significant' without any inferential test. With 'at least 5 sessions per technique,' the variance across sessions is not reported, so the reader cannot assess whether the visual differences in Figures 4–6 are robust. The central claims—that NAA converges faster, that it uses advice more efficiently, and that even 50%-accurate advice is better than no advice—require either per-session variability reporting or appropriate significance tests.
minor comments (5)
  1. [Methods, Eq. (2)] The expression for relativeCost appears malformed as printed ('relativeCost = -1 ln( sqrt(... ) - 1 )'); the intended formula is unclear. Please clarify the mathematical expression and how the 0.25 threshold is motivated.
  2. [Results, Figure 7 caption and text] The caption says 'Mean accumulated actions across techniques and the KL Divergence between FA and NAA,' but the text describes the first three heatmaps as location visit frequencies and the rightmost image as a pointwise difference. Since KL divergence is a scalar, please specify the exact computation that produced the rightmost heatmap.
  3. [Results, advice count paragraph] The phrase 'No significant difference was observed' should be replaced by a report of the actual distribution or a statistical test result; as written, it is misleading without a test.
  4. [Frequency and Accuracy Variation (transfer experiment)] The transfer experiment is described in one sentence and lacks essential details: what 'reconverge' means in terms of a reward threshold, whether the map rotation preserves the same cardinal-direction semantics, whether the agent was trained from the converged weights or reset, and how FA or baseline behave under the same transfer. Without these details, the transfer claim cannot be evaluated.
  5. [Methods (hyperparameters)] The confidence threshold (0.25) and friction parameter (2) are fixed without sensitivity analysis. A brief report of how these values affect the main comparisons would help the reader judge the robustness of the conclusions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported comparisons are empirical and the cited algorithms function as baselines, not as premises that force the conclusions.

full rationale

The paper makes empirical claims about how Feedback Arbitration (FA) and Newtonian Action Advice (NAA) perform in a Minecraft navigation task with perceptual aliasing. It does not attempt a formal derivation of a predicted quantity from fitted inputs. The friction parameter (friction = 2) and the confidence threshold (relativeCost <= 0.25) are presented as fixed experimental hyperparameters, not as values fitted to the outcome, and the oracle accuracy and frequency are controlled independent variables. The citations to Lin et al. 2017 and Krening 2018, including prior work by co-author Riedl, introduce the two advice mechanisms being compared, but the paper's central conclusion that NAA converges faster and handles aliased regions better is supported by measured training curves, heatmaps, and a map-rotation transfer experiment conducted in this paper. No equation or construction in the paper makes the reported advantage true by definition. The noted confound between NAA's cardinal-direction advice representation and its persistence mechanism is a possible internal-validity concern, not a circularity, because it does not reduce the conclusion to the input definitions. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced. The synthetic oracle is a simulation tool, not an invented physical or mathematical entity. The free parameters are the friction window and the confidence threshold, both hand-chosen hyperparameters that influence the main comparison.

free parameters (2)
  • friction = 2
    Newtonian Action Advice persistence window in timesteps. The authors call it a 'very conservative' choice but provide no sensitivity analysis, so the claimed benefit of NAA may depend on this hand-picked value.
  • confidence_threshold = 0.25
    Threshold on relativeCost in Equation 2 below which the agent uses its own policy instead of pending advice. It is set by hand (inherited from prior work) and directly controls how often advice is used.
assumptions (4)
  • domain assumption The synthetic oracle with perfect knowledge of the maze is a valid proxy for real human teachers.
    The entire experimental design substitutes a parameterized oracle for human advice, and the paper's conclusions about robustness to human error and frequency rely on this proxy being representative.
  • domain assumption Perceptual aliasing is the primary cause of the baseline agent's learning difficulty in this map.
    The paper interprets the baseline's struggling behavior (spending time near walls and in the hedge corridor) as being caused by aliasing, but other factors such as sparse rewards or partial observability could also contribute.
  • domain assumption The specific DQN architecture and hyperparameters are a representative deep RL instantiation for drawing general conclusions.
    Only one network architecture, one loss function, and one set of hyperparameters are tested; the paper extrapolates that the results would hold for more sophisticated RL algorithms.
  • domain assumption Location visit frequency (heatmaps) is a valid measure of robustness to aliasing.
    The heatmap analysis in Figure 7 is used to conclude that NAA navigates aliased areas more uniformly, but no formal measure or statistical test is attached to this claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Deep Reinforcement Learning in Minecraft with Action Advice." pith.science (2026). https://pith.science/paper/XPJTTPGP

@misc{pith2026190801007,
  author       = {Pith},
  title        = {Pith review of: Improving Deep Reinforcement Learning in Minecraft with Action Advice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPJTTPGP}},
  note         = {Machine review of arXiv:1908.01007}
}
read the original abstract

Training deep reinforcement learning agents complex behaviors in 3D virtual environments requires significant computational resources. This is especially true in environments with high degrees of aliasing, where many states share nearly identical visual features. Minecraft is an exemplar of such an environment. We hypothesize that interactive machine learning IML, wherein human teachers play a direct role in training through demonstrations, critique, or action advice, may alleviate agent susceptibility to aliasing. However, interactive machine learning is only practical when the number of human interactions is limited, requiring a balance between human teacher effort and agent performance. We conduct experiments with two reinforcement learning algorithms which enable human teachers to give action advice, Feedback Arbitration and Newtonian Action Advice, under visual aliasing conditions. To assess potential cognitive load per advice type, we vary the accuracy and frequency of various human action advice techniques. Training efficiency, robustness against infrequent and inaccurate advisor input, and sensitivity to aliasing are examined.

Figures

Figures reproduced from arXiv: 1908.01007 by the authors.

Figure 2
Figure 2. An example of perceptual aliasing in Minecraft. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Environment map with 3D structure and key points [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Agent performance for all techniques as a 10- [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Agent performance for all techniques as a 10- [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Mean accumulated actions across techniques and the KL Divergence between FA and NAA. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Abbeel, P., & Ng, A. Y. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st international conference on Machine learning

    work page 2004

  3. [3]

    Abel, D., Agarwal, A., Diaz, F., Krishnamurthy, A., & Schapire, R. E. 2016. Exploratory gradient boosting for reinforcement learning in complex domains. arXiv:1603.04119

    arXiv 2016

  4. [4]

    D., Chernova, S., Veloso, M., & Browning, B

    Argall, B. D., Chernova, S., Veloso, M., & Browning, B. 2009. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5), 469-483

    work page 2009

  5. [5]

    F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. 2017. Deep reinforcement learning from human preferences. In Proceedings of NeurIPS 2017

    work page 2017

  6. [6]

    P., & Zaremba, W

    Duan, Y., Andrychowicz, M., Stadie, B., Ho, J., Schneider, J., Sutskever, I., Abbeel. P., & Zaremba, W. 2017. One-shot imitation learning. In Proceedings of NeurIPS 2017

    work page 2017

  7. [7]

    A., & Olsen Jr, D

    Fails, J. A., & Olsen Jr, D. R. 2003. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent User Interfaces

    work page 2003

  8. [8]

    Gao, Yang, et al. 2018. Reinforcement learning from imperfect demonstrations. arXiv:1802.05313

    arXiv 2018

Show all 33 references
  1. [9]

    L., & Thomaz, A

    Griffith, S., Subramanian, K., Scholz, J., Isbell, C. L., & Thomaz, A. L. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Proceedings of the 2013 Conference on Neural Information Processing Systems

    2013

  2. [10]

    Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. 2016. Continuous deep q-learning with model-based acceleration. In Proceedings of the 2016 International Conference on Machine Learning

    2016

  3. [11]

    Hausknecht, M., & Stone, P. 2015. Deep recurrent Q-learning for partially observable MDPs. arXiv:1507.06527

    2015 arXiv

  4. [12]

    L., & Shelton, C

    Isbell Jr, C. L., & Shelton, C. R. 2002. Cobot: A social reinforcement learning agent. In Proceedings of the 2002 Conference on Neural Information Processing Systems

    2002

  5. [13]

    Jaderberg, M., Czarnecki, W., Dunning, I., Marris, L., Lever, G., Castaneda, A., Beattie, C., Rabinowitz, N., Morcos, A., Ruderman, A., Sonnerat, N., Green, T., Deason, L., Leibo, J., Silver, D., Hassabis, D., Kavukcuoglu, K., & Graepel, T. 2018. Human-level performance in fir...

    2018 arXiv

  6. [14]

    Johnson, M., Hofmann, K., Hutton, T., & Bignell, D. 2016. The Malmo Platform for Artificial Intelligence Experimentation. In Proceedings of IJCAI 2015

    2016

  7. [15]

    P., Littman, M

    Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 99-134

    1998

  8. [16]

    Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Ja\'skowski, W. 2016. Vizdoom: A doom-based AI research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games

    2016

  9. [17]

    B., Stone, P., & Breazeal, C

    Knox, W. B., Stone, P., & Breazeal, C. 2013. Training a robot via human feedback: A case study. In Proceedings of the 105 International Conference on Social Robotics

    2013

  10. [18]

    Krening, S. 2018. Newtonian Action Advice: Integrating Human Verbal Instruction with Reinforcement Learning. arXiv:1804.05821

    2018 arXiv

  11. [19]

    & Feigh, K

    Krening, S. & Feigh, K. M. 2018. Interaction Algorithm Effect on Human Experience with Reinforcement Learning. ACM Transactions on Human-Robot Interaction, 7(2), 16

    2018

  12. [20]

    Krakov, D., & Feitelson, D. G. 2013. Comparing performance heatmaps. In Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing

    2013

  13. [21]

    Laird, J. E. 2002. Research in human-level AI using computer games. Communications of the ACM, 45(1), 32-35

    2002

  14. [22]

    Lin, Z., Harrison, B., Keech, A., & Riedl, M. 2017. Explore, exploit or listen: Combining human feedback and policy model to speed up deep reinforcement learning in 3D worlds. arXiv:1709.03969

    2017 arXiv

  15. [23]

    Littman, M. L. 2015. Reinforcement learning improves behaviour from evaluative feedback. Nature, 521(7553)

    2015

  16. [24]

    Liu, L. T. and Dogan, U. & Hofmann, K. 2016. Decoding multitask DQN in the world of Minecraft. In Proceedings of the 13th European Workshop on Reinforcement Learning

    2016

  17. [25]

    & Hassabis, D

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S. & Hassabis, D. 2015. Human-level control th...

    2015

  18. [26]

    Oh, J., Chockalingam, V., Singh, S., & Lee, H. 2016. Control of memory, active perception, and action in Minecraft. arXiv:1605.09128

    2016 arXiv

  19. [27]

    B., Kanazawa, A., Malik, J., Abbeel, P., & Levine, S

    Peng, X. B., Kanazawa, A., Malik, J., Abbeel, P., & Levine, S. 2018. Sfv: Reinforcement learning of physical skills from videos. In SIGGRAPH Asia 2018 Technical Papers

    2018

  20. [28]

    Schaal, S. 1997. Learning from demonstration. In Proceedings of NeurIPS 1998

    1997

  21. [29]

    & Sorg, J

    Singh, S., Lewis, R.L., Barto, A.G. & Sorg, J. 2010. Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Transactions on Autonomous Mental Development 2(2), 70–82

    2010

  22. [30]

    S., & Barto A

    Sutton R. S., & Barto A. G. eds. 1998. Reinforcement Learning: A n Introduction , MIT Press, Cambridge, MA

    1998

  23. [31]

    L., & Breazeal, C

    Thomaz, A. L., & Breazeal, C. 2006. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Proceedings of AAAI

    2006

  24. [32]

    J., & Dayan, P

    Watkins, C. J., & Dayan, P. 1992. Q-learning. Machine learning, 8(3-4), 279-292

    1992

  25. [33]

    Zhumatiy, V., Gomez, F., Hutter, M., & Schmidhuber, J. 2006. Metric state space reinforcement learning for a vision-capable mobile robot. In Proceedings of the 9th International Conference on Intelligent Autonomous Systems

    2006

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.