{"id":"77b2419a-a641-4583-9f3f-130460833d14","arxiv_id":"2411.11457","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Random forests and extra trees match a neural network as UDRL behavior functions on CartPole, Acrobot, and Lunar Lander, and provide feature importance explanations.","lead":"This paper tests whether non-neural machine learning models, mainly random forests and extremely randomized trees, can replace neural networks as the action predictor in Upside-Down Reinforcement Learning. On three classic control tasks the tree ensembles performed roughly on par with a neural network and added feature importance scores as an interpretability byproduct.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's comparison is confounded: each behavior function is queried at its own training-history (dr,dt), so 'performs just as well' is not fairly tested.","rationale":"The reader's weakest assumption is exactly the load-bearing point. The comparison in Table 4 is not a controlled experiment because the command (dr,dt) is part of the input to the behavior function; different commands define different tasks. Per-algorithm selection from 'most common used values in last 100 episodes' means each model is tested at a point tailored to its own distribution. On Acrobot, NN's target (-63,64) is far more achievable than XGBoost's (-114,120), so the reported NN advantage may be an artifact. On Lunar Lander, XGBoost's (136,389) is extreme, while RF's (57,102) is moderate, so RF's reported best performance could be due to an easier query. The paper's own training curves show overlapping, high-variance performance, and no significance tests are reported, but those are secondary. The Section 6 limitation about simple benchmarks is honest and not the central issue. The interpretability contribution (feature importance from RF/ET) is plausible and could stand even if performance parity is not established; however, the abstract and conclusion claim parity, so the fairness of the comparison is the single load-bearing assumption. The concrete test would settle it; until then, a conditional verdict requiring a fixed/justified command protocol and significance testing is appropriate. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":11946,"tokens_out":6210,"duration_ms":62534,"concrete_test":"Fix a single command protocol and re-run inference: for Acrobot query every trained behavior function with command pairs {(-60,60), (-80,80), (-100,100), (-120,120)}; for Lunar Lander use {(0,100), (50,100), (50,200), (100,200)}; run 100 seeded episodes per (model, command) and report mean +/- std. Compare NN vs RF/ET on the same commands using paired bootstrap confidence intervals. If the best model or the conclusion 'just as well' changes across commands, Table 4's per-algorithm commands are the driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the inference protocol in Section 4 (Table 3). UDRL's behavior function is conditional on the command (dr,dt); comparing models by querying each at the 'most common' command from its own last 100 training episodes is not a controlled comparison. For Acrobot, NN is evaluated at (dr,dt)=(-63,64), while RF at (-79,82) and XGBoost at (-114,120); for Lunar Lander, RF at (57,102), NN at (49,101), and XGBoost at (136,389). These commands differ in difficulty, so Table 4's rankings (NN best on Acrobot, RF best on Lunar Lander) may reflect command selection rather than policy quality. The abstract's 'perform just as well' claim depends on this comparison; if a common command grid changes the ranking, the claim is unsupported. Secondary concerns (no significance tests, large standard deviations, no released code) compound the issue but do not replace it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether non-neural supervised learners can serve as the behavior function in Upside-Down Reinforcement Learning. The authors train six function approximators (an MLP, random forest, extra-trees, AdaBoost, XGBoost, and KNN) on CartPole, Acrobot, and Lunar Lander using a standard UDRL protocol with epsilon-greedy exploration and an experience replay buffer. They report training curves and inference results, and they use impurity-based feature importance to argue that tree-based behavior functions provide interpretable policies. The central claim is that random forests and extra-trees can perform as well as neural networks while being more interpretable.","tokens_in":12126,"tokens_out":6385,"duration_ms":63003,"significance":"The question is timely: UDRL is usually implemented with neural networks, and showing that tabular supervised learners work would extend the paradigm to settings where interpretability matters. The paper's strengths are its clear formulation of UDRL with non-neural regressors, the inclusion of five random seeds, the release of an interactive demo of trained behavior functions, and the feature-importance analyses for three environments. If the main comparison were adequately controlled, the result would be a useful practical finding. As it stands, however, the inference protocol confounds algorithm quality with command selection, and the statistical evidence is too weak to support the headline claim.","major_comments":[{"comment":"The central comparison is confounded by the inference-command protocol. For Acrobot and Lunar Lander, each behavior function is queried with the most common (dr, dt) from its own last 100 training episodes; the commands differ across algorithms (e.g., on Acrobot the NN uses f(., -63, 64) while XGBoost uses f(., -114, 120); on Lunar Lander the RF uses f(., 57, 102) while XGBoost uses f(., 136, 389)). Since the behavior function is explicitly conditional on dr and dt, the differences in Table 4 can reflect command difficulty rather than policy quality. The 'perform just as well' claim in the abstract and Section 6 depends on this comparison, so the authors should evaluate all behavior functions on a common set of commands (or a pre-specified fixed protocol) and report results per command.","section":"Section 4, Tables 3 and 4"},{"comment":"No significance tests are provided, and the standard deviations are large relative to the reported differences. On Acrobot, NN (-75.00 ± 15.36) overlaps with RF (-100.05 ± 62.80) and ET (-100.00 ± 93.72); on Lunar Lander, RF (-54.74 ± 96.22), XGBoost (-76.96 ± 89.69), and NN (-157.04 ± 71.26) all have wide, overlapping intervals. The ranking of the 'best' behavior function and the assertion that tree methods 'perform just as well' therefore require paired statistical tests or confidence intervals over the five seeds; the current evidence is not sufficient.","section":"Section 4, Table 4"},{"comment":"The interpretability claim is only illustrated, not validated. The feature-importance plots come from a trained RF or ET behavior function, but no stability analysis across seeds is shown, and there is no comparison with post-hoc explanations of the NN baseline. Since the paper's stated contribution is 'more interpretable optimal control,' the authors should at least show that the importances are stable across the five training runs and discuss how faithfully they reflect the policy's decisions.","section":"Section 5"}],"minor_comments":[{"comment":"In the caption, 'd f' should be 'dt'.","section":"Table 1"},{"comment":"The caption says 'desired reward dt' but should say 'desired reward dr'.","section":"Table 4"},{"comment":"The text says 'avoid queer-ing the behavior function'; this should be 'avoid querying'.","section":"Section 4"},{"comment":"KNN is described as a 'clustering method'; it is an instance-based regression and classification method, not a clustering method.","section":"Section 3.1"},{"comment":"XGBoost is cited to Friedman (2001); the standard reference is Chen and Guestrin (2016).","section":"Section 3.1"},{"comment":"Adding shaded confidence bands or per-seed traces would make the convergence claims checkable; the averaged curves alone do not convey run-to-run variance.","section":"Figure 2"},{"comment":"The text refers to 'Appendix 6' but the appendix is unnumbered; adding a label would help readers locate the pseudo-code.","section":"Appendix"},{"comment":"For reproducibility, the source code for the UDRL training loop should be released alongside the trained behavior functions.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is an interesting empirical study, but the headline claim is not yet supported by the evidence. The confounded inference protocol is fixable with additional experiments, so I recommend major revision rather than rejection. The paper would also benefit from a clearer statement of what 'perform just as well' means statistically."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a small, honest empirical paper that asks whether you can swap the neural network in Upside-Down RL for random forests, extra trees, boosting, or k-NN. The honest answer is \"maybe, but the paper's headline claim is ahead of its evidence.\" The new bit is real: no one has apparently tried non-neural function approximators in UDRL, and the paper shows they can learn on CartPole, Acrobot, and Lunar Lander. That is a legitimate contribution, and the feature-importance analysis is a useful extra.\n\nWhat the paper does well: it is clearly written, uses standard benchmarks, runs five seeds, and acknowledges the main limitation (simple environments). The training curves are informative, and the authors point to a demo site.\n\nThe soft spots are proportionate, and the main one is the one flagged in the stress test. The inference comparison in Table 4 is confounded. For Acrobot and Lunar Lander, each behavior function is queried at a (dr,dt) pair taken from its own last 100 training episodes. That means the models are not evaluated under the same command distribution; the NN is queried at (-63,64) on Acrobot, while XGBoost is at (-114,120), and so on. Different commands mean different difficulty, so the ranking of models could reflect command selection rather than policy quality. This is a real problem for the abstract's claim that tree methods \"perform just as well\" as NNs. The claim also does not fully hold on the numbers: the NN is best on CartPole and Acrobot, and only on Lunar Lander does a forest win. Given the large standard deviations and no significance tests, the honest reading is that the comparison is inconclusive, not that trees are on par.\n\nThe interpretability angle is a bit overstated. Impurity-based feature importance from an ensemble is a global heuristic, not \"inherently interpretable\" in the way a single decision tree is. The authors do acknowledge that forests are less interpretable than a single tree, so they are not naive about it.\n\nThe appendix pseudocode has rough edges, and code is not released (the demo is a browser app, not the training pipeline). Those are minor.\n\nAll in all: the direction is sound and worth a serious referee, but the paper needs a controlled command protocol, significance testing, and a softer claim before the comparative result is settled. This is for researchers interested in interpretable RL or UDRL specifically. I would not cite it in the next year as evidence that trees match NNs in UDRL, but I would send it to review.\n\nBest.","headline":"A useful first look at non-neural UDRL, but the headline comparison is confounded by per-model command selection and lacks statistical support.","tokens_in":12709,"tokens_out":3081,"would_cite":false,"duration_ms":29198,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that UDRL's behavior function can be learned by tree ensembles, matching neural network performance on three control benchmarks while offering inherent interpretability.","keywords":["Upside-Down Reinforcement Learning","Random Forests","Extremely Randomized Trees","Interpretability","Feature Importance","Optimal Control","Supervised Learning","Behavior Function"],"falsifier":"Run the same UDRL agents on Acrobot and Lunar Lander but query every behavior function with a single common (dr, dt) pair, say the values used for the neural network, and measure cumulative reward over 100 episodes. If the random forest and extra-trees scores fall well below the neural network score under this fixed query, the paper's central claim of on-par performance would be contradicted.","tokens_in":11703,"feed_emoji":"🌲","tokens_out":5542,"duration_ms":65446,"temperature":0.7,"pith_summary":"The paper asks whether Upside-Down Reinforcement Learning (UDRL) truly requires neural networks as its function approximator, or whether other supervised learners can serve as the behavior function. UDRL recasts control as supervised learning: the agent learns a mapping from a state, a desired reward, and a desired time horizon to an action. On CartPole, Acrobot, and Lunar Lander, the authors test five non-neural learners and report that random forests and extremely randomized trees achieve rewards comparable to a multilayer perceptron, with forests even besting the network on Lunar Lander. The interpretability benefit is that tree ensembles natively provide feature importance scores, so the learned policy can be inspected and audited. This matters because transparent policies are easier to trust in high-stakes control applications.","feed_headline":"Tree models rival neural nets in upside-down reinforcement learning","feed_subtitle":"Random forests and extra trees match network policies on control benchmarks and show what the policy actually uses.","key_machinery":"The central object is the behavior function f(s, dr, dt) = a, a supervised map from the current state, a desired reward dr, and a desired time horizon dt to the action that supposedly achieves that reward within that horizon. Commands are sampled from recent successful episodes, transitions are stored in a replay buffer, and each learner fits this map from the collected data. The interpretability machinery is the mean impurity decrease feature importance computed from the tree ensembles, which ranks how much each state component and command contributes to the splitting decisions at inference time.","core_discovery":"The central claim is that the UDRL behavior function f(s, dr, dt) -> a does not need to be a neural network: tree-based supervised learners can learn it just as well. On CartPole, Acrobot, and Lunar Lander, the authors find that random forests and extremely randomized trees yield cumulative rewards comparable to a fully connected network after 500 training episodes, with forests performing best on Lunar Lander and XGBoost close behind. Because trees expose mean impurity decrease, the learned policy can be inspected: the paper reports which state variables dominate decisions at different phases of an episode, such as pole angular velocity for CartPole, angular velocities of both links for Acrobot, and the vertical position of the spaceship for Lunar Lander. The authors conclude that UDRL is not a neural-network-only paradigm and that forests offer an intrinsically interpretable alternative for optimal control.","pith_inferences":["The reported importance scores hint that UDRL controllers discover physically meaningful variables — pole angular velocity for balancing, altitude for landing — which could be used to validate a policy against human intuition before deployment.","Extending the same tree-based UDRL recipe to image-based tasks would likely require random subwindows or convolutional ensembles, a direction the authors mention but do not test; a concrete next step is to measure whether such forests scale beyond the three low-dimensional benchmarks.","A fixed-query evaluation protocol, where all behavior functions are tested with identical desired-return and horizon commands, would provide a sharper comparison than the per-model commands used here, and would clarify whether tree methods truly match neural networks off-policy.","The finding suggests a broader design principle: behavior functions are supervised maps, so any inductive bias that suits tabular data — including gradient boosting — can be dropped into UDRL, potentially making the paradigm a test bed for interpretable control."],"forward_implications":["UDRL agents can be built with transparent models, removing the black-box nature of neural policies on low-dimensional control tasks.","Feature importance scores give a direct explanation of which physical state variables drive a learned controller, without post-hoc explanation methods.","Boosting algorithms such as XGBoost are viable behavior functions in UDRL, expanding the toolset beyond tree ensembles.","The approach suggests a practical route to policy auditing: inspect the ranking of state features before deploying a controller in a high-stakes setting.","K-nearest neighbours performs poorly, indicating that instance-based methods lack the generalisation needed for UDRL behavior functions."],"supporting_citations":[{"why":"Introduces the UDRL formalism of mapping desired rewards and horizons to actions, which is the framework the paper extends.","marker":"Schmidhuber (2019)"},{"why":"Provides the first successful UDRL implementation with a neural network and supplies the training and exploration protocol the paper follows.","marker":"Srivastava et al. (2019)"},{"why":"Defines Random Forests, the primary tree-based behavior function tested in the paper.","marker":"Breiman (2001)"},{"why":"Defines Extremely Randomized Trees, the second tree-based behavior function tested in the paper.","marker":"Geurts et al. (2006)"},{"why":"Provides the mean impurity decrease feature importance method used for the interpretability analysis.","marker":"Louppe et al. (2013)"},{"why":"Shows tree-based methods applied to batch reinforcement learning, motivating the use of trees as function approximators in RL.","marker":"Ernst et al. (2005)"}],"fun_headline_variants":["Tree-based UDRL matches neural nets with interpretable policies","Upside-down RL: trees match neural nets, add interpretability","UDRL without neural nets: forests match networks and explain actions","Interpretable optimal control via tree-based upside-down RL","Trees rival neural nets in UDRL, with built-in interpretability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that querying each behavior function with command values taken from its own recent training episodes — rather than with one fixed, neutral command — is a fair way to compare how well the models control the environment.","fun_headline_variants_meta":{"raw":{"variants":["Tree-based UDRL matches neural nets with interpretable policies","Upside-down RL: trees match neural nets, add interpretability","UDRL without neural nets: forests match networks and explain actions","Interpretable optimal control via tree-based upside-down RL","Trees rival neural nets in UDRL, with built-in interpretability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001219,"raw_usage":{"total_tokens":4983,"prompt_tokens":886,"completion_tokens":4097,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":4008}},"tokens_in":502,"tokens_out":4097,"duration_ms":28127,"temperature":1.0,"reasoning_tokens":4008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:30:48.614045+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same UDRL agents on Acrobot and Lunar Lander but query every behavior function with a single common (dr, dt) pair, say the values used for the neural network, and measure cumulative reward over 100 episodes. If the random forest and extra-trees scores fall well below the neural network score under this fixed query, the paper's central claim of on-par performance would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mean impurity decrease feature importance method used for the interpretability analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows tree-based methods applied to batch reinforcement learning, motivating the use of trees as function approximators in RL."}],"review_version":1}