{"id":"bef02028-8a49-486a-ba48-8c622195a7e3","arxiv_id":"2506.14162","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Simple neural policy variants can match programmatic policies on several OOD generalization benchmarks, challenging the claim that programmatic representations are inherently better at generalizing.","lead":"This position paper re-examines benchmarks where programmatic policies appeared to beat neural policies at out-of-distribution generalization. It shows that simple changes to neural policies, such as sparser observations or a safety-oriented reward, can close much of the gap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TORCS/KAREL gap-closure relies on programmatic baselines imported from different codebases and seed protocols, and TORCS OOD rates are computed only over training-successful seeds, so the 'simple changes' explanation is confounded.","rationale":"Agreeing with the reader: the weakest assumption is comparability of imported baselines. I find this genuinely load-bearing because both key supporting experiments quote external numbers, and the causal claims ('cautious reward', 'sparse observation') are supposed to isolate one variable but instead change many. The TORCS denominator issue is a separate visible flaw that strengthens the concern. Credit: the PARKING results are newly generated for both representations and the paper is transparent about metric dependence; the SparseMaze idea is a proposal, not evidence for the main claim. Since the experimental weaknesses are addressable by a matched re-benchmark and the paper is framed as a position paper, the appropriate verdict remains CONDITIONAL rather than REJECT or ACCEPT.","tokens_in":15272,"tokens_out":16173,"duration_ms":179150,"concrete_test":"Re-run the TORCS and KAREL comparisons in the authors' released environment wrappers with all conditions matched: same seed count (e.g., 30), same evaluation protocol, and same observation/reward definitions, including NDPS/PROPEL and LEAPS in the same wrapper, plus DRL β=1.0 as a control in the authors' DDPG code. If β=1.0 also generalizes, or if the imported programmatic baselines shift materially, the paper's causal story and gap-closure conclusion do not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that simple pipeline changes close the programmatic/neural OOD gap requires a controlled comparison, but Tables 1 and 2 are not controlled. In §4.1 the NDPS and DRL β=1.0 rows are imported from Verma et al. (2018), while DRL β=0.5 is run in the authors' own DDPG code; in §4.2 the LEAPS/PPO-ConvNet/PPO-LSTM rows are imported from Trivedi et al. (2021, rows marked †), while PPO with at−1 is run by the authors. Codebase, environment wrapper, evaluation seeds, and sometimes reward settings change together with the claimed causal variable (β or observation). A second independent problem is the TORCS denominator: the parenthetical success fractions 0.76/0.69/1.00 are 'out of 13 and 4' seeds that completed training laps; as fractions of all 30/15 runs they are 0.33, 0.30, and 0.27. Without β=1.0 rerun in the new code and without the programmatic methods' run-to-run failure rates under the same protocol, the 'gap disappears' conclusion is confounded with implementation and selection effects.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper revisits out-of-distribution (OOD) generalization claims from four programmatic-policy papers and argues that the apparent superiority of programmatic representations is partly an artifact of uncontrolled pipeline choices. The authors present three empirical re-analyses. In TORCS (§4.1), they train DDPG agents with a cautious reward (β=0.5) and report that many of the resulting policies generalize to unseen tracks, contrasting with the earlier DRL β=1.0 failures imported from Verma et al. (2018). In KAREL (§4.2), they train PPO with a fully connected network and the previous action appended to a sparse observation, and report generalization to 100×100 grids comparable to or better than the LEAPS/PPO baselines imported from Trivedi et al. (2021). In PARKING (§4.3), they compare PSM and DQN and conclude that both struggle but PSM generalizes slightly better on some metrics. Finally (§5), they propose SparseMaze, a sparse-corridor short-est-path task, and report that a FunSearch-synthesized Python BFS program generalizes while PPO with a GRU fails. The paper concludes that benchmarks undervalue programmatic policies because simple neural training changes close the OOD gap, and suggests future benchmarks that require algorithmic data structures.","tokens_in":15490,"tokens_out":2897,"duration_ms":32027,"significance":"If the central comparison were tightly controlled, this would be a valuable contribution: it would undermine the evidence base for the common claim that programmatic representations generalize better on TORCS and KAREL, and it would redirect attention to benchmark design. The paper's strengths include public code, explicit hyperparameter sweeps in appendices, and an honest presentation of the metric-dependent PARKING result. However, as it stands, the load-bearing comparisons are not controlled: programmatic and neural baselines are drawn from different codebases, seed protocols, and sometimes reward/observation settings, and the neural success rates are computed after filtering out seeds that failed to learn the training task. The evidence therefore supports a weaker claim than the abstract states: certain neural training choices can help OOD generalization on these benchmarks, but the paper does not yet show that the programmatic-vs-neural gap disappears under a fair controlled comparison.","major_comments":[{"comment":"The central TORCS comparison is not controlled. The NDPS and DRL β=1.0 rows are imported from Verma et al. (2018), while DRL β=0.5 is run in the authors' own DDPG implementation. Thus the reward coefficient β changes together with codebase, environment wrapper, observation preprocessing, and evaluation seeds. To support the claim that 'simple changes to the reward function' close the OOD gap, the authors must re-run DRL β=1.0 in their own code and, ideally, run NDPS under the same protocol; otherwise the observed gap closure could be an implementation effect rather than a reward effect.","section":"§4.1, Table 1"},{"comment":"The reported generalization fractions 0.76, 0.69, and 1.00 are computed only over seeds that successfully completed training laps (13 of 30 and 4 of 15). As fractions of all trained seeds, the corresponding success rates are 0.33, 0.30, and 0.27 (10/30, 9/30, and 4/15). Selecting on training success inflates the OOD generalization estimate, and the paper does not report the training-success rate of the imported programmatic and β=1.0 baselines under the same criterion. Without a common denominator, the 'most models trained with β=0.5 generalize' statement is not supported.","section":"§4.1, Table 1"},{"comment":"The KAREL comparison has the same confound: LEAPS, PPO-ConvNet, and PPO-LSTM rows are imported from Trivedi et al. (2021) (marked †), while the new 'PPO with at−1' rows are produced by the authors' code. The observation change (sparse perception plus last action) is therefore entangled with differences in environment implementation, reward setup, training protocol, and evaluation seeds. A stronger design would run at least the fully observable PPO-ConvNet and partially observable PPO-LSTM baselines in the same codebase, or report that the imported numbers were reproduced under identical conditions. The claim that sparse observations and a simpler model cause the improvement is not yet isolated.","section":"§4.2, Table 2"},{"comment":"The PARKING conclusion is metric-dependent and the text acknowledges this but still draws a directional conclusion. By test 'Success Rate', DQN (0.18) outperforms PSM (0.16); by the training-to-test gap, PSM looks better (0.10 vs 0.68); by 'Successful-on-100', both are near zero (0.06 vs 0.00). The summary statement that 'programmatic policies generalized slightly better than neural policies' is not robust to the choice of metric, and no confidence intervals or significance tests are provided. The paper should either commit to a prespecified primary metric or present the result as inconclusive for both representations.","section":"§4.3, Table 3"},{"comment":"The SparseMaze experiment is a proof-of-concept but is not yet evidence for the paper's benchmarking proposal. FunSearch results are reported for a single run ('a single run of the system'), while PPO results are averaged over 30 seeds; moreover, the PPO-GRU baseline fails to learn the training task (return 0.09 on Original), so the comparison does not demonstrate OOD generalization advantage over a neural policy that actually solves the training distribution. Multiple FunSearch runs (or at least a statement of run-to-run variance) and a neural baseline that learns Original are needed before concluding that programmatic representations uniquely handle this task.","section":"§5, Table 4"}],"minor_comments":[{"comment":"The reward expression is referred to as 'Equation 2' in the text, but it is not numbered in the manuscript; please add a number or remove the reference.","section":"§4.1, Equation (2)"},{"comment":"The table header says 'Successful Rate' while the text consistently uses 'Success Rate'; please harmonize the terminology.","section":"§4.3, Table 3"},{"comment":"For 'PPO with at−1', please specify exactly which observation features are included (the four local sensors, the last action, and any absolute information such as marker presence) and how this maps to the perception functions of LEAPS. The current description leaves ambiguity about whether the observation spaces are truly matched.","section":"§4.2, Table 2"},{"comment":"The selected value for 'Action resolution' is listed as '2 million', which appears to be a copy-paste error from the replay buffer size; this makes the hyperparameter table internally inconsistent.","section":"Appendix C, Table 10"},{"comment":"Several reference entries contain formatting artifacts, e.g., '733?742' in the Grooten et al. entry and 'wake?sleep' in the Ellis et al. entry; these should be corrected to proper en-dashes or hyphens.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful message and is likely to provoke discussion, but the current evidence is not yet as strong as the abstract claims. The main risk is that the imported baselines from Verma et al. (2018) and Trivedi et al. (2021) are not directly comparable to the authors' new runs; reviewers and readers may see the central gap-closure conclusion as an artifact of uncontrolled variables. I would encourage the editor to request a revision that either re-runs the baselines in the same codebase or substantially tones down the causal language in the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper shows that simple changes to the neural training pipeline — sparser observations, a safer reward, last-action input — can close the OOD generalization gap on TORCS and KAREL that earlier papers attributed to programmatic representations. That is a useful corrective for the programmatic RL community. But the experiments as reported do not fully support the strong version of the claim, because the programmatic baselines come from different codebases and the TORCS success rates are computed over a selected subset of seeds.\n\nWhat is genuinely new: PPO with last-action input on KAREL, DDPG with beta=0.5 on TORCS, DQN with shaped rewards on PARKING, and a SparseMaze proof-of-concept with FunSearch. The KAREL result is the most striking — a feedforward net with last-action input hits 1.00 on four of five 100x100 tasks while the published ConvNet/LSTM baselines collapse. The paper is also honest about its own weaknesses: it flags the PARKING metric ambiguity and admits SparseMaze is a single FunSearch run.\n\nWhere it gets soft: Tables 1 and 2 mix the authors' runs with rows pulled from Verma et al. and Trivedi et al. Those baselines used different codebases, observation pre-processing, and seed protocols. When the independent variable is observation sparsity or the reward coefficient and the compared systems change several things at once, the gap closure could be an implementation artifact. TORCS is the bigger problem: the 0.76/0.69/1.00 success fractions are out of the 13 and 4 seeds that completed training laps, not out of all 30 and 15 runs. As fractions of all runs, they are roughly 0.33/0.30/0.27. Without re-running beta=1.0 in the same code and reporting the programmatic methods' own seed-to-seed failure rates, the 'simple changes close the gap' conclusion is shaky. The beta=0.5 choice is also post hoc, discovered after observing the speed-generalization pattern; that is parameter fitting, not a hypothesis test. None of this kills the thesis, but it means the thesis is plausible and partially supported rather than demonstrated.\n\nWho should read it: anyone working on programmatic policies or OOD generalization in RL. Editors should send it to peer review. A good referee should demand matched-codebase baseline runs, all-seed denominators, and significance tests. With those, the paper could become a solid reference point.","headline":"Useful empirical critique of programmatic-policy generalization claims, but the central claim outruns the evidence because baselines are imported and TORCS success rates use a selected denominator.","tokens_in":16100,"tokens_out":3229,"would_cite":true,"duration_ms":29817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural policies can match programmatic policies on out-of-distribution problems when given the same sparse observations and a reward function that encourages caution.","keywords":["programmatic reinforcement learning","out-of-distribution generalization","neural policies","sparse observations","partial observability","reward shaping","benchmark design","KAREL"],"falsifier":"Retrain the programmatic methods under exactly the same sparse observations and cautious reward used for the neural baselines, with the same number of seeds and the same evaluation protocol. If the programmatic policies still generalize while the neural policies fail, the paper's artifact explanation collapses; if the programmatic policies also lose the advantage, the paper's account is supported.","tokens_in":15034,"feed_emoji":"🧠","tokens_out":8787,"duration_ms":81483,"temperature":0.7,"pith_summary":"This paper re-examines the widely reported result that programmatic (code-based) policies generalize better than neural policies to out-of-distribution (OOD) problems. It argues that on the TORCS car-racing and KAREL grid-world benchmarks, the apparent gap comes from uncontrolled variables, not from the representation: when neural policies are given the same sparse observations as the programmatic policies and a reward that encourages safer driving, they generalize just as well. The paper also proposes that future benchmarks should target algorithmic structure such as stacks and queues, where programmatic policies have a real advantage, and demonstrates this with SparseMaze.","feed_headline":"Sparse inputs let neural policies match programmatic generalization","feed_subtitle":"Re-running TORCS and KAREL with matched observations and slower-driving rewards erases the gap.","key_machinery":"The argument runs on two simple interventions plus one new benchmark. The first intervention is observation matching: the neural policy receives only the sparse percepts the programmatic language exposes (in KAREL, the local perception functions frontIsClear, leftIsClear, rightIsClear, markersPresent) augmented with its previous action, which removes the spurious correlations that a fully observable convolutional network can latch onto. The second is reward caution: lowering the coefficient β in the TORCS reward from 1.0 to 0.5 makes the learned driving policy slower and therefore safer on unseen sharp turns. The new benchmark, SparseMaze, is the load-bearing object for the paper's positive proposal: a sparse maze wide enough that a 'follow the right wall' strategy fails, so solving it requires remembering visited cells and using a queue or stack, which is exactly the algorithmic structure that programmatic policies can express and that the tested neural policies could not learn.","core_discovery":"On its own terms, the paper's central claim is that the OOD generalization advantage previously attributed to programmatic policies on TORCS and KAREL is not a property of the representation. With a fully connected network given the agent's last action as part of a sparse observation, PPO reaches near-perfect returns on 100x100 versions of four KAREL tasks (STAIRCLIMBER, MAZE, TOPOFF, FOURCORNER); with a cautious reward coefficient (β=0.5), DDPG generalizes to unseen TORCS tracks in the majority of trained seeds. In PARKING, neither representation generalizes reliably, so the paper treats it as an open problem. The paper then constructs SparseMaze, a maze with no wall to follow, where a FunSearch-synthesized program implementing breadth-first search generalizes to 100x100 maps while PPO with a GRU fails to learn even the small map.","pith_inferences":["If this argument extends, many published programmatic-vs-neural OOD comparisons may shrink once training pipelines are matched, so the field would benefit from standardized evaluation protocols that report observation, reward, and seed counts.","The last-action augmentation trick is a cheap way to give feedforward policies a limited memory; it may solve other benchmarks that were thought to require recurrent networks.","The cautious-reward intervention suggests a testable symmetry: training a programmatic search method with a fast, aggressive reward may erase its OOD advantage too, which the paper conjectures but does not test.","SparseMaze points to a broader classification of benchmarks by the data structures they require; such a taxonomy could predict where programmatic policies are genuinely needed rather than merely preferred."],"forward_implications":["The OOD results on TORCS and KAREL should not be used as evidence that programmatic representations generalize better than neural ones; comparisons must match observation spaces, reward functions, and training budgets.","Adding the previous action to a sparse observation can turn a feedforward network into a policy that generalizes to much larger KAREL grids, suggesting that reported LSTM advantages in partially observable benchmarks deserve re-examination.","Reward shaping that slows the agent down can convert a non-generalizing TORCS policy into one that generalizes, so claims about representation effects in continuous control should control for the learned policy's risk profile.","PARKING remains a domain where both programmatic and neural policies struggle, so it does not support either side's generalization claims.","The SparseMaze proof of concept shows a concrete benchmark where a programmatic, queue-based solution generalizes perfectly while a neural policy fails, which is the kind of problem the field should use to test programmatic advantages."],"supporting_citations":[{"why":"Supplies the NDPS programmatic results and TORCS benchmark setup that the paper re-analyzes, with Table 1 importing its data.","marker":"[Verma et al., 2018]"},{"why":"Supplies the LEAPS programmatic results and the KAREL protocol of training on small grids and testing on 100x100 grids reused in Table 2.","marker":"[Trivedi et al., 2021]"},{"why":"Introduces the PARKING domain and the PSM programmatic state-machine policies used as the programmatic baseline in Table 3.","marker":"[Inala et al., 2020]"},{"why":"Supplies FunSearch, the program-search method that synthesized the breadth-first search policy for SparseMaze.","marker":"[Romera-Paredes et al., 2023]"},{"why":"Provides the PPO algorithm used to train the neural baselines in KAREL and SparseMaze.","marker":"[Schulman et al., 2017]"},{"why":"Provides the DDPG algorithm used to train the neural driving policies in TORCS.","marker":"[Lillicrap et al., 2019]"}],"fun_headline_variants":["Neural policies match programmatic OOD generalization with simple fixes","Benchmark choices inflate the programmatic policy advantage","Sparse inputs close the generalization gap between policy types","Programmatic policies' OOD edge shrinks under fair benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the programmatic results imported from the earlier TORCS and KAREL studies are directly comparable to the newly trained neural policies, even though they came from different codebases, seeds, observation preprocessing, and possibly reward settings.","fun_headline_variants_meta":{"raw":{"variants":["Neural policies match programmatic OOD generalization with simple fixes","Benchmark choices inflate the programmatic policy advantage","Sparse inputs close the generalization gap between policy types","Programmatic policies' OOD edge shrinks under fair benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2258,"prompt_tokens":915,"completion_tokens":1343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1276}},"tokens_in":531,"tokens_out":1343,"duration_ms":11200,"temperature":1.0,"reasoning_tokens":1276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:18:03.458621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the programmatic methods under exactly the same sparse observations and cautious reward used for the neural baselines, with the same number of seeds and the same evaluation protocol. If the programmatic policies still generalize while the neural policies fail, the paper's artifact explanation collapses; if the programmatic policies also lose the advantage, the paper's account is supported.","supporting_citations":[{"cited_title":"Programmatically interpretable reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the NDPS programmatic results and TORCS benchmark setup that the paper re-analyzes, with Table 1 importing its data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LEAPS programmatic results and the KAREL protocol of training on small grids and testing on 100x100 grids reused in Table 2."},{"cited_title":"Synthesizing programmatic policies that inductively generalize","cited_arxiv_id":null,"evidence_quote":"Introduces the PARKING domain and the PSM programmatic state-machine policies used as the programmatic baseline in Table 3."},{"cited_title":"Pawan Kumar, Emilien Dupont, Francisco J","cited_arxiv_id":null,"evidence_quote":"Supplies FunSearch, the program-search method that synthesized the breadth-first search policy for SparseMaze."}],"review_version":1}