{"id":"9341bda0-70e0-4405-bc14-af521193e0c2","arxiv_id":"2411.17253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An imitation-learning planner that reuses its own historical plan embeddings improves NuPlan closed-loop reactive score to 81.25, marginally above the expert baseline of 81.24.","lead":"LHPF is a driving planner that stores its own plans from recent frames and feeds them into the current decision step, so intentions stay continuous across time. On the NuPlan closed-loop benchmark it reports a reactive score of 81.25, just above the expert baseline of 81.24, a first for a purely learning-based planner if the margin holds.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first purely learning-based planner to surpass the expert' claim rests on a 0.01-point R-Score margin from a single training run with no seed or variance information; this headline result is not yet established.","rationale":"The reader's weakest_assumption correctly identifies the 0.01-point margin over the expert baseline as the critical fragility. My stress-test confirms that this is the most load-bearing concern: the paper's central claim of being the first purely learning-based planner to surpass the expert depends entirely on that single number, and the paper provides no evidence that the margin is repeatable across training seeds or robust to evaluation details. The Progress-number inconsistency in Section 5.1 is a concrete reporting error that amplifies the concern about numerical reliability, but even without it, the absence of any variance estimate for the primary headline result would warrant a conditional verdict. The proposed mechanism (historical planning embeddings) is plausible and the improvement over PLUTO (78.76 to 81.25) is larger, but the milestone claim is specifically about surpassing the expert, so it needs stronger statistical support. A 5-seed training study is a direct, feasible test that would settle whether the margin is real or an artifact of a single run. I agree with the reader's CONDITIONAL verdict: the paper should be accepted only if the headline claim is either backed by repeated runs and error bars or explicitly softened to a tentative result.","tokens_in":13,"tokens_out":10103,"duration_ms":156893,"concrete_test":"Train the STD module (with the frozen PLUTO backbone) from 5 independent random seeds, evaluate each checkpoint on the full Val14 reactive closed-loop benchmark, and report the mean, standard deviation, and minimum R-Score. If the minimum R-Score across seeds exceeds 81.24, the headline claim is reproducible; if any seed falls below, the claim is not established. Additionally, re-run the evaluation of the released checkpoint (if available) to verify the exact 81.25 score and confirm the NuPlan simulator's determinism under fixed seeds, so that the only variance source is training stochasticity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central novelty is the claim that LHPF is the first purely learning-based planner to outperform the expert, supported by R-Scores of 81.25 (LHPF) vs. 81.24 (Expert Log-Replay) in Table 1. This 0.01-point gap is smaller than typical run-to-run variation in training stochastic neural planners, yet the paper reports a single evaluation with no random seeds, no repeated runs, and no error bars or significance test. The uncertainty is entirely on LHPF's side: the expert score is a published fixed baseline, so any training-seed or evaluation variance could push LHPF below 81.24, collapsing the headline milestone. The paper's own internal inconsistency in Section 5.1 (it claims Progress improved 'from 78.59 to 79.21', while Table 1 shows PLUTO at 75.08 and LHPF at 78.59) further weakens confidence in the precise numerical reporting that the headline depends on. If the margin is noise, the method may still improve over PLUTO, but the 'first to outperform the expert' claim—repeated in the abstract and introduction—would be unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LHPF, an imitation-learning planner built on the PLUTO backbone. It augments a query-based decoder with a historical intention pool: planning embeddings from past frames are stored and fused with current spatial queries, and a comfort auxiliary loss is added during a short fine-tuning phase. The authors report a reactive closed-loop R-Score of 81.25 on the NuPlan Val14 benchmark, which they claim is the first purely learning-based planner to surpass the expert log-replay baseline of 81.24. They also report experiments on CommonRoad with a PGP backbone, and ablations on historical interval, fusion method, and training epochs.","tokens_in":15453,"tokens_out":2972,"duration_ms":28871,"significance":"If the headline result is reproducible, the paper makes a useful empirical contribution: a lightweight plug-in module that improves an existing strong planner without modifying its backbone, requires only a few fine-tuning epochs, and improves closed-loop progress. The cross-backbone experiment (PGP on CommonRoad) strengthens the claim that the historical-intention mechanism generalizes. However, the central milestone claim—that LHPF is the first learning-based planner to exceed expert performance—rests on a 0.01-point gap from a single evaluation, with no variance, multiple seeds, or significance testing. The contribution is therefore plausible but not yet established at the level the abstract and introduction assert. The paper also contains an internal numerical inconsistency in Section 5.1 that directly affects confidence in the reported scores. I would support publication after the authors address the statistical support for the headline claim and correct the reporting inconsistencies.","major_comments":[{"comment":"The headline claim that LHPF is the first purely learning-based planner to surpass the expert rests on a 0.01-point margin in R-Score (81.25 vs. 81.24). The expert baseline is a fixed log-replay score, but LHPF is a trained neural planner evaluated once in a stochastic simulator; no seeds, repeated runs, error bars, or significance tests are reported. A single training run or evaluation can plausibly shift the score by more than 0.01, which would erase the claimed milestone. Please report the variance over multiple training seeds and evaluation runs, or explicitly soften the claim to 'matched or marginally exceeded in this evaluation' until statistical support is available.","section":"Section 5.1, Table 1, Abstract"},{"comment":"The text states that 'Compared to the baseline method, Pluto, the integration of historical planning information results in an improvement of the Progress metric from 78.59 to 79.21,' but Table 1 reports PLUTO Progress as 75.08 and LHPF Progress as 78.59. Table 2 also lists the scratch baseline Progress as 75.08. The reported 78.59 value appears to be LHPF's own Progress, not PLUTO's, and the 79.21 value appears nowhere in the tables. This internal inconsistency in the exact numbers supporting the narrative must be corrected, and the corrected values should be checked against the evaluation logs.","section":"Section 5.1, paragraph after Table 1"},{"comment":"The ablation study states that 'we use 10 scenarios per type from the nuPlan val14 dataset' for the ablation, which is roughly 140 of the 1090 scenarios in Val14. Yet Table 2 lists the same R-Score values (80.53, 81.25, etc.) that appear to be full-benchmark scores, and the baseline row is 78.76, identical to the full-benchmark PLUTO score in Table 1. Please clarify whether Table 2 reports full Val14 results or subset results. If it is a subset, the choice of the best configuration (summation fusion plus comfort loss) should be validated on the full benchmark, and the subset results need error bars because random subset selection can change scores by more than the 0.01-point margin that is central to the paper.","section":"Section 5.4, Table 2"},{"comment":"The text says L-PGP 'substantially improves collision metrics' relative to PGP, but the numbers show FCT at 26 vs. 27 and SCT at 9 vs. 16, with JERK increasing from 41.89 to 45.67. The only large improvement is in SCT, and it is measured as a count over 54 scenarios with no measure of variability. Please report per-scenario statistics or another suitable error assessment, and phrase the claim to match the mixed result: improved side collisions at the cost of higher jerk and essentially unchanged front collisions.","section":"Table 3, CommonRoad results"}],"minor_comments":[{"comment":"The sentence 'Despite these advances, many methods rely on still suffer from the sim-to-real problem. specifically' is garbled and should be rewritten.","section":"Section 2, Related Work"},{"comment":"The summation notation '0X i=-TH' is unusual and the bounds are ambiguous; please rewrite with standard summation limits, e.g., sum over historical timesteps i = -TH, ..., 0.","section":"Section 3.5, Eq. for Qst"},{"comment":"The discussion of historical planning intervals and training epochs refers to performance peaks and drops, but Figure 5 appears to lack axis labels and error bars, and no numerical values are given. Adding a table of the plotted values would make the ablation reproducible.","section":"Figure 5 and Section 5.4"},{"comment":"The row label 'Scratch' is confusing because the text identifies the scratch model as PLUTO; renaming the row to 'PLUTO (backbone)' would align with Table 1.","section":"Table 2"},{"comment":"Table 6 lists both 'PLUTO [9] 92.06' and 'PLUTO (w/o post.) [9] 78.76' without explaining that the difference is post-processing; a footnote or column header would prevent reader confusion.","section":"Table 6"},{"comment":"References [26] and [27] both cite UrbanDriver and appear to be duplicate entries; please merge them and update the citation numbers.","section":"References"},{"comment":"There are typos in the supplementary material, e.g., 'aslo' in the inference description; a careful proofread is needed.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely acceptable for the robotics venue if the authors can provide the missing statistical support for the expert-surpassing claim or reframe the claim as a single-run observation. The 0.01-point margin is simply too fragile for a 'first to surpass expert' milestone, especially because the authors' own ablation is run on a subset. I do not see evidence of bad faith, but the internal Progress inconsistency in Section 5.1 needs a clear correction. If the authors cannot supply multi-seed results, I would recommend that the editor ask them to remove or strongly qualify the milestone claim rather than reject outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: the core mechanism is real and the main comparison holds up, but the expert-surpassing claim is not supported by the evidence as presented.\n\nWhat's new: LHPF feeds historical plan embeddings back into the spatial query of a query-based decoder, and adds a comfort auxiliary loss. The idea is borrowed from motion-prediction temporal ensembling (HPNet, DCMS), so the novelty is in the transfer to planning and the closed-loop demonstration, not in the invention of the concept. The paper earns its keep on the main experimental comparison: LHPF improves the PLUTO backbone by 2.49 R-Score points on NuPlan Val14 reactive (81.25 vs 78.76), a meaningful gain. The ablation isolates the fusion method and the comfort loss, and the paper is transparent about the trade-off: more assertive driving raises progress but slightly increases collisions. The CommonRoad transfer to PGP, though smaller and with higher jerk, shows the module is not tied to PLUTO.\n\nSoft spots, in order of severity. First, the headline milestone—\"first purely learning-based planner to surpass the expert\"—is a 0.01-point gap (81.25 vs 81.24) over a fixed log-replay baseline, from a single evaluation with no seeds, no variance, no significance test. That claim should not stand as stated. Second, Section 5.1's text says Progress improved 'from 78.59 to 79.21' while Table 1 shows PLUTO at 75.08 and LHPF at 78.59; the numbers are inconsistent and need cleanup. Third, the ablation is on 10 scenarios per type with no selection details, so those results are indicative, not definitive. The self-referential comparison to a coauthored PLUTO is fine because the checkpoint is public and the evaluation is standard.\n\nBottom line: this is a solid empirical paper with an incremental but useful idea. It deserves a serious referee. I'd recommend acceptance after major revision, with the expert-surpassing claim either backed by repeated runs/error bars or softened to 'comparable to the log-replay expert in this single run.'","headline":"A credible temporal-ensembling extension to imitation planners with a solid gain over its backbone, but the 'surpasses the expert' claim rides on a 0.01-point margin with no error bars.","tokens_in":16007,"tokens_out":3583,"would_cite":true,"duration_ms":31097,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A purely learning-based driving planner reports its first win over the expert baseline by remembering its own past plans.","keywords":["imitation learning","autonomous driving planning","closed-loop planning","historical intention aggregation","NuPlan benchmark","spatio-temporal query","comfort loss","trajectory planning"],"falsifier":"Re-run the NuPlan Val14 reactive closed-loop evaluation for LHPF multiple times with different random seeds and/or fine-tuning checkpoints using the official simulator and scoring code. If the distribution of LHPF scores overlaps or falls below the expert log-replay score of 81.24, in particular if 81.25 is not reproducible, the central claim of surpassing the expert fails. A cheaper check is to report per-scenario score variance across the 1,090 scenarios and see whether a 0.01 gap is within the standard error of the mean.","tokens_in":15023,"feed_emoji":"🚗","tokens_out":4682,"duration_ms":40986,"temperature":0.7,"pith_summary":"LHPF is an imitation-learning planner that tries to fix a specific failure: conventional planners treat each frame's plan in isolation, so once small errors accumulate the vehicle forgets its own earlier intention (for example, an overtake) and drifts off course. The paper's central claim is that feeding the planner its own historical planning embeddings, pooled over past frames and fused with current spatial queries, restores temporal continuity and, together with a comfort auxiliary loss, improves closed-loop performance. On the NuPlan Val14 reactive benchmark the method scores 81.25, above the expert log-replay baseline of 81.24, which the authors report as the first time a purely learning-based planner has surpassed the expert. If this holds, imitation planners can use their own past decisions as a memory mechanism rather than re-planning from scratch at every step, and the module can be fine-tuned onto an existing backbone in about five epochs.","feed_headline":"First learning-based planner edges expert on driving test","feed_subtitle":"LHPF reaches 81.25 by pooling its own past plans, beating expert log-replay at 81.24.","key_machinery":"The load-bearing component is the historical intention aggregation module. At each timestep the query-based decoder's latent output is stored as a planning embedding; over the past observation window these embeddings are pooled and fused with the current spatial queries (lateral queries from the reference-line polyline encoder and longitudinal queries from learnable embeddings) to form spatio-temporal queries. A spatio-temporal decoder with lateral self-attention, longitudinal self-attention, and query-to-scene cross-attention then produces the final trajectory and score. The comfort auxiliary loss, computed from max-exceedance penalties over longitudinal and lateral acceleration, yaw rate, yaw acceleration, and jerk, is what lets the method recover comfort lost by the more aggressive driving style that historical intentions induce.","core_discovery":"On the paper's own terms, the discovery is that a planner's previously generated plans are a usable and useful signal for its current decision. LHPF collects the latent planning embeddings from earlier timesteps into a historical intention pool, combines them with the current spatial reference-line queries to form spatio-temporal queries, and decodes the final trajectory from those queries with a query-to-scene transformer decoder. A comfort loss that penalizes trajectories exceeding human-like acceleration, jerk, and yaw limits is added during fine-tuning. The result on the NuPlan Val14 reactive closed-loop benchmark is a total score of 81.25, the highest among purely learning-based planners and, against the expert log-replay baseline of 81.24, a reported first for a purely learning-based method beating the expert. The same historical-intention module, attached to a different backbone and evaluated on CommonRoad, also improves completion distance and reduces side collisions.","pith_inferences":["Beyond the paper: the 0.01-point margin over expert log-replay is within what a single run of a stochastic planner could produce by chance; without repeated seeds or error bars, the 'first to beat the expert' headline should be read as provisional until replicated.","Beyond the paper: historical planning embeddings may act as a temporal regularizer or low-pass filter on intentions; if so, similar gains could come from simpler mechanisms such as exponential moving averages over plan embeddings, which is a direct testable extension.","Beyond the paper: the fact that denser history (interval 1) hurts suggests the module is not simply 'more memory is better'; finding the right temporal resolution may matter more than the pooling operator, an effect the paper reports but does not explain.","Beyond the paper: applying the same historical-intention module to end-to-end planners that consume raw sensor data, where distribution shift is larger, would test whether the memory mechanism's benefit grows with the degree of covariate shift."],"forward_implications":["If correct, imitation-learning planners no longer need to re-derive intent from scratch every frame; keeping a memory of one's own plan is a cheap way to resist covariate shift and error accumulation.","The module is a plug-in: starting from a pretrained PLUTO checkpoint, five epochs of fine-tuning on the spatio-temporal decoder yields the reported gain of roughly 2.5 reactive-score points over PLUTO without post-processing.","On CommonRoad with a PGP backbone, adding the historical-intention module improves average completion distance from 25.18 to 26.46 and cuts side collisions from 16 to 9, suggesting the mechanism transfers across planners and environments.","Because the improvement comes mainly through progress and assertiveness rather than safety metrics, the method changes the trade-off frontier: more intent-consistent driving at the cost of slightly more collisions (95.82 versus PLUTO's 97.43).","The best fusion is summation rather than attention, and a 10-frame historical interval beats both the 20-frame setting (no history) and the 1-frame setting (dense, noisy history)."],"supporting_citations":[{"why":"Supplies the PLUTO backbone, the pretrained checkpoint, and the main learning-based baseline that LHPF extends and compares against.","marker":"[9]"},{"why":"Provides the NuPlan dataset, the reactive simulator, and the closed-loop scoring metrics that define the experimental setting.","marker":"[3]"},{"why":"Introduces the Val14 benchmark and the PDM baselines, including the expert log-replay reference that LHPF claims to surpass.","marker":"[14]"},{"why":"Provides the state-dropout encoder and PlanTF baseline, contributing architectural components and a strong learning-based comparison.","marker":"[8]"},{"why":"Supplies the IDM rule-based model used in reactive simulation and as a rule-based baseline in the results tables.","marker":"[30]"},{"why":"Supplies the PGP backbone used for the CommonRoad generalization experiments.","marker":"[15]"},{"why":"Provides the CommonRoad environment and scenarios for the closed-loop generalization study.","marker":"[1]"}],"fun_headline_variants":["LHPF beats expert driver by pooling its own past plans","LHPF scores 81.25, beating expert's 81.24","History-aware LHPF outperforms expert driver","Look back: LHPF's history pooling beats expert","LHPF: past plans improve driving decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline result rests on a single evaluation in which LHPF's reactive score of 81.25 exceeds the expert log-replay score of 81.24 by 0.01 points; the paper gives no error bars, multiple seeds, or significance test, so the claim that a purely learning-based planner has beaten the expert assumes that 0.01-point gap is not evaluation noise.","fun_headline_variants_meta":{"raw":{"variants":["LHPF beats expert driver by pooling its own past plans","LHPF scores 81.25, beating expert's 81.24","History-aware LHPF outperforms expert driver","Look back: LHPF's history pooling beats expert","LHPF: past plans improve driving decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3306,"prompt_tokens":921,"completion_tokens":2385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2316}},"tokens_in":537,"tokens_out":2385,"duration_ms":15814,"temperature":1.0,"reasoning_tokens":2316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:22:12.417073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the NuPlan Val14 reactive closed-loop evaluation for LHPF multiple times with different random seeds and/or fine-tuning checkpoints using the official simulator and scoring code. If the distribution of LHPF scores overlaps or falls below the expert log-replay score of 81.24, in particular if 81.25 is not reproducible, the central claim of surpassing the expert fails. A cheaper check is to report per-scenario score variance across the 1,090 scenarios and see whether a 0.01 gap is within the standard error of the mean.","supporting_citations":[{"cited_title":"Parting with misconceptions about learning- based vehicle motion planning","cited_arxiv_id":null,"evidence_quote":"Introduces the Val14 benchmark and the PDM baselines, including the expert log-replay reference that LHPF claims to surpass."},{"cited_title":"Rethinking imitation-based planner for autonomous driving, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the state-dropout encoder and PlanTF baseline, contributing architectural components and a strong learning-based comparison."},{"cited_title":"Con- gested traffic states in empirical observations and micro- scopic simulations","cited_arxiv_id":null,"evidence_quote":"Supplies the IDM rule-based model used in reactive simulation and as a rule-based baseline in the results tables."},{"cited_title":"Multimodal trajectory prediction conditioned on lane-graph traversals","cited_arxiv_id":null,"evidence_quote":"Supplies the PGP backbone used for the CommonRoad generalization experiments."},{"cited_title":"Commonroad: Composable benchmarks for motion plan- ning on roads","cited_arxiv_id":null,"evidence_quote":"Provides the CommonRoad environment and scenarios for the closed-loop generalization study."}],"review_version":1}