{"id":"05c44bde-0d8b-42db-96d1-a58f0afa09cf","arxiv_id":"2412.02136","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Linear graph-kernel models beat GNNs for learned planning heuristics, and ranking objectives beat cost-to-go regression.","lead":"This paper reviews graph-based machine learning for automated planning and argues that simple classical models beat deep neural networks for learning planning heuristics. It also proposes five open challenges that the field should tackle next.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scalability claim is not yet supported: WL features are trained on small tasks, but no size-stratified coverage shows they remain informative on the largest test tasks.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the small-task-to-large-task transfer premise. My stress-test sharpens it into a concrete, testable gap: the paper reports only aggregate coverage, so it cannot distinguish genuine object-count generalization from solving mostly near-training-size tasks. This is the single most load-bearing concern because the central claim is empirical and its value depends on scaling, not because the recipe is theoretically unsound. The paper's own Section 8.II reinforces the concern by noting that out-of-distribution generalization is not covered by standard bounds, while no alternative evidence for WL feature transfer is supplied. The reader's CONDITIONAL verdict remains appropriate: the manuscript is plausible and benchmarked on an external competition, but the key scaling claim needs a size-stratified evaluation before it can be accepted as established. I therefore recommend no change to the reader's verdict.","tokens_in":13707,"tokens_out":4577,"duration_ms":50397,"concrete_test":"Re-run or re-analyse the IPC23LT classical results from Section 7.1 and stratify coverage by the ratio of test-task object count to the domain's training-task object count (bins: <=1, 1–2, 2–5, >5). Report per-bin coverage for hGNN, hWL_cost, and hWL_rank under the same 30-minute/8GB limit. If hWL_rank's advantage over hGNN and over hWL_cost disappears or reverses in the >5 bin, the transfer premise fails and the recipe is not validated for the stated scalability goal; if the advantage persists in the largest bin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 7.3—that a linear model over WL color-refinement features trained with a ranking objective is the best graph-learning recipe for planning heuristics—depends on the premise that features learned from small tasks remain informative on tasks up to an order of magnitude larger. Section 2 defines L4P precisely as generalization across arbitrary object counts, and Section 7.1/Figure 7 shows IPC23LT test tasks are often more than 10 times larger than training tasks in object count. The only evidence offered is aggregate coverage in Figure 8. Aggregate coverage does not establish transfer: it could be dominated by the easiest, near-training-size tasks while failing on the large tasks that motivate the framework. The paper's own Section 8.II acknowledges L4P is inherently out-of-distribution and that standard generalization bounds do not apply, but it provides no analysis—theoretical or empirical—of how WL color refinement behaves under object-count shift for these domains. Without size-stratified results, the headline comparison (hWL_rank 520, hWL_ptfl 619) does not demonstrate that the recipe scales to the large tasks that are the stated purpose of L4P. This is a load-bearing evidential gap, not a disagreement with the graph-learning community.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys graph learning for planning and argues for three main claims: (1) a taxonomy and expressive-power hierarchy of graph representations of planning tasks, (2) classical machine learning with Weisfeiler-Lehman graph features substantially outperforms GNNs for planning heuristics, and (3) formulating heuristic learning as a ranking problem outperforms cost-to-go regression. These claims are supported by experiments on the IPC23LT benchmarks, summarized in Figure 8, where a WL-based heuristic with parallel portfolio reaches coverage 619 versus LAMA's 557 in classical planning. The paper also introduces the GOOSE framework, reports training-time and parameter-count advantages of WL models over GNNs, and ends with five open challenges for the Learning for Planning field.","tokens_in":13947,"tokens_out":3667,"duration_ms":38954,"significance":"If the central claims hold, the paper makes a useful and somewhat contrarian contribution: cheap linear models over WL kernels can be more data-efficient, faster, and better than GNNs for planning heuristics, and the ranking objective is better aligned with GBFS than cost-to-go regression. The paper credits the ranking idea to prior independent work [GKL16, CEKP23] and evaluates on an external benchmark (IPC23LT), which are strengths. The open-challenges discussion, especially on expressivity and out-of-distribution generalization, is thoughtful. However, much of the material summarizes the authors' own prior papers, and the new synthesis does not yet provide the derivations, experimental details, or stratified analysis needed to verify the strongest comparative and transfer claims.","major_comments":[{"comment":"The load-bearing claim that the WL-based recipe transfers from small training tasks to the much larger test tasks of IPC23LT is not established by the aggregate coverage numbers in Fig. 8. Figure 7 shows that test tasks are often up to an order of magnitude larger than training tasks in object count (e.g., Blocksworld 488 vs 21, Floortile 1461 vs 35), and §8.II itself states that L4P is inherently out-of-distribution. Aggregate coverage over 900 problems could be dominated by near-training-size tasks while failing on the large tasks that motivate the framework. The authors should report coverage broken down by task-size buckets (per domain or globally) for hWL_rank, hWL_ptfl, and the GNN baselines, or otherwise show that WL features remain informative on the largest tasks.","section":"§7.1, §7.3, Fig. 8"},{"comment":"The expressive-power hierarchy in Fig. 4 is presented as a set of results, but the paper gives no formal definitions, theorem statements, or proofs, nor does it point to specific theorems in the cited papers. In particular, 'more expressive', 'incomparable', and 'weaker notion of expressivity' are never defined in the text, and the distinction between incomparability with and without the h* difference requirement is unexplained. Since this hierarchy is one of the three contributions announced in the introduction, the authors should state the exact notion of distinguishability (e.g., which MPNN variant, which readout, which input features) and either provide the derivations or cite precise theorem numbers in [CTT24a, CTT24b].","section":"§4.2, Fig. 4"},{"comment":"The comparison between WL models and GNNs is missing the experimental configuration needed to assess fairness and reproducibility. The paper reports no GNN architecture details (number of layers, hidden dimensions, normalization, pooling), no training hyperparameters (learning rate, optimizer, number of epochs), no number of seeds, and no variance or confidence intervals. The claim 'Classical ML consistently outperform deep learning' is a central takeaway (§7.3), and it depends on the GNN baselines being reasonably configured. The authors should provide these details and, if possible, report the distribution of coverage across seeds.","section":"§5, §7.1, §7.2, Fig. 5, Fig. 8"},{"comment":"The advantage of ranking over cost-to-go is motivated by examples (Fig. 6) and credited to [GKL16], but the paper does not provide a formal statement of when ranking enlarges the hypothesis space or why zero cost-to-go loss is insufficient. Moreover, the empirical evidence in classical planning is a difference of 18 problems (hWL_rank 520 vs hWL_cost 502), which may be within noise; no confidence intervals or per-domain breakdowns are given. The authors should either provide a formal argument for the ranking advantage or present per-domain results and significance measures.","section":"§6.3, §7.3, Fig. 8"}],"minor_comments":[{"comment":"The notation [ [n] ] appears with a formatting artifact; if it denotes {1, ..., n}, use [n] or define it cleanly.","section":"§3.1"},{"comment":"The caption mentions 'incomparable with no h* difference requirement', but this notion is never defined in the text; please define it.","section":"§4.2, Fig. 4"},{"comment":"The abbreviation hWL_grid is listed but not explained; add a sentence describing the grid-search configuration and the range of hyperparameters searched.","section":"§7.1"},{"comment":"The sentence 'to the use of various different graphs and graph algorithms' contains a repeated 'to the use' and should be edited for readability.","section":"§4.1"},{"comment":"For several domains the difference between training and test sizes is hard to read on the log scale; consider adding numerical annotations or a table.","section":"Fig. 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is largely a synthesis of the authors' own prior papers (CTT24a, CTT24b, HTT+24, CT24a), which is acceptable for a survey-style contribution but limits the novelty of the present manuscript. The main experimental numbers are already published in those earlier works. If the journal expects substantial new results, the editor may wish to weigh the survey character of the paper against that expectation. The reported code availability is a positive point, but the missing experimental details and unverified hierarchy currently prevent a stronger recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a review-style preprint that re-presents the authors' own prior work, so don't read it expecting a new result. The real value is the taxonomy of graph representations for planning tasks, the claimed expressivity hierarchy (presented as ongoing work, no proofs here), and a sharp empirical thesis: classical ML with Weisfeiler-Lehman features beats GNNs for planning heuristics, and ranking losses beat cost-to-go losses. Those claims are directly backed by coverage figures on IPC23LT, an external benchmark, and the paper is refreshingly honest about the open challenges, including the out-of-distribution nature of L4P.\n\nWhat it does well: the comparison is meaningful, the baselines include strong non-learned planners like LAMA, and the authors ship code. The ranking idea is properly credited to earlier work, so the citation pattern is not problematic.\n\nSoft spots: the expressivity hierarchy is described without derivations; no hyperparameters, seeds, or variance for the GNN baselines; and per-domain grid selection is used for the best WL configuration. More importantly, the stress-test concern lands: the coverage numbers are aggregate only. The paper motivates L4P as scaling from small training tasks to tasks up to an order of magnitude larger, but never shows coverage broken down by task size. Aggregate coverage can hide failure on the large tasks. That is a real evidential gap for the central claim, and the paper's own Section 8.II admits generalisation theory doesn't apply, so it needs empirical stratification instead.\n\nThat said, the overall direction is plausible and the comparative claims are not fabricated. It's a synthesis, so it shouldn't be judged as a primary research paper, but as a survey/position piece with empirical support it deserves a serious referee. The referee should ask for size-stratified results, full hyperparameters, and ideally a table of per-domain coverage.\n\nFor my own work, I'd cite it for the taxonomy and the empirical comparison, though I'd wait for the version with stronger transfer evidence.\n\nRecommendation: send it to peer review, but expect a major revision. It's worth engaging with.\n\nRegards.","headline":"A useful, honest synthesis of the authors' own recent results, but the headline scalability claim rests on aggregate coverage rather than size-stratified evidence.","tokens_in":14505,"tokens_out":2200,"would_cite":true,"duration_ms":22434,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that for planning, the best graph-learning recipe is a linear model on Weisfeiler-Lehman features trained to rank states instead of estimating costs to go.","keywords":["graph learning","automated planning","Weisfeiler-Lehman graph kernel","learned heuristics","ranking objective","generalisation across object counts","GOOSE","IPC learning track"],"falsifier":"Run the GOOSE pipeline on a domain where the skill needed to solve a task changes with object count—for example, a domain where small tasks are solved by a local rule but large tasks require global connectivity or counting. If the WL linear ranking heuristic's coverage falls below the GNN or cost-to-go baseline on such a domain under the same training-testing protocol, the claim that this recipe is consistently best is falsified.","tokens_in":13499,"feed_emoji":"🧩","tokens_out":7059,"duration_ms":70331,"temperature":0.7,"pith_summary":"The paper sets out a position on graph learning for planning: after surveying graph representations, architectures, and training objectives, it argues the winning recipe is cheap, classical machine learning on Weisfeiler-Lehman graph features, trained with a ranking objective, rather than deep graph neural networks trained to regress optimal costs. It grounds this in a theory hierarchy showing which planning-task graph representations are more expressive under message-passing, and in experiments on the 2023 planning-competition learning track where the resulting heuristics guide greedy search to solve up to 619 of 900 problems, ahead of a strong baseline planner's 557. The practical stake is that learned planning heuristics can be built from small training tasks and scale to much larger test tasks, at a fraction of the training cost of deep models.","feed_headline":"Simple graph features beat deep learning for planning","feed_subtitle":"A ranking objective and Weisfeiler-Lehman features let small training tasks guide much larger planning problems.","key_machinery":"The load-bearing mechanism is the Weisfeiler-Lehman (WL) colour-refinement algorithm, used as a graph kernel to turn a planning task's relational structure into a feature vector for a linear model; the WL algorithm is also the known expressivity ceiling for message-passing GNNs, which is why the paper views it as a sufficient feature extractor. The second mechanism is ranking as a constrained optimisation problem: rather than fitting costs to go, the learner fits a function that respects per-state inequalities derived from optimal plan traces, comparing a state against its parent and siblings. These two mechanisms combine in GOOSE, which learns domain knowledge from small training instances and applies it to test instances with many more objects.","core_discovery":"The paper's central claim is that the choice of training objective matters more than model size for learned planning heuristics: formulating heuristic learning as ranking states, with a constrained-optimisation objective that only preserves the inequalities needed by greedy best-first search, yields better guidance than regressing h* values. On the model side, the paper claims that linear models over Weisfeiler-Lehman colour-refinement features match or beat message-passing neural networks across classical and numeric planning domains while training orders of magnitude faster, and that grounded graph representations are strictly more expressive than lifted ones in the relevant sense. Assembled into the GOOSE framework—learn on small tasks, plan on large tasks—these choices make learned heuristics competitive with strong hand-built planners on the 2023 IPC learning track.","pith_inferences":["The same WL-plus-ranking recipe may transfer to other search problems where cost-to-go labels are expensive but pairwise ordering information is cheap, such as theorem proving, program synthesis, or puzzle solving; the paper does not test this.","If grounded representations' extra expressivity is the reason for their advantage, then architectures that approximate higher-order WL refinement could close the remaining gap to strong planners without moving to full deep learning; this is an extrapolation from the paper's expressivity hierarchy.","The paper's setup assumes task-size scaling within a fixed domain; a natural challenge it does not address is cross-domain transfer, where a model is trained in one domain and evaluated in another, or where object-count scaling is not the only distribution shift.","Because GOOSE's training is so cheap, one could in principle use active learning or iterative re-training during search, updating the ranking function on the fly; the paper's data-collection discussion implies this but does not implement it."],"forward_implications":["New learned-heuristic systems for classical and numeric planning should be compared against WL-kernel linear models trained with ranking, not only against GNN baselines, because the paper shows the cheap model wins on coverage.","Since WL features with a linear model train orders of magnitude faster, the bottleneck in learning for planning shifts to label generation and data collection rather than model fitting.","Because grounded graph representations are more expressive than lifted ones, representation choice should be a first-class design decision in learning-for-planning systems, not an implementation detail.","Ranking objectives that preserve only comparison inequalities are a drop-in replacement for mean-squared-error cost-to-go in greedy best-first search; adopting them should improve coverage without changing the search algorithm.","On the IPC23LT classical benchmarks, a parallel portfolio of WL heuristics (619) exceeds LAMA (557), so learned heuristics with simple search are now a serious alternative to engineered satisficing planners."],"supporting_citations":[{"why":"Supplies the grounded and lifted graph representations with instantiation relations used in the GOOSE framework, and the argument that exact h* regression is unnecessary.","marker":"[CTT24a]"},{"why":"Establishes that linear models over WL features outperform GNN heuristics for classical planning on runtime, parameters, and coverage.","marker":"[CTT24b]"},{"why":"Introduces the LP ranking formulation and extends the WL-versus-GNN comparison to numeric planning.","marker":"[CT24a]"},{"why":"Extends ranking-based heuristic learning to neural architectures and shows pairwise ranking data can be reduced to a linear number of constraints.","marker":"[HTT+24]"},{"why":"Original proposal that heuristic functions be learned as ranking functions rather than cost-to-go estimates.","marker":"[GKL16]"},{"why":"Establishes that Weisfeiler-Lehman colour refinement bounds the expressivity of message-passing GNNs, motivating WL feature extraction.","marker":"[MRF+19]"},{"why":"Defines the Weisfeiler-Lehman graph kernel used to construct features for the classical linear models.","marker":"[SSVL+11]"},{"why":"Defines the IPC23 learning-track benchmarks and evaluation protocol whose coverage numbers are the paper's main empirical evidence.","marker":"[TAE+24]"},{"why":"LAMA is the strong satisficing planner baseline against which learned heuristics are compared on classical planning.","marker":"[RW10]"}],"fun_headline_variants":["Planning heuristics: ranking beats regression, simple beats deep","GOOSE: small tasks train heuristics for large planning problems","Weisfeiler-Lehman features beat message-passing for planning","Training objective matters more than model size in planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scaling story rests on the assumption that a domain's structure relevant to planning is stable as task size grows, so features learned from small problems remain the right features on much larger problems.","fun_headline_variants_meta":{"raw":{"variants":["Planning heuristics: ranking beats regression, simple beats deep","GOOSE: small tasks train heuristics for large planning problems","Weisfeiler-Lehman features beat message-passing for planning","Training objective matters more than model size in planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2917,"prompt_tokens":814,"completion_tokens":2103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":2035}},"tokens_in":430,"tokens_out":2103,"duration_ms":14382,"temperature":1.0,"reasoning_tokens":2035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:47:40.357137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the GOOSE pipeline on a domain where the skill needed to solve a task changes with object count—for example, a domain where small tasks are solved by a local rule but large tasks require global connectivity or counting. If the WL linear ranking heuristic's coverage falls below the GNN or cost-to-go baseline on such a domain under the same training-testing protocol, the claim that this recipe is consistently best is falsified.","supporting_citations":[],"review_version":1}