{"id":"bef7b83d-3de2-4c12-8e9c-6f63256c8b79","arxiv_id":"2412.09666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Across three planning domains, LLMs perform better as comparative rankers of intermediate plans than as direct solvers or verifiers, and one-shot heuristic guidance improves ranking.","lead":"This paper tests whether large language models are better at ranking partial plans than at generating or verifying complete plans. Across three planning tasks, the authors find LLMs rank intermediate solutions more reliably, and they introduce a new fitness-planning benchmark for learning user preferences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The heuristic oracle in §4.2 measures edit distance to a single gold plan, not distance to the set of valid goal states; a complete feasible plan can rank below an incomplete one, so the hit@k scores do not establish heuristic value for search.","rationale":"The reader correctly flagged that synthetic ranking performance may not transfer to real search; I go further and argue that the synthetic target itself is misspecified. In Section 4.2, the ground-truth ranking is defined as distance to a single gold plan Pgold, but a planning problem's goal set contains all feasible complete assignments. The edit-distance-to-gold oracle gives positive value to a perfect solution and can prefer an incomplete plan that resembles Pgold. Thus the hit@k numbers measure a reference-matching skill, not heuristic guidance. This is load-bearing because the paper's main claim is specifically about heuristic value for tree search. The fix is straightforward: re-evaluate with an oracle that measures distance to the goal set (or run an actual tree search). Since the issue concerns the evaluation target rather than LLM capabilities per se, the framework remains useful and the appropriate verdict is conditional, matching the reader's assessment.","tokens_in":14869,"tokens_out":5494,"duration_ms":53633,"concrete_test":"For each easy-instance course-planning problem, enumerate all feasible complete room assignments (or use a solver to compute the true distance-to-goal-set). Create pairs (ŷ1, ŷ2) where ŷ1 is a complete feasible plan not equal to Pgold and ŷ2 is an incomplete prefix matching Pgold, so that dist(ŷ2, Pgold) < dist(ŷ1, Pgold). The paper's oracle ranks ŷ2 above ŷ1. Re-score the LLM rankings from Table 5 against a corrected oracle that assigns value 0 to every complete feasible plan and otherwise counts remaining unassigned sections plus constraint violations. If hit@1 under the corrected oracle is near chance or substantially lower, the reported heuristic advantage is an artifact of the gold-plan distance target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.3, the oracle comparison function g_T is defined via an oracle heuristic f_T(ŷ, y, x) where y is a single ground-truth solution. In the course-planning heuristic evaluation (§4.2), the ground truth for ranking is explicitly 'a distance dist(ŷ, Pgold) from current plan to the correct plan,' with Pgold being one gold plan. This is not a valid planning heuristic. A heuristic should estimate distance or cost to the nearest goal state, meaning any plan satisfying all constraints and the occupancy threshold. Course planning has many valid goal states, e.g., different room assignments are all acceptable. A complete, feasible, optimal plan that differs from Pgold in room assignments has nonzero edit distance to Pgold, while an incomplete prefix that happens to match Pgold can have smaller edit distance. The oracle therefore ranks a non-solution above a solution. Consequently, the LLM's task as evaluated—predict which candidate is closer to a hidden reference plan—is a reference-inference task, not the task of guiding tree search. The reported hit@k numbers (Table 5) measure performance on that artificial target. The central conclusion that 'LLMs are much better at providing feedback signals ... in the form of comparative heuristic functions' thus rests on a target function that would mislead a real search algorithm. No end-to-end tree search is run in the paper, so there is no direct evidence that ranking by this oracle improves planning success.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes decomposing LLM planning contributions into three independent roles—solver, verifier, and comparative heuristic—and evaluates them on TravelPlanner, a new course-scheduling benchmark, and a new interactive fitness-planning benchmark. The experiments, run with GPT-4o, Claude-3.5-Sonnet, and DeepSeek-V2.5, show low end-to-end solve rates, moderate verifier pass rates, and relatively high hit@k scores on pairwise ranking of candidate plans. From this, the paper concludes that LLMs are more effective as comparative heuristic functions than as generators or verifiers, and it presents the fitness benchmark as a testbed for learning user preferences on the fly.","tokens_in":15178,"tokens_out":7134,"duration_ms":68623,"significance":"If the conclusion were established, it would provide useful guidance for LLM-based tree search: use LLMs to rank intermediate candidates rather than to emit final plans. The paper's decomposition is clean and the comparative formulation in Eq. (1) is well motivated; the new course and fitness datasets fill an obvious gap, and the multi-model comparison is a strength. However, the current experiments do not validate the oracle heuristics as search guidance, and the headline comparison treats incomparable metrics as if they were on one scale. The central claim is plausible but not yet demonstrated.","major_comments":[{"comment":"Course planning has many valid goal states, but the ground-truth comparison function is defined through dist(P_i, P_gold) to a single gold plan. A complete feasible and even optimal plan that assigns a section to a different valid classroom has nonzero distance to P_gold, while an incomplete prefix that happens to match P_gold can have smaller distance. The oracle in Eq. (1) therefore ranks a non-solution above a solution, so the hit@k scores in Table 5 measure how well the LLM infers a hidden reference plan rather than how useful the ranking would be for guiding search. Since no end-to-end tree search is run, the central conclusion that LLMs are much better as comparative heuristics is not supported by the course-planning experiment.","section":"Section 4.2 and Eq. (1); Table 5"},{"comment":"Solver, verifier, and heuristic scores are reported on different scales and over different units: final pass rate for plans, pass or accuracy rate for verification, and hit@k for rankings. A hit@1 of 0.59 and a solver pass rate of 0.07 are not comparable quantities, and no confidence intervals or multiple-seed runs are given, so the abstract's comparative claim that LLMs are much better as heuristics than as solvers is not statistically grounded.","section":"Section 5, Tables 2-5"},{"comment":"The fitness-planning oracle is the author-defined weighted utility with free parameters alpha and beta plus a hidden preference vector U. This is a constructed target: no evidence is offered that it approximates real user satisfaction, and the ranking experiments only show agreement with this synthetic utility. Consequently, the novel benchmark's claim about learning user preferences on the fly is not yet validated for real users, and ranking accuracy on this target does not by itself establish heuristic value in interactive optimization.","section":"Section 4.1, Eq. (3), Table 4"},{"comment":"The comparative heuristic is motivated by tree search, but all heuristic experiments are static ranking tasks over candidate sets with no search algorithm. The chain from ranking synthetic corruptions to search guidance is incomplete; a direct evaluation, for example best-first search with the LLM heuristic versus a baseline, or at least a correlation analysis between ranking accuracy and search progress, is required to support the central conclusion.","section":"Section 3.3 and Section 5"}],"minor_comments":[{"comment":"The text contains an unresolved reference to a table number; please replace it with the actual table identifier.","section":"Section 5.3"},{"comment":"The verifier table mixes different metrics across datasets, reporting feasibility and optimality for course planning but only pass rate for fitness and travel planning, which makes cross-domain conclusions difficult to draw.","section":"Table 3"},{"comment":"The optimality objective J(P) is written as a sum with ambiguous index associations, and the delta = 1.3 threshold is not explained; please rewrite the formula with explicit ranges and justify the threshold.","section":"Section 4.2"},{"comment":"The in-text references to Table 6 are confusing because the main-text solver table is numbered Table 2 and the detailed results appear in the appendix; please renumber or make the references unambiguous.","section":"Sections 5.1 and 5.2"},{"comment":"The paper alternates between few-shot and one-shot for the heuristic settings; please unify the terminology and state clearly how many exemplars are used in each experiment.","section":"Sections 4.1 and 4.2"}],"recommendation":"major_revision","confidential_remarks":"The course-planning oracle issue is the main technical blocker; I would expect the authors either to redefine the ground truth for ranking using a set of valid solutions or to add an end-to-end tree-search evaluation before acceptance. The benchmark novelty is moderate, and the paper needs a careful pass on table references and metric reporting. I do not see any ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central claim—that LLMs are much better as comparative heuristics than as solvers or verifiers—is plausible but not actually supported by the evidence as presented. The course-planning oracle in §4.2 defines distance to a single gold plan, not to the set of valid goal states. Since course planning has many acceptable assignments, a complete feasible plan that differs from the gold plan in room choices can rank below an incomplete prefix that happens to match it. That makes the reported hit@k scores a measure of reference inference, not of guidance for search. I agree with the stress-test note: the target function would mislead a real search algorithm, and no end-to-end search is run anywhere in the paper.\n\nThat said, the paper does something genuinely useful. The three-role decomposition (solver, verifier, comparative heuristic) is a clean framing, and the comparative formulation—asking the LLM to rank candidates rather than produce real-valued heuristic estimates—is a sensible way to sidestep the calibration problems of numeric prediction. The fitness-preference benchmark is also a real addition, with a reasonable hidden-utility construction and iterative feedback setup. On Travel Planning, the micro-pass-rate oracle is closer to a legitimate heuristic, and the fitness oracle, though author-defined, at least measures satisfaction rather than distance to one reference.\n\nThe soft spots are real but concentrated. No error bars or multiple seeds, so the cross-model differences (e.g., Claude vs. GPT-4o in Table 5) could be noise. Solver, verifier, and heuristic scores are on different metrics and tasks, so the \"much better\" comparison in the abstract is not a direct comparison. There are also unresolved table references (“Table ??” in §5.3) and the heuristic table in the appendix seems to be the actual one cited in the main text. All of these are fixable, but together they mean the paper is a framework-plus-data-dump rather than a demonstration.\n\nWho should read it: researchers working on LLM-based tree search or on using LLMs as evaluators/rankers. They will find the comparative heuristic idea worth testing in their own systems. It deserves a serious referee, but the referee should push hard on the oracle definition for course planning and ask for either an end-to-end search experiment or a corrected oracle that measures distance to the goal set. I would not cite the headline result as it stands, but I would cite the framework if the evaluation issues are repaired in a revised version.","headline":"Useful three-role framework and a plausible comparative-heuristic idea, but the course-planning oracle is not a valid planning heuristic, so the headline claim about LLM heuristic value is not established.","tokens_in":15709,"tokens_out":1605,"would_cite":false,"duration_ms":17437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that across three planning domains, LLMs are more reliable when used to rank candidate partial plans than when used to generate or verify complete plans, and it provides an evaluation framework supporting this asymmetry.","keywords":["LLM planning","comparative heuristic","tree search","plan verification","travel planning","course scheduling","fitness planning","test-time scaling"],"falsifier":"Implement the same LLM as the ranking heuristic inside an actual tree-search planner on these three benchmarks and compare the resulting plan quality against the same search guided by the dataset's oracle score; the central claim fails if high ranking accuracy on the paper's candidate pairs does not produce better search outcomes than random ordering or a simpler baseline.","tokens_in":14646,"feed_emoji":"🧭","tokens_out":8810,"duration_ms":76013,"temperature":0.7,"pith_summary":"This paper argues that when large language models are embedded in planning systems, their most reliable contribution is neither generating complete plans nor verifying them, but ranking partial candidate solutions against one another. Across three planning tasks—travel itineraries, course scheduling, and personalized fitness planning—the authors measure the same models as direct solvers, as verifiers, and as comparative heuristic functions. Solver pass rates are low, verification is unreliable on complex constraints, while ranking of candidate plans reaches substantially higher hit rates, especially after one worked example of how to compare. The authors conclude that LLM-based tree-search algorithms should use LLMs primarily to provide comparative feedback on intermediate solutions rather than to produce final plans. They also introduce an interactive fitness-planning benchmark in which the agent must learn a user's hidden preferences from iterative feedback.","feed_headline":"LLMs rank plans better than they generate or verify them","feed_subtitle":"Across travel, course, and fitness planning, ranking beats solving and verifying in the paper's tests.","key_machinery":"The load-bearing object is the comparative heuristic function $\\hat{g}_T(\\hat{y}_1, \\hat{y}_2, x)$, an LLM-parameterized approximation of the ground-truth pair-ordering $g_T(\\hat{y}_1, \\hat{y}_2, y, x)$. Instead of asking the model to estimate an exact distance to an optimal plan, the framework asks only which of two partial candidates is closer, and that ordinal question is what makes the heuristic tractable. Around it, the paper builds an evaluation protocol: each dataset defines an oracle $f_T$ (constraint pass rate for travel, distance to a gold schedule for course planning, a hidden utility score for fitness), and LLM ranking accuracy is measured by hit@k and comparison accuracy against that oracle. The one-shot course-planning condition shows how a single example of the comparison function focuses the model's reasoning and lifts performance.","core_discovery":"The central discovery is an asymmetry across three roles an LLM can play in a planning system. Given a task, the paper defines an oracle heuristic $f_T(\\hat{y}, y, x)$ scoring how close a candidate solution is to the correct one, and a comparison function $g_T$ that picks the better of two candidates. The LLM's approximation $\\hat{g}_T(\\hat{y}_1, \\hat{y}_2, x)$ needs only to order the candidates, not to predict a numeric distance, and this ordinal formulation is what the models handle well. In course planning, for instance, direct solvers almost never produce feasible schedules, yet the same models rank four candidate plans with hit@1 around 0.41 zero-shot and 0.48 one-shot for Claude-3.5-Sonnet; in travel planning, GPT-4o reaches hit@1 0.5911 as a ranker while its solver pass rate is 0.072 direct and 0.083 with chain-of-thought. The paper reads this as evidence that LLM-based tree search should delegate guidance to a comparative heuristic and reserve generation for proposing candidates.","pith_inferences":["What is not yet shown is end-to-end gain: the paper measures ranking accuracy on constructed candidate sets, so the strongest reading is that an LLM ranker could guide search, not that it already does.","A natural extension is to replace the random corruption of gold plans with search-generated partial plans and test whether ranking accuracy predicts downstream plan quality in an actual tree search.","The comparative formulation may transfer to any task where exact scoring is hard but pairwise preference is easy, such as document triage or plan repair in robotics."],"forward_implications":["LLM-based tree-search planners should be built with the LLM as a ranker of intermediate candidates, with candidate generation handled separately, since generation and verification are the weak links.","One-shot demonstrations of the comparison heuristic are cheap and materially improve ranking, so prompting a search algorithm with a worked comparison example is a low-cost lever.","Self-verification by an LLM is not a reliable substitute for an oracle when constraints are commonsense or global, so planning systems should avoid depending on it.","The fitness-planning benchmark offers a controllable environment for studying online preference learning, where success is measured by how fast an agent converges on a hidden utility function."],"supporting_citations":[{"why":"Introduces tree-of-thought search, the class of tree-based methods that motivates using LLMs as heuristics.","marker":"Yao et al. [2023a]"},{"why":"Presents an MCTS planner with an LLM as world model, another tree-search approach the paper tests the heuristic assumption against.","marker":"Hao et al. [2023]"},{"why":"Shows repeated sampling and majority voting for solution selection, the verifier baseline the paper contrasts with comparative ranking.","marker":"Brown et al. [2024]"},{"why":"Argues test-time compute scaling is effective, the rationale for studying how LLMs can contribute inside search.","marker":"Snell et al. [2024]"},{"why":"Provides the TravelPlanner benchmark and its constraint-evaluation script, used for the travel planning experiments and the oracle heuristic.","marker":"Xie et al. [2024]"},{"why":"Earlier personalized fitness planning framework whose reliance on human feedback motivates the new interactive fitness benchmark.","marker":"Shin et al. [2023]"}],"fun_headline_variants":["LLMs shine as plan rankers, not generators or verifiers","Planning: LLMs better at ranking than solving or verifying","In planning, LLMs rank better than they solve or verify","LLMs rank plans better than generate or verify them","For planning, ranking beats solving and verifying for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion assumes that an LLM's accuracy at ranking deliberately altered or author-scored candidate plans carries over to ranking the messy intermediate states a real tree search would encounter.","fun_headline_variants_meta":{"raw":{"variants":["LLMs shine as plan rankers, not generators or verifiers","Planning: LLMs better at ranking than solving or verifying","In planning, LLMs rank better than they solve or verify","LLMs rank plans better than generate or verify them","For planning, ranking beats solving and verifying for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2732,"prompt_tokens":900,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1749}},"tokens_in":516,"tokens_out":1832,"duration_ms":11690,"temperature":1.0,"reasoning_tokens":1749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:57:04.815835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Implement the same LLM as the ranking heuristic inside an actual tree-search planner on these three benchmarks and compare the resulting plan quality against the same search guided by the dataset's oracle score; the central claim fails if high ranking accuracy on the paper's candidate pairs does not produce better search outcomes than random ordering or a simpler baseline.","supporting_citations":[],"review_version":1}