{"id":"7cc4ae0f-51fe-489e-9442-be6980cdce52","arxiv_id":"2505.04480","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-driven evolution automatically designs trajectory prediction heuristics that beat handcrafted baselines and generalize better than tested deep learning models to an unseen dataset.","lead":"This paper presents TrajEvo, a system that uses a large language model inside an evolutionary loop to write simple code rules that predict where pedestrians will walk. On a dataset the system never trained on, these evolved rules beat both handwritten heuristics and several deep learning models while running hundreds of times faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's cross-dataset advantage rests on unverifiable baseline training/evaluation protocol; no details on SDD preprocessing, K=20 sampling, or identical checkpoints, so the headline 'outperforms deep learning' is not yet supported.","rationale":"I read TrajEvo as an applied contribution: LLM-driven evolutionary search produces fast, interpretable, multi-sample trajectory heuristics that beat classical heuristics on ETH-UCY and, per Table 3, transfer better to SDD than three neural baselines. The strongest claim is the cross-dataset generalization result, and the load-bearing assumption is that the deep learning numbers in Table 3 were produced under the same training splits, SDD preprocessing, and K=20 evaluation protocol as TrajEvo. The paper does not document this, and the reader's weakest_assumption identifies exactly this gap. I agree with that assessment. A secondary concern is the absence of seeds and error bars for the stochastic LLM-based evolutionary process; the ablations and main results are single runs, so even the TrajEvo-vs-baseline gap has no variance estimate. Both concerns are addressable by rerunning with a shared harness and multiple seeds. I do not see an internal inconsistency in the method, and the released code plus the in-distribution ablations give the paper independent support; the limitations section appropriately concedes the in-distribution accuracy gap. Thus the correct outcome is conditional acceptance with added experimental detail, matching the reader's conditional verdict rather than changing it. I would not reject or mark unverdictable because the core mechanism is plausible and the requested checks are straightforward, but the headline claim as currently written should not be taken as settled until Table 3 is reproducible.","tokens_in":17648,"tokens_out":3358,"duration_ms":34439,"concrete_test":"Re-evaluate the three neural baselines on SDD using the released TrajEvo evaluation harness: take official or author-provided checkpoints trained on each ETH-UCY leave-one-out split, preprocess SDD identically to TrajEvo (same resampling to 8 observed and 12 predicted frames, same units, same scene selection), sample K=20 with matched random seeds and the same best-set selection rule, and recompute minADE20/minFDE20. If EigenTrajectory and MoFlow reproduce within roughly 0.5 px of Table 3, the claim stands; if the gap shrinks, reverses, or the baseline numbers move by several pixels, the headline generalization claim must be weakened or reported with explicit protocol details.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—TrajEvo outperforms deep learning when generalizing to unseen SDD—is carried entirely by Table 3. Yet §4.3 reports only minADE20/minFDE20 in pixels and gives no protocol for the three neural baselines (Trajectron++, EigenTrajectory, MoFlow): no statement that they were trained by the authors on the same ETH-UCY leave-one-out splits, no checkpoint source, no SDD preprocessing details (frame-rate resampling, observation/prediction horizons, units, coordinate conventions, scene filtering), and no K=20 sampling procedure. The neural rows vary per split, implying per-split training, but consistency with TrajEvo's evaluation pipeline is unverified. If these numbers are taken from published papers using different observation lengths, prediction horizons, or sample counts, the comparison is invalid. Additionally, TrajEvo evolution is stochastic—LLM temperature is 1, crossover and mutation are random, and no seeds or error bars are reported—so a single run per split gives no sense of variance; the roughly 1.9-pixel minADE gap over EigenTrajectory could fall within run-to-run variation. The paper's own Limitations section concedes that in-distribution accuracy trails state-of-the-art neural methods, so the 'bridge the gap' contribution hinges almost entirely on Table 3. This is an evidence and reproducibility gap, not a logical inconsistency, but it is load-bearing for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TrajEvo, a framework that uses an LLM (Gemini 2.0 Flash) as the generator and reflector inside an evolutionary loop to produce Python-coded trajectory prediction heuristics. The main algorithmic additions are Cross-Generation Elite Sampling (CGES), which samples mutation targets from a history archive, and a Statistics Feedback Loop, which feeds per-sample prediction statistics to the reflector. The heuristics are evaluated on the ETH-UCY datasets under a leave-one-out protocol and on the Stanford Drone Dataset (SDD) for cross-dataset generalization. The central claims are that TrajEvo outperforms existing heuristic baselines on ETH-UCY, outperforms deep learning baselines on the unseen SDD dataset, and does so with substantially lower training and inference cost while remaining interpretable.","tokens_in":17894,"tokens_out":6284,"duration_ms":61194,"significance":"If the reported results are reproducible, TrajEvo is a valuable demonstration that LLM-driven evolutionary search can discover non-neural predictors that are fast, interpretable, and competitive with or better than deep baselines under distribution shift. The paper ships executable heuristic code and a public repository, and the evaluation loop is concrete enough to be machine-checked. The cross-dataset SDD experiment is the most interesting and potentially important result. However, the headline claim currently rests on an under-specified baseline comparison and on single stochastic runs, so the significance is conditional on the authors providing a complete protocol and variance-aware results. The work is not circular: evolution optimizes on training splits and evaluation is on held-out splits.","major_comments":[{"comment":"The multi-sample metric is described in a non-standard and ambiguous way. The text says the method selects a single set index k* that minimizes an unspecified “overall error” and then uses the trajectories of that set for all agents, rather than the usual per-pedestrian best-of-K protocol used in the cited trajectory prediction literature. If the deep learning baseline numbers in Tables 2 and 3 were obtained with per-agent minADE20/minFDE20 while TrajEvo uses a global set index, all comparisons are not apples-to-apples. Please define “overall error,” state explicitly which protocol is used for every method in every table, and, if the two protocols differ, report both sets of values.","section":"§3.1"},{"comment":"The cross-dataset generalization claim—the central contribution stated in the abstract—is supported only by Table 3, but the table gives no experimental protocol for the deep learning baselines. There is no statement of whether Trajectron++, EigenTrajectory, and MoFlow were trained by the authors on the same ETH-UCY leave-one-out splits, no checkpoint provenance, no description of SDD preprocessing (frame-rate resampling, observation/prediction horizons, units, agent filtering, coordinate conventions), and no details on how K=20 sampling was performed for each baseline. Without these details, the “outperforms deep learning on unseen SDD” claim cannot be independently verified. Please provide a complete protocol and, ideally, run all baselines in the same code infrastructure.","section":"§4.3, Table 3"},{"comment":"The evolutionary search is stochastic: the LLM temperature is 1, CGES uses softmax sampling with temperature 1, and crossover and mutation are randomized, yet every reported result is a single run with no seeds or error bars. The margins in Tables 1–4, including the roughly 1.9-pixel minADE advantage over EigenTrajectory on SDD, could be within run-to-run variation. Please re-run each configuration with multiple seeds and report the mean, standard deviation (or full distribution), and the number of seeds used.","section":"§4.2–§4.4, Table 5"}],"minor_comments":[{"comment":"The inference-time comparison that supports the 300× speedup claim lacks measurement details for the neural baselines: no batch size, no number of CPU cores used for the multi-core CPU timings, no GPU warm-up protocol, and no statement of whether the same preprocessing and code paths were used. Please specify these details.","section":"§4.5, Appendix A.2"},{"comment":"The prompts inform the LLM that the task uses the ETH/UCY dataset, but the paper does not address the possibility that Gemini's pretraining has already seen SDD trajectories. Since SDD is described as “unseen,” the authors should discuss this leakage risk and, if possible, test it by comparing against an LLM that has no exposure to such data or by analyzing whether the evolved heuristics encode SDD-specific patterns.","section":"Appendix B.2, Prompt 12"},{"comment":"The right panel of Figure 3 plots objective value over function evaluations for “TrajEvo w/o CGES” and “TrajEvo,” but the curves have no error bars, no axis labels with units, and no indication of how many runs were averaged; the claimed benefit of CGES is therefore not statistically substantiated.","section":"Figure 3"},{"comment":"The repeated header “A VG→SDD” is confusing, and it is not explained why the heuristic baselines (SocialForce, CVM, CVM-S) are constant across the ETH, HOTEL, UNIV, ZARA1, and ZARA2 columns while the neural baselines vary; please clarify the split-dependence of each row.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of cs.AI and the central idea is timely and interesting. My main concern is evidentiary rather than conceptual: the headline cross-dataset claim depends on an under-specified baseline evaluation, and the stochastic search is reported without variance information. These issues are fixable with additional experiments and protocol details, so I recommend major revision rather than rejection. I would also encourage the editor to request the code and data-processing scripts as part of the revision to verify the baseline protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2505.04480. First, it is a genuine step forward in applying LLM-based evolutionary search to trajectory prediction: the authors adapt ReEvo to this domain, add two concrete mechanisms (Cross-Generation Elite Sampling and a Statistics Feedback Loop), and show via ablations that both help. Second, the paper's headline claim—that evolved heuristics beat recent deep learning models on the unseen SDD dataset—is not yet supported by the evidence as written.\n\nWhat is good: the in-distribution results against classical heuristics are solid and consistent. TrajEvo handily beats CVM, Social Force, and other heuristics on ETH-UCY, and the generated code is genuinely interpretable. The inference speed (0.65 ms per instance on a single CPU core) is meaningful for robotics. The authors are also honest in the Limitations section, conceding that in-distribution accuracy trails state-of-the-art neural methods. That is not a paper hiding its weaknesses. Code is promised, which is good practice.\n\nThe soft spots are proportional to how load-bearing they are. Table 3 reports minADE20/minFDE20 for TrajEvo versus Trajectron++, EigenTrajectory, and MoFlow on SDD, with no protocol for the neural rows: nothing about how those models were trained on the ETH-UCY splits, what SDD preprocessing was used (frame rate, observation/prediction horizons, units, coordinate conventions), or whether the same K=20 sampling pipeline was applied. If those numbers come from published papers with different settings, the comparison is invalid. The gap over EigenTrajectory is about 1.9 px in minADE, small enough that it could vanish with proper baseline tuning or fall within the run-to-run variance of TrajEvo's own stochastic search—the LLM temperature is 1, crossover and mutation are random, and every table shows a single run with no seeds or error bars.\n\nThe circularity concern raised by the stress-test is a red herring: the evolution optimizes a training-split objective and is evaluated on held-out test splits, which is ordinary supervised training. That part holds up.\n\nThe paper is for researchers who want fast, interpretable prediction for autonomous driving or robot navigation, and for anyone interested in LLM+EA as a way to design lightweight models. It deserves a serious referee because the idea is novel and the ablations are informative, but it needs major revision. I'd send it to review with a clear request: document the baseline training and evaluation, report multiple seeds with variance, and soften the abstract's claim to match what was actually tested. With those changes, this could be a useful contribution.","headline":"TrajEvo is a credible, well-scoped application of LLM-driven evolution to trajectory prediction, but its headline claim of beating deep learning on SDD rests on an under-documented Table 3 and single stochastic runs.","tokens_in":18494,"tokens_out":3595,"would_cite":true,"duration_ms":30904,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM inside an evolutionary loop writes trajectory-prediction heuristics that beat handcrafted rules in-distribution and transfer to unseen data better than neural models.","keywords":["trajectory prediction","large language models","evolutionary algorithms","automated heuristic design","cross-dataset generalization","Cross-Generation Elite Sampling","Statistics Feedback Loop","multi-agent pedestrian prediction"],"falsifier":"Independently train EigenTrajectory and MoFlow on the same ETH-UCY leave-one-out splits used for TrajEvo and evaluate them on SDD with identical frame scaling, units, $K=20$ sampling, and code infrastructure. If either model then matches or beats TrajEvo's average $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $12.65/24.14$ pixels, the paper's headline claim—that evolved heuristics outperform deep learning under distribution shift—is falsified.","tokens_in":17379,"feed_emoji":"🚶","tokens_out":10177,"duration_ms":87942,"temperature":0.7,"pith_summary":"TrajEvo tries to establish that trajectory-prediction heuristics—short, readable pieces of code—can be designed automatically by a large language model inside an evolutionary loop, instead of being handcrafted by experts or encoded in large neural networks. The framework seeds a population with a constant-velocity predictor, then repeatedly evaluates, reflects on, recombines, and mutates candidate Python heuristics on the ETH-UCY benchmark. If its results hold, the payoff is practical: evolved heuristics match or beat handcrafted heuristics in-distribution, and on the unseen Stanford Drone Dataset they outperform recent neural models in average-position error while running about 300 times faster on a single CPU core. The authors present this as a first step toward automatically designed, explainable, and generalizable prediction rules.","feed_headline":"LLM-evolved heuristics beat neural nets on unseen data","feed_subtitle":"On Stanford Drone traffic they beat deep baselines while running 300x faster on CPU.","key_machinery":"The engine is a reflective evolutionary loop, adapted from the Reflective Evolution paradigm, in which an LLM implements the genetic operators: it writes the initial population from a constant-velocity seed function, performs crossover by mixing code from two parents under textual reflections, and mutates an elite candidate. Two mechanisms carry the novel argument. Cross-Generation Elite Sampling is a mutation-target selector: instead of always mutating the current best heuristic, it keeps a history of high-performing heuristics and samples the one to mutate with a Softmax over their recorded objective values, a design meant to escape local optima. The Statistics Feedback Loop is a per-slot diagnostic: after evaluating $K=20$ predicted trajectories per agent, it counts how often each of the 20 slots produced the lowest average displacement on individual instances and feeds that distribution, together with the heuristic code, into the reflection and mutation prompts. That lets the LLM discover which diversification strategies actually earn their keep—for instance, the evolved Zara1 heuristic reserves one slot for a near-deterministic linear extrapolation. The objective being minimized is $J = 0.6\\, \\mathrm{minADE}_{20} + 0.4\\, \\mathrm{minFDE}_{20}$.","core_discovery":"On its own terms, the paper's discovery is that simple automatically evolved code can transfer across scene distributions better than neural networks trained on the source distribution. Evolved on ETH-UCY, TrajEvo's best heuristics report an average $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $0.36/0.71$ m on the ETH-UCY test sets, ahead of every heuristic baseline it compares against. In the headline cross-dataset experiment, applying heuristics evolved on each ETH-UCY split to the unseen SDD gives an average $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $12.65/24.14$ pixels, versus $14.55/25.71$ for EigenTrajectory and $17.14/28.56$ for MoFlow. The paper is explicit that in-distribution the strongest recent neural models still lead; the claimed advantage is specifically on unseen data, plus speed and explainability. Separately, the authors show that the two added mechanisms—Cross-Generation Elite Sampling and the Statistics Feedback Loop—each improve the search, and they report inference at 0.65 ms per instance on a single CPU core.","pith_inferences":["The same evolution recipe could be applied to other forecasting tasks where kinematic baselines still compete, such as vehicle motion or sports tracking; the paper only evaluates pedestrian datasets.","The Statistics Feedback Loop's observed benefit of keeping one deterministic linear extrapolation among the 20 samples suggests a general design principle for multimodal predictors: reserve a 'safe' anchor slot before sampling diversity. This is an interpretation, not a claim the paper tests directly.","Because the evolved heuristics are plain Python, they could be compiled or translated for embedded deployment; the paper notes C++ conversion as future work and reports a preliminary more than 20x speedup from a zero-shot translation request.","If the cross-dataset result holds under a shared protocol, it would indicate that simple inductive biases written as code can be more transferable than learned representations—a causal claim the paper's experiments do not directly establish."],"forward_implications":["On ETH-UCY, the best evolved heuristic averages $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $0.36/0.71$ m, placing it ahead of every heuristic baseline listed in Table 1 on every dataset.","Under leave-one-out training on ETH-UCY and testing on the unseen SDD, evolved heuristics average $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $12.65/24.14$ px, better than EigenTrajectory ($14.55/25.71$) and MoFlow ($17.14/28.56$).","Generated heuristics run at 0.65 ms per instance on one CPU core, more than 300 times faster than the CPU inference time reported for MoFlow, making real-time CPU-only deployment plausible.","Ablation of either the Statistics Feedback Loop or Cross-Generation Elite Sampling raises ETH-UCY errors, so both mechanisms contribute to the evolved heuristics' quality.","A full evolution run costs roughly $0.05 in API usage and about five minutes, two orders of magnitude cheaper than the reported one-day GPU training of neural baselines."],"supporting_citations":[{"why":"Supplies the reflective LLM-driven evolutionary loop that TrajEvo adapts.","marker":"[38]"},{"why":"Seed heuristic for the initial population and the strongest kinematic baseline.","marker":"[46]"},{"why":"Classic handcrafted heuristic baseline and source of the repulsion idea that appears in evolved code.","marker":"[10]"},{"why":"Seminal neural baseline that TrajEvo claims to outperform on ETH-UCY.","marker":"[18]"},{"why":"Neural baseline included in both the ETH-UCY and cross-dataset SDD comparisons.","marker":"[19]"},{"why":"Strongest deep baseline in the SDD generalization table and source of the training-cost comparison.","marker":"[54]"},{"why":"Recent neural baseline with the best in-distribution errors and a reported SDD transfer result.","marker":"[28]"},{"why":"Unseen target dataset used for the cross-dataset generalization test.","marker":"[65]"},{"why":"Defines the multi-sample K=20 evaluation protocol and minADE/minFDE metrics.","marker":"[22]"},{"why":"Define the ETH and UCY datasets that compose the benchmark.","marker":"[61, 62]"}],"fun_headline_variants":["Evolved heuristics beat neural nets on unseen scenes","LLM-driven evolution designs transferable predictors","Evolutionary search outdoes deep learning on new data","TrajEvo: fast, generalizable heuristics via LLM evolution","Automated heuristic evolution surpasses deep models on SDD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline cross-dataset comparison assumes the deep learning baselines were trained and evaluated under exactly the same leave-one-out splits, preprocessing, units, and $K=20$ protocol as TrajEvo, a protocol Section 4.3 does not describe.","fun_headline_variants_meta":{"raw":{"variants":["Evolved heuristics beat neural nets on unseen scenes","LLM-driven evolution designs transferable predictors","Evolutionary search outdoes deep learning on new data","TrajEvo: fast, generalizable heuristics via LLM evolution","Automated heuristic evolution surpasses deep models on SDD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3513,"prompt_tokens":996,"completion_tokens":2517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2437}},"tokens_in":612,"tokens_out":2517,"duration_ms":18483,"temperature":1.0,"reasoning_tokens":2437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:27:49.891843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently train EigenTrajectory and MoFlow on the same ETH-UCY leave-one-out splits used for TrajEvo and evaluate them on SDD with identical frame scaling, units, $K=20$ sampling, and code infrastructure. If either model then matches or beats TrajEvo's average $\\mathrm{minADE}_{20}/\\mathrm{minFDE}_{20}$ of $12.65/24.14$ pixels, the paper's headline claim—that evolved heuristics outperform deep learning under distribution shift—is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reflective LLM-driven evolutionary loop that TrajEvo adapts."},{"cited_title":"Sch ¨oller, V","cited_arxiv_id":null,"evidence_quote":"Seed heuristic for the initial population and the strongest kinematic baseline."},{"cited_title":"Helbing and P","cited_arxiv_id":null,"evidence_quote":"Classic handcrafted heuristic baseline and source of the repulsion idea that appears in evolved code."},{"cited_title":"Alahi, K","cited_arxiv_id":null,"evidence_quote":"Seminal neural baseline that TrajEvo claims to outperform on ETH-UCY."},{"cited_title":"Salzmann, B","cited_arxiv_id":null,"evidence_quote":"Neural baseline included in both the ETH-UCY and cross-dataset SDD comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recent neural baseline with the best in-distribution errors and a reported SDD transfer result."},{"cited_title":"```python ... ```","cited_arxiv_id":null,"evidence_quote":"Unseen target dataset used for the cross-dataset generalization test."},{"cited_title":"Gupta, J","cited_arxiv_id":null,"evidence_quote":"Defines the multi-sample K=20 evaluation protocol and minADE/minFDE metrics."}],"review_version":1}