{"id":"5da42219-8ecb-4b98-9fab-ba2f478b9563","arxiv_id":"2608.00700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DGA2D evolves LLM-generated algorithm pipelines as directed walks in an operator graph with first-order transition credit, outperforming LLM-based heuristic design baselines on 12 combinatorial optimization benchmarks.","lead":"This paper presents DGA2D, a framework that uses a directed graph to represent algorithm structures and lets an LLM evolve both the graph topology and operator code for solving combinatorial optimization problems. It reports consistent performance gains over other LLM-based heuristic design systems across 12 NP-hard problem domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvement numbers are internally inconsistent: the 10.96 pp and 9.67 pp claims do not match Table 1, and the GPT comparison is against a non-strongest baseline.","rationale":"The reader's weakest_assumption is the first-order sufficiency condition in Eq. (127). I agree this assumption is not directly tested, and Theorem 2 only establishes first-order optimality conditional on it. However, it is not the most load-bearing point for the central claim: the paper's own ablation in Table 3 directly compares zero-, first-, second-, and full-path credit and finds first-order best, so the empirical case for the mechanism does not depend on the sufficiency assumption, and Appendix D.2 explicitly qualifies that assumption's scope. The more damaging issue is that the headline quantitative claim, the 'up to 10.96 pp' reduction, is contradicted by the paper's own Table 1 arithmetic. Section 4.2.2 misidentifies the strongest baseline under GPT-5.6 Sol and contains an incorrect DGA2D average under DeepSeek-V4-Flash. This is directly load-bearing because the abstract and contributions are quantified by these numbers, and there is no public repository or data link to audit the raw calculations. The per-task pattern in Tables 1 and 8 is consistent and supports a qualitative advantage, so the appropriate verdict remains conditional rather than rejection; the paper should correct the averages and either release raw data/reproducibility artifacts or clearly state the comparison baseline. This is why I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":43762,"tokens_out":14908,"duration_ms":125321,"concrete_test":"Recompute the four-domain average normalized gaps from the raw per-instance data behind Table 1 and compare with the text in Section 4.2.2: (i) verify whether the DeepSeek DGA2D average is 6.02% or 6.52%; (ii) identify the lowest-mean LLM baseline under each backbone and recompute the improvement over that baseline. If the recomputed values are 6.52% and 9.17 pp (DeepSeek) and 14.51% and 9.91 pp (GPT-5.6 Sol), the reported 9.67 and 10.96 pp claims and the abstract's 'up to 10.96 pp' must be corrected; if raw data produce the original numbers, the discrepancy is only in rounding or reference choice and the central claim is unaffected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.2's headline numbers are the paper's central quantitative evidence, and they do not survive recomputation from Table 1. Under DeepSeek-V4-Flash, DGA2D's four-domain average is (5.24+9.94+10.40+0.51)/4 = 6.52%, not the stated 6.02%; the strongest LLM baseline is MEoH at 15.69%, so the improvement is 9.17 pp, not 9.67. Under GPT-5.6 Sol, the strongest baseline average is MCTS-AHD at (18.50+23.32+15.16+1.05)/4 = 14.51%, not ReEvo at 15.56%; the improvement over the true strongest baseline is 14.51−4.60 = 9.91 pp, not 10.96. Thus the main text's claim that this is over the strongest baseline is false, and the DeepSeek claim depends on a wrong DGA2D average. Because no public code or data link is provided, these discrepancies cannot be resolved by inspection of raw results. The qualitative pattern (DGA2D best on each task) may still hold, but the headline magnitude reported for the central claim is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DGA2D, a framework for LLM-driven automated heuristic design in which the space of complete solver programs is represented as a directed graph: nodes are functional operators, each with a pool of code implementations, and directed walks constitute algorithmic pipelines. An LLM proposes new implementations and edge edits, while a first-order path-dependent credit assignment mechanism, based on standardized terminal rewards of complete pipelines, guides implementation sampling and dual-level evolution of pools and graph topology. The authors claim consistent superiority over four prior LLM-based AHD baselines across 12 NP-hard combinatorial optimization problems under two LLM backbones, reporting average normalized-gap reductions of up to 10.96 percentage points, and they present ablations, convergence analyses, resource accounting, and extensions to continuous numerical tasks. The appendices contain formal statistical and spectral analysis of the credit assignment and graph representation.","tokens_in":44037,"tokens_out":9042,"duration_ms":70848,"significance":"If the claims hold after correction, the work is significant: it moves LLM-based automated heuristic design from isolated module tuning toward system-level algorithm architecture search, and it provides a concrete mechanism (directed-graph search space plus first-order transition credit) that appears broadly effective across very different problem domains. The paper is strong in breadth: 12 COP domains, two LLM backbones, ten independent runs, multiple ablations, wall-clock and token accounting, out-of-distribution tests, and extensions to ODE solving, control, and root finding. The formal appendices are careful and self-scoped, and the qualitative result that DGA2D is best on essentially every reported domain-backbone combination is consistent across Tables 1 and 8. However, the headline quantitative claims in the abstract and Section 4.2.2 do not survive direct recomputation from Table 1, and there are further table-to-text conflicts in the extended results. The central empirical direction appears defensible, but the reported magnitudes must be corrected and reconciled before the claims can be accepted as stated.","major_comments":[{"comment":"The headline improvement numbers are inconsistent with Table 1. Under DeepSeek-V4-Flash, DGA2D's four gaps in Table 1 average to (5.24+9.94+10.40+0.51)/4 = 6.52%, not the stated 6.02%. The strongest LLM baseline is MEoH at (19.17+26.37+16.48+0.73)/4 = 15.69%, so the actual improvement is 15.69−6.52 = 9.17 percentage points, not 9.67. Under GPT-5.6 Sol, the strongest baseline by Table 1 is MCTS-AHD at (18.50+23.32+15.16+1.05)/4 = 14.51%, not ReEvo at 15.56%; the improvement is 14.51−4.60 = 9.91 percentage points, not 10.96. Thus the claim that the 10.96 pp reduction is against the strongest baseline is false as written, and the DeepSeek claim relies on an incorrect DGA2D average. The authors should recompute all averages, correct the abstract and Section 4.2.2, and state explicitly which baseline is used for each comparison.","section":"§4.2.2 and Table 1"},{"comment":"The extended results contain a further numerical inconsistency for GPT-5.6 Sol. The text states that DGA2D's average gap of 2.14% is compared with 9.77% for the strongest baseline, yielding a 7.63 pp improvement. From Table 8, the per-method baseline averages are EoH (6.75+24.36+0.25+9.07+9.68+16.53+6.59+6.01)/8 = 9.91%, MCTS-AHD 13.91%, MEoH 12.29%, and ReEvo 11.71%; no baseline average equals 9.77%. If EoH is the strongest baseline, the improvement is 9.91−2.14 = 7.77 pp, not 7.63. The authors need to reconcile the stated baseline average with the values in Table 8 or explain which subset of benchmarks was used.","section":"Appendix F.1 and Table 8"},{"comment":"Table 9 reports an FJSP gap of 22.20% for MCTS-AHD under DeepSeek-V4-Flash, whereas Table 1 reports 24.59±5.25% for the same configuration, backbone, and benchmark. This is not a rounding difference. The claimed reduction range of 55.2%–60.1% for DGA2D relative to MCTS-AHD depends on the Table 9 value for its lower bound; using Table 1 would give a range of roughly 59.6%–60.1%. The authors should determine which table is correct and ensure that all backbone-sensitivity statistics are computed from the same underlying runs.","section":"Table 9 vs. Table 1"}],"minor_comments":[{"comment":"The abstract and the introduction repeat the 10.96 percentage-point improvement and the 'strongest baseline' framing; these statements should be revised after the Table 1 recomputation so that the abstract does not overstate the headline result.","section":"Abstract and §1"},{"comment":"The pipeline notation uses L_{t,j} in Eq. (1) before the length bound and validity constraints are introduced; defining L_{t,j} and L_max before Eq. (1) would improve readability.","section":"§3.3, Eq. (1)"},{"comment":"The theoretical support for first-order credit assignment explicitly relies on the first-order sufficiency assumption in Eq. (127), and the appendix itself notes that this assumption may fail. The main text should acknowledge this dependence when citing the statistical analysis, since the empirical ablation in Table 3 is the actual evidence for first-order credit in practice.","section":"Appendix D.2"},{"comment":"The text states that an anonymized repository is submitted with the supplementary material, but no link or access instructions appear in the manuscript. Given the table-to-text discrepancies noted above, providing the code, raw result files, and evaluation scripts is essential for verification.","section":"Appendix E.1"},{"comment":"The sentence 'DGA2D ranks first in all 16 combinations of the eight additional benchmarks and two LLM backbones' should say 'all 16 domain-backbone combinations' to avoid ambiguity about what is being counted.","section":"Appendix F.1, final paragraph"}],"recommendation":"major_revision","confidential_remarks":"The qualitative pattern in the experiments is remarkably consistent: DGA2D is best on every domain and backbone combination in Tables 1 and 8, and the ablations support the design choices. My recommendation is driven by the arithmetic of the headline claims rather than by any suspicion about the underlying results. The central text should be corrected to match the tables, the extended-results baseline averages should be recomputed, and the FJSP/MCTS-AHD discrepancy should be resolved. I would also ask the editor to require a functioning code/data link, because the current discrepancies cannot be resolved by inspection of the text alone."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a real contribution to LLM-driven automated heuristic design, and the qualitative claim survives, but the headline numbers in Section 4.2.2 are wrong as written. Averaging Table 1 under DeepSeek-V4-Flash gives DGA2D a 6.52% average gap, not 6.02%, and the improvement over the strongest baseline (MEoH, 15.69%) is 9.17 percentage points, not 9.67. Under GPT-5.6 Sol, the strongest baseline is MCTS-AHD at 14.51%, not ReEvo at 15.56%, so the improvement is 9.91 percentage points, not 10.96. The abstract's 'up to 10.96' is therefore not supported by the table.\n\nWhat's genuinely new: the directed graph representation, where nodes are operators with their own implementation pools and pipelines are directed walks, is a natural and useful step beyond fixed-template heuristic evolution. The first-order transition credit is a sensible addition. The experimental breadth is real: 12 COPs, two LLM backbones, 10 seeds, plus ablations on structure and credit order. The ablations show DG beating Fixed/Linear/DAG and first-order credit beating zero-order, second-order, and full-path, which makes the design choices credible. The appendices include honest formal work—Theorem 2's bias-variance comparison is correctly scoped, and the spectral analysis of pipeline expressivity is a nice addition. No obvious citation problems.\n\nSoft spots, in proportion. The numeric inconsistency is the main one; it touches the central quantitative claim. It is fixable but should be caught before publication. No code or data is actually linked, despite the appendix saying an anonymized repository was submitted; that needs resolution. The 3D-CLP reference is the EP+FF heuristic, not a best-known solution, so negative gaps are possible and that domain's numbers are not directly comparable to the others. The first-order sufficiency assumption (Appendix D.2, Eq. 127) is load-bearing for Theorem 2 and is not directly tested; the ablation comparing credit orders is consistent with it but doesn't isolate the assumption because the search space itself changes. The baseline set is reasonable, and skipping A2DEPT because no official implementation exists is fair.\n\nWho this is for: researchers in LLM-based algorithm design or hyper-heuristics. It advances the subfield, even with the reporting issues. I'd send it to peer review after the authors fix the arithmetic and make code and data available. The qualitative rank-first pattern across tasks is too consistent to be a fluke.","headline":"Strong empirical framework for LLM-driven algorithm design, but the headline gains in Section 4.2.2 are mis-reported and the true gap versus the strongest baseline is smaller than claimed.","tokens_in":44590,"tokens_out":3225,"would_cite":true,"duration_ms":28972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims graph-guided evolution of both operator code and operator connections cuts average optimality gaps by up to 10.96 percentage points on 12 NP-hard optimization problems.","keywords":["automated heuristic design","large language models","combinatorial optimization","directed graph search space","credit assignment","program evolution","NP-hard problems","algorithm generation"],"falsifier":"Record every activated implementation transition together with the two preceding implementations and its standardized terminal reward; fit $\\mathbb{E}[Y \\mid c_{i-1}, c_i]$ versus $\\mathbb{E}[Y \\mid c_{i-2}, c_{i-1}, c_i]$ on one half of the transitions and compare held-out predictive error. If the second-order context predicts significantly better, the first-order sufficiency assumption is false and the theoretical basis of the credit-assignment mechanism collapses; the empirical benchmark claims could also be tested by independent reruns on fresh instance splits.","tokens_in":43549,"feed_emoji":"⚙️","tokens_out":12412,"duration_ms":104900,"temperature":0.7,"pith_summary":"This paper claims that automated algorithm design works better when the search space is a directed graph whose nodes are functional operators and whose directed walks are complete algorithms, rather than a fixed solver template with tunable modules. The proposed DGA2D framework lets a large language model edit operator code, rewire operator connections, and choose code implementations, while a first-order path-dependent credit mechanism assigns each sampled algorithm's performance to individual code transitions. Across 12 NP-hard combinatorial optimization problems spanning scheduling, routing, packing, and graph optimization, the method reports reducing the average normalized gap by up to 10.96 percentage points relative to the strongest LLM-driven heuristic-design baselines. If correct, this matters because it suggests whole-algorithm design, not just module tuning, is achievable with current LLMs while keeping the search tractable enough to improve many problem classes.","feed_headline":"LLM-designed solvers gain up to 10.96 points from graph-guided search","feed_subtitle":"A directed-graph search space with per-transition credit assignment outsearches module-level LLM heuristics on all 12 benchmarks.","key_machinery":"The machinery is a directed graph $D_t = (O_t, E_t)$ where each node $O_i$ is a functional operator carrying a pool $\\mathcal{C}^{(t)}_i$ of code implementations, together with a first-order transition-credit estimator $Q_t(e)$ for transitions $e = (c_{t,j,i-1}, c_{t,j,i})$. Candidate algorithms are constructed by sampling a bounded directed walk from $O_{\\mathrm{in}}$ to $O_{\\mathrm{out}}$ and then sampling one implementation per operator from a softmax over transition credits. After evaluating $N$ candidates, the standardized terminal reward $\\tilde{R}_{t,j}$ updates every activated transition via $Q_{t+1}(e) = (1-\\alpha)Q_t(e) + \\alpha \\tilde{R}_{t,j}$, and aggregated credits drive dual-level evolution: delete, add, or replace the lowest-credit implementation of the lowest-credit operator, and replace the lowest-credit edge with a compatible new edge proposed by the LLM. The appendix proves that under a first-order sufficiency assumption on expected terminal rewards, first-order context minimizes finite-sample estimation risk among cell-mean estimators, and that the number of structural pipelines grows at the spectral radius of the input-output-relevant strongly connected components.","core_discovery":"The central discovery is that jointly evolving operator implementations and operator connectivity is not only feasible but systematically better than tuning heuristics inside a fixed template, provided the algorithm space is represented as a directed graph with per-operator implementation pools. A pipeline is a bounded directed walk from an input sentinel to an output sentinel; cycles are allowed, so an operator motif can appear repeatedly as iterative refinement. Performance is observed only on complete candidate algorithms, so DGA2D distributes each standardized terminal reward back to first-order implementation transitions, pairs $(c_{i-1}, c_i)$, and uses the accumulated transition credits to sample implementations, to add or delete low-credit operator implementations, and to replace low-credit graph edges. The reported experiments show consistent wins over four LLM-based heuristic-design baselines across all 12 domains, with the largest average normalized-gap reduction of 10.96 percentage points under one of the two LLM backbones.","pith_inferences":["The first-order credit update is essentially temporal-difference-style reward estimation over program fragments; one could import exploration bonuses, upper-confidence bounds, or off-policy corrections into the same update, which the paper does not explore.","Because the representation separates operator structure from operator implementation, a credit table learned on one domain might warm-start evolution on a related domain; the paper evaluates each domain from scratch.","The poor performance of full-path credit is reported, not explained; a plausible reading is finite-sample variance, and a regularized higher-order estimator could test whether longer histories carry any true signal beyond first-order transitions.","The paper itself acknowledges remaining limitations: computational cost of some generated algorithms, sensitivity to the initial operator pool, and potential cross-module inconsistencies, which delimit rather than overturn the reported benchmark gains."],"forward_implications":["LLM-based automated heuristic design can move from module-level tuning to system-level evolution, and on the 12 tested NP-hard domains the average normalized gap drops by 9.67 percentage points under one backbone and 10.96 under the other.","Cyclic directed-graph structure is useful: the DG representation beats fixed, linear, and DAG variants on CVRP, FJSP, MIS, and 3D-CLP, meaning repeated operator motifs can express iterative refinement.","First-order path-dependent credit is the best context granularity among those tested, outperforming zero-order, second-order, and full-path credit, which suggests that overly long histories add noise rather than signal.","The gains are not tied to a single LLM: DGA2D ranks first under both main backbones and across all eight backbones tested on flexible job-shop scheduling.","The framework also produces competitive algorithms on continuous tasks such as ODE integration, feedback control, and root-finding, so the design principle may transfer beyond discrete combinatorial benchmarks."],"supporting_citations":[{"why":"Supplies the no-free-lunch argument that motivates problem-specific automated search rather than one universal solver.","marker":"Wolpert and Macready 1997"},{"why":"Defines automated heuristic design and the hyper-heuristic research context the paper extends.","marker":"Burke et al. 2013"},{"why":"Demonstrates LLM-driven program search, the lineage DGA2D pushes from isolated heuristics to whole pipelines.","marker":"Romera-Paredes et al. 2024"},{"why":"Provides the evolution-of-heuristics baseline that DGA2D compares against under a shared budget.","marker":"Liu et al. 2024"},{"why":"Provides the tree-structured search baseline used in the benchmark comparison.","marker":"Zheng et al. 2025"},{"why":"Provides the multi-objective heuristic-evolution baseline used in the benchmark comparison.","marker":"Yao et al. 2025"},{"why":"Provides the reflection-guided hyper-heuristic baseline used in the benchmark comparison.","marker":"Ye et al. 2024"},{"why":"Concurrent system-level automatic algorithm design work that motivates DGA2D's graph representation but had no public implementation for comparison.","marker":"Chen et al. 2026"},{"why":"Supplies the iterated local search classical baseline used alongside OR-Tools.","marker":"Lourenço, Martin, and Stützle 2018"}],"fun_headline_variants":["Graph-guided LLM algorithm design gains 10.96 points on 12 benchmarks","DGA2D: directed-graph search yields 10.96-point gap reduction","Automated heuristic design with graph-guided LLM closes gap by 10.96","LLM-designed algorithms improve 10.96 points via graph-guided search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is first-order sufficiency: the expected terminal reward of a code transition depends only on the immediate predecessor implementation and the current implementation, not on longer history; if that fails, higher-order contexts carry useful information and the theoretical case for first-order credit assignment weakens.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided LLM algorithm design gains 10.96 points on 12 benchmarks","DGA2D: directed-graph search yields 10.96-point gap reduction","Automated heuristic design with graph-guided LLM closes gap by 10.96","LLM-designed algorithms improve 10.96 points via graph-guided search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3353,"prompt_tokens":952,"completion_tokens":2401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2315}},"tokens_in":568,"tokens_out":2401,"duration_ms":17118,"temperature":1.0,"reasoning_tokens":2315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:18:16.487196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record every activated implementation transition together with the two preceding implementations and its standardized terminal reward; fit $\\mathbb{E}[Y \\mid c_{i-1}, c_i]$ versus $\\mathbb{E}[Y \\mid c_{i-2}, c_{i-1}, c_i]$ on one half of the transitions and compare held-out predictive error. If the second-order context predicts significantly better, the first-order sufficiency assumption is false and the theoretical basis of the credit-assignment mechanism collapses; the empirical benchmark claims could also be tested by independent reruns on fresh instance splits.","supporting_citations":[{"cited_title":"2025 , editor =","cited_arxiv_id":null,"evidence_quote":"Provides the tree-structured search baseline used in the benchmark comparison."}],"review_version":1}