{"id":"5466c12f-d8c4-4a3f-a2d1-6686a7c0484a","arxiv_id":"2510.27353","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The LLM-generated bin packing heuristics from Nature's FunSearch paper reduce to simple two-parameter threshold rules and don't constitute a mathematical discovery.","lead":"The celebrated claim that an AI found a new mathematical result for packing boxes is re-examined and found to reduce to two simple dials. If right, it changes how the field should validate AI-assisted 'discoveries' in optimization.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's verdict is ACCEPT with high confidence, and the stress-test agrees. The weakest assumption identified by the reader—the fidelity of the c14 threshold interpretation—is indeed a real caveat: the authors explicitly state that their ab-WorstFit interpretation 'differs non negligibly' from c14's actual behavior. However, this caveat does not rise to a load-bearing flaw in the central claim. The paper's conclusion that the LLM-evolved heuristics do not amount to a mathematical discovery rests on a broader evidence base: (i) c12 is exactly reducible to a simple threshold rule; (ii) EoH's priority function is transparently a variant of the same principle; (iii) a simple two-parameter family can match or beat the LLM heuristics on the same distributions, showing the instances are not algorithmically deep; (iv) prior benchmarking (Sim et al.) demonstrates poor generalization of the LLM heuristics beyond narrow parameter settings; and (v) no subsequent work has built on the so-called discoveries. Even if c14's exact decisions deviate from ab-WorstFit, the absence of conceptual innovation remains well-supported. Thus the concern is worth a quantitative check to tighten the paper's language, but it does not invalidate the ACCEPT verdict. The proposed decision-agreement test directly addresses the one interpretive step that is admittedly approximate, providing a concrete way to settle whether the 'reduces to' claim for c14 is faithful or should be qualified.","tokens_in":15797,"tokens_out":8292,"duration_ms":86177,"concrete_test":"Run the authors' public code on 100 Weibull(3.0,45) instances of 5000 items and compute the exact fraction of items for which c14 and ab-WorstFit(a=1,b=22) make the same decision (same bin or both open a new bin). Report the agreement rate and also the wasted-space difference. If agreement is below, say, 90%, the paper's 'reduces to' phrasing for c14 should be softened to 'approximated by'; if above, the interpretation is validated. This test settles the only soft spot in the argument.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No load-bearing concern identified. The paper's central claim—that FunSearch's bin-packing heuristics largely rediscover simple threshold policies and do not constitute mathematical discovery—is supported by multiple independent lines: c12 simplifies almost exactly; EoH's score function visually encodes the same threshold principle; ab-Baselines matches or beats c14/EoH on the training distributions; LLM-evolved heuristics are known to generalize poorly (Sim et al.); and no subsequent work builds on the 'discovered' heuristics. The weakest spot is the admitted non-negligible divergence of the c14-to-ab-WorstFit interpretation (Section 3.3.1). But the conclusion does not hinge on exact decision-level equivalence; it hinges on the absence of conceptual insight, which is independently evidenced. A quantitative agreement check would strengthen the paper but is not required to sustain the current verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reassesses the claim that LLM-based program search (FunSearch) produced mathematically interesting discoveries for online bin packing. The authors analyze the c12 heuristic for Uniform(20,100) and the c14/EoH heuristics for Weibull(3,45), arguing that these human-readable but opaque programs reduce to two-parameter threshold policies. They introduce ab-FirstFit, ab-WorstFit, and ab-BestFit, and report that on the original distributions these simple heuristics match or slightly exceed the LLM-evolved heuristics (ab-FirstFit 2.1% improvement over BestFit vs c12's 2.0%; ab-WorstFit 3.3% vs c14's 3.3%). The paper concludes that the LLM frameworks did not develop conceptual understanding of bin packing but rediscovered known strategies through stochastic search.","tokens_in":15997,"tokens_out":7805,"duration_ms":69949,"significance":"If the conclusions hold, this is a valuable, sober corrective to high-profile claims about LLM-driven mathematical discovery. The paper's strengths include released code, clearly specified experimental protocols (100-1000 instances, boxplots, heatmaps), a nearly exact simplification of c12, and a concrete, interpretable, two-parameter algorithmic baseline that is orders of magnitude cheaper to produce than the LLM-evolved heuristics. The paper is also candid about the limits of its c14 interpretation. However, two issues in the present version need attention before the stronger claims (generalization, outperformance, 'largely rediscover') are fully supported: the in-sample selection of the ab parameters and the absence of a quantitative fidelity measure for the c14-to-ab-WorstFit approximation.","major_comments":[{"comment":"The parameters a,b of ab-FirstFit/ab-WorstFit/ab-BestFit are 'selected separately for each problem instance,' and the heatmaps in Figs. 9-10 appear to use the same 100 instances on which performance is reported. This is in-sample tuning: the ab heuristics effectively see the evaluation set, while c12/c14/EoH are fixed. The claim that 'ab-Baselines outperforms all baselines and LLMs-evolved heuristics,' as well as the word 'generalizable' in the abstract, needs a held-out protocol. Please tune on one set (or use cross-validation) and report results on a disjoint test set, and state explicitly how the parameters in Fig. 11 were chosen.","section":"§4.3, Fig. 11"},{"comment":"The proposed interpretation of c14 (BestFit on perfect fit, else WorstFit if residual exceeds item size by more than 20, else open a new bin) is admitted to 'differ non negligibly' from c14's actual behavior. Since this interpretation is the main evidence that c14 'largely' rediscovers a known threshold pattern, the paper should quantify its fidelity: for example, the fraction of decisions where c14 and ab-WorstFit choose the same bin, split by the three cases, and a performance comparison between c14 and ab-WorstFit with tuned parameters on held-out instances. Without this, the reader cannot judge how much of c14's behavior is captured, and the 'largely rediscover' claim remains qualitative.","section":"§3.3.1, Figs. 5-6"}],"minor_comments":[{"comment":"The text reports the best ab-WorstFit parameters as a=1, b=22, while the Figure 10 caption says b=21. Reconcile.","section":"§4.3, Fig. 10"},{"comment":"Line 8 uses an undefined variable x; it should be bin or bins. The function declaration also contains the typo 'ab-Worst!rray'.","section":"Algorithm 6"},{"comment":"capacity is used in the priority functions but is not passed as an argument; define it or note that it is a global constant.","section":"Algorithms 4-6"},{"comment":"The sentence 'the performance can degrade to as much as c/(c-b) times the optimal 1' contains a stray '1' and the derivation is only a sketch. State the formal claim and proof more carefully.","section":"§4.3"},{"comment":"The introduction says the Nature paper has been cited by more than 800 papers as of October 31st, while §2 says 744 as of October 1st. The discrepancy should be explained or harmonized.","section":"§1 and §2"},{"comment":"The 'convenience sample biased toward the most cited' of the remaining 653 papers is not described. If retained, give sampling details or label it as an informal observation.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and makes a useful contribution, but the in-sample parameter selection in §4.3 is the main methodological issue. The c14 fidelity concern is secondary but worth addressing with a quantitative agreement check. I would be happy to accept after a revision that adds a held-out evaluation and a decision-level comparison with c14. The editorializing about the Nature review process in §4.4 is not the technical core and could be toned down."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is the close look at FunSearch's bin-packing heuristics that the Nature paper never provided, and the core conclusion holds up. The authors show that c12 is essentially Best-Fit on tight fits and First-Fit on loose fits, and that a simple two-parameter family (ab-FirstFit, ab-WorstFit, ab-BestFit) matches or beats c12, c14, and EoH on the same distributions they were trained on. The deflationary claim—that these heuristics rediscover simple threshold effects and don't amount to a mathematical discovery—is well supported.\n\nThe empirical work is honest and reproducible: code is public, experiments are clearly specified, and comparisons against the published heuristics are direct. The biggest soft spot is the c14 interpretation: the authors admit in Section 3.3.1 that their threshold model 'differs non negligibly' from actual c14 behavior. So the reduction of c14 is approximate, not exact. But the conclusion doesn't depend on decision-level equivalence. The EoH score function visually encodes the same threshold principle, the LLM heuristics are known to generalize poorly, and nobody has built on the 'discovery.' The evidence converges.\n\nMinor caveats: the a and b parameters are fitted per distribution, so the ab-heuristics are a distillation with two free parameters rather than a derivation—but the paper discloses this and doesn't overclaim. The absence-of-prior-work argument is inherently a negative claim based on a literature search; they hedge it properly. The citation-pattern analysis uses a convenience sample, which is fine as supporting color, not as evidence.\n\nThis paper is for anyone evaluating LLM-assisted discovery claims, and it's a good template for testing a 'discovery' by reconstructing the insight as a simple baseline. It deserves a serious referee; I'd send it to peer review, expecting the usual back-and-forth on the c14 approximation and parameter fitting.","headline":"A careful, reproducible deflation of FunSearch's bin-packing 'discovery' claim; the core conclusion holds, with minor caveats about fitted parameters and the approximate c14 interpretation.","tokens_in":16492,"tokens_out":2352,"would_cite":true,"duration_ms":28160,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68W27","90C27"],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM bin-packing heuristics reduce to two simple thresholds, undercutting their claimed mathematical novelty.","keywords":["bin packing","online algorithms","large language models","evolutionary search","heuristic interpretability","program search","stochastic optimization","mathematical discovery"],"falsifier":"Run c14 side-by-side with ab-WorstFit(a=1, b=21) on a large hold-out sample from Weibull(3.0,45), recording every decision where they diverge (different bin chosen, or one opens a new bin while the other does not). If the divergence rate is high, or if any alternative simple rule explains c14's decisions better than the ab-model, the paper's reduction claim is weakened; conversely, if divergence is rare and bin counts stay within noise, the conclusion is supported.","tokens_in":1485,"feed_emoji":"📦","tokens_out":2008,"duration_ms":63549,"temperature":0.7,"pith_summary":"This paper reassesses the claim that LLM-based program search produced mathematical discoveries by solving online bin packing. Through behavioral analysis of heuristics evolved for the Uniform(20,100) and Weibull(3.0,45) instances, the authors show that, despite human-readable code, the heuristics are opaque and resist expert interpretation. All of them can be approximated and outperformed by a simple two-parameter priority rule: fit tightly if a bin leaves little slack, otherwise use a baseline like WorstFit on loose bins, else open a new bin. The authors conclude that the LLM frameworks discovered effective strategies through stochastic search and empirical feedback, not conceptual understanding, and that the instances were previously unstudied, so the 'mathematical discovery' label is misleading. This matters because it establishes a concrete bar for validating AI-generated heuristics: distill them into a compact, parameterized rule and test generalization.","feed_headline":"LLM bin-packing heuristics reduce to two simple thresholds","feed_subtitle":"A two-parameter rule matches or beats LLM-evolved heuristics and exposes their lack of conceptual novelty.","key_machinery":"The central object is the ab-heuristic family, a two-parameter priority rule: parameter a defines a 'tight fit' zone (use BestFit among bins whose residual space after placing item size s is ≤ s+a), parameter b defines a forbidden gap (bins with residual in (s+a, s+b] are skipped), and a baseline strategy (FirstFit, BestFit, or WorstFit) acts on bins with residual > s+b; if no such bin exists, a new bin opens. This family distills the opaque LLM-evolved heuristics: c12 approximates ab-FirstFit with a=5, b=24 on Uniform(20,100), and c14/EoH approximate ab-WorstFit with a≈1, b≈21 on Weibull(3.0,45). The key work of the ab-family is converting an unreadable score function into a testable, trans","core_discovery":"The paper claims that heuristics produced by LLM-based evolutionary search for online bin packing do not amount to conceptual or mathematical contributions. On the trained distributions, the behaviors of evolved heuristics — including priority functions c12, c14, and a later variant — collapse to a single principle: place an item in the tightest bin if the leftover space is within a threshold a; otherwise place it in a loose bin (leftover beyond b) using a standard rule; otherwise open a new bin. The authors instantiate this as ab-FirstFit, ab-BestFit, and ab-WorstFit, and show that with two tuned parameters these outperform the LLM heuristics and all baselines on the training distributions","pith_inferences":["The distillation strategy used here — approximating an opaque evolved program with a compact parameterized rule and then re-searching the parameter space — could serve as a general post-processing audit step for LLM-evolved heuristics in other combinatorial optimization domains.","The two-threshold principle may be specific to distributions with a positive minimum item size and large item counts; a testable extension is to apply ab-Baselines to discretized or bounded distributions with small minimum sizes to map exactly when the tight-fit/loose-bin dichotomy stops paying off.","The paper's analysis implies a cheap falsification protocol for future 'AI discovery' claims: run the same evolutionary setup while simply enumerating parameterized variants of known heuristics, and compare — this would make assertions of insight empirically testable.","If the pattern generalizes, some celebrated LLM-search results may be explainable as stochastic optimization over a low-dimensional manifold of known algorithm families, tempering expectations for LLM-driven theoretical progress."],"forward_implications":["The LLM-generated heuristics for the considered bin-packing instances are captured by a two-threshold principle, so their apparent novelty is a reparameterization of familiar strategies rather than new algorithmic concepts.","The ab-Baselines heuristic matches or outperforms the LLM-evolved heuristics on the distributions they were trained on and generalizes to other uniform and Weibull instances with a lower bound on item sizes.","The paper derives a worst-case gap of c/(c−b) for ab-Baselines when many arriving items fall below the b threshold, explaining the regime in which this style of heuristic degrades.","A laptop-based grid search over two parameters accomplishes in minutes what required millions of LLM queries and days of evolutionary search, with better or equal performance.","Claims of mathematical discovery from LLM program search need validation against well-established problems with prior literature and simple parameterized baselines; the bin-packing case fails that test."],"fun_headline_variants":["LLM bin-packing 'insights' boil down to two thresholds","Simpler two-parameter rules outperform LLM-evolved heuristics","Bin packing: LLM ideas reduce to simple threshold rules","LLM contribution to bin packing collapses to a simple rule","LLM bin-packing 'discoveries' are just two thresholds"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The paper's core conclusion depends on its interpretation of c14's opaque score function as equivalent to an ab-WorstFit threshold rule — an approximation the authors themselves state differs non-negligibly from what c14 actually does.","fun_headline_variants_meta":{"raw":{"variants":["LLM bin-packing 'insights' boil down to two thresholds","Simpler two-parameter rules outperform LLM-evolved heuristics","Bin packing: LLM ideas reduce to simple threshold rules","LLM contribution to bin packing collapses to a simple rule","LLM bin-packing 'discoveries' are just two thresholds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001249,"raw_usage":{"total_tokens":4939,"prompt_tokens":707,"completion_tokens":4232,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":4157}},"tokens_in":451,"tokens_out":4232,"duration_ms":28915,"temperature":1.0,"reasoning_tokens":4157,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:57:16.557132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run c14 side-by-side with ab-WorstFit(a=1, b=21) on a large hold-out sample from Weibull(3.0,45), recording every decision where they diverge (different bin chosen, or one opens a new bin while the other does not). If the divergence rate is high, or if any alternative simple rule explains c14's decisions better than the ab-model, the paper's reduction claim is weakened; conversely, if divergence is rare and bin counts stay within noise, the conclusion is supported.","supporting_citations":[],"review_version":1}