{"id":"6e3e0e61-e7c5-4717-95c7-c280d94cd4dc","arxiv_id":"2502.10060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DiSciPLE uses LLM-guided evolution to discover interpretable Python programs that predict geospatial quantities, outperforming black-box deep nets on population density and on out-of-distribution generalization.","lead":"This paper introduces an automated search that writes short Python programs which read satellite images, segment them into concepts like roads and forests, and combine those concepts to estimate population, poverty, and biomass. The best programs beat standard deep-learning baselines on the tested tasks while remaining readable by humans.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 35% error reduction over deep baselines may be an input advantage: DiSciPLE is given OSM land-use masks (population) and auxiliary geospatial features (poverty/AGB) while the baselines are not described as receiving them, confounding the central claim.","rationale":"The paper's central contribution is a search method, not a feature set. The evaluation must therefore hold the input information constant across methods. The text is silent on whether deep baselines receive the auxiliary data, and this omission is load-bearing because the reported gains are large and consistent with an input advantage. I agree with the reader's weakest assumption. A concrete rerun with matched inputs would settle it. The paper has strengths: a plausible evolutionary framework with LLM crossover/mutation, an ablation showing critic and simplifier help, and OOD evaluation. But until the input-fairness question is resolved, the headline accuracy claim is not yet established. The 'no prior literature' novelty claim is also questionable given cited work [4], but that is secondary to the correctness-fairness issue.","tokens_in":13792,"tokens_out":3619,"duration_ms":31771,"concrete_test":"Run Deep-Small and Deep-Large with the same auxiliary inputs DiSciPLE receives: for population density, append the 42 OSM land-use masks as extra input channels to the network; for poverty and AGB, concatenate temperature, precipitation, nightlight, and elevation as additional scalar features (or as extra channels after broadcasting). Keep architecture, training hyperparameters, and data splits identical. If the ID/OOD numbers in Tab. 1 shift such that the margin over DiSciPLE disappears or reverses, the input-fairness concern is confirmed. Also report each baseline with a fixed random seed and error bars over 3-5 runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest empirical claim is a 35% lower error than the closest non-interpretable baseline for population density (Abstract), and strong OOD gains on poverty and AGB (Tab. 1). But the experimental protocols are not input-fair. Sec. 4.2.1 states that for population density, 'we also use OpenStreetMaps masks for 42 different land-use concepts as part of the input.' Sec. 4.2.2 and 4.2.3 add 'functions that return average annual temperature, precipitation, nightlight intensity, and elevation at the input location' as primitives for poverty and AGB. Sec. 4.3 lists baselines (Mean, CB, Deep-Small, Deep-Large, Zero-shot, Random Search) but never states whether these deep networks receive the same OSM masks or auxiliary scalar inputs. If Deep baselines receive only raw Sentinel-2 imagery, the comparison conflates a difference in input information with a difference in program-discovery method. The central claim of state-of-the-art scientific program discovery would not be established; the method might simply be a wrapper that selects among precomputed geospatial features.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiSciPLE, an LLM-guided evolutionary search algorithm that synthesizes interpretable Python programs interleaving open-vocabulary segmentation with arithmetic and logical operations, for scientific visual tasks. The method is evaluated on population density estimation, poverty indicator prediction, and aboveground biomass (AGB) estimation from satellite imagery, reporting lower errors than deep baselines and better out-of-distribution generalization, together with a new benchmark and ablations of the proposed critic and simplifier components.","tokens_in":13991,"tokens_out":4224,"duration_ms":37673,"significance":"If the empirical claims hold, DiSciPLE is a meaningful contribution toward interpretable scientific discovery: it demonstrates that LLM-guided evolution can synthesize readable programs on high-dimensional visual tasks, and the critic and simplifier are plausible mechanisms for improving search efficiency. The release of a three-task benchmark for visual program discovery is also a useful community resource. However, the current evidence does not yet establish the headline claim of state-of-the-art accuracy because the evaluation may be confounded by input differences and lacks statistical confidence measures.","major_comments":[{"comment":"The experimental protocol does not establish input parity between DiSciPLE and the deep baselines. Section 4.2.1 states that OpenStreetMaps masks are 'part of the input' for population density, and Sections 4.2.2 and 4.2.3 add temperature, precipitation, nightlight, and elevation as primitives for poverty and AGB. Section 4.3 lists the baselines but never states whether the deep models receive these same inputs. If the baselines see only raw Sentinel-2 imagery, the reported 35% error reduction and the OOD gains may reflect an information advantage rather than the program-search method. Please specify the exact input channels for every baseline, and run an ablation in which DiSciPLE is given the same inputs as the deep baselines.","section":"Sec. 4.2.1-4.2.3 vs Sec. 4.3"},{"comment":"None of the reported scores include error bars or repeated-run statistics. DiSciPLE relies on stochastic LLM sampling and evolutionary operators, so the margin over Deep-Large on population density (0.2607 vs 0.3974 L2-Log) and the OOD gains may be within run-to-run variability. Please report mean and standard deviation over at least five seeds for DiSciPLE and for the stochastic baselines (Zero-shot, Random Search), and state the number of runs used for the deep baselines. This is essential for supporting the 'state-of-the-art' claim.","section":"Sec. 4.4, Tables 1-3"},{"comment":"The claim that these are 'novel tasks with no prior literature' and 'no prior documented solutions in existing literature' is contradicted by the paper's own references, which include prior population-mapping works (Metzger et al. [30,31]), poverty prediction (Xie [41], Yong and Zhou [44]), and AGB estimation (Nathaniel et al. [32]). Please reframe the claim to something like 'no prior interpretable program-synthesis approach for these tasks' to avoid overstating novelty.","section":"Abstract and Section 1"}],"minor_comments":[{"comment":"The 'Oracle' curve in Figure 5 is not defined in the experimental setup; please describe how this baseline is constructed.","section":"Sec. 4.3 / Figure 5"},{"comment":"The Random Search baseline is described only as relying on the stochasticity of LLMs; please specify the number of LLM calls and the sampling procedure so that it is comparable to DiSciPLE.","section":"Sec. 4.3"},{"comment":"The header 'Perfomance' should be 'Performance'.","section":"Table 4"},{"comment":"The phrase 'withcritic' should be 'with critic'.","section":"Figure 2 caption"},{"comment":"The sentence 'We shows its prowess' should be 'We show its prowess'.","section":"Conclusion"},{"comment":"The simplification threshold of 5% is a free parameter; please state whether it is fixed across all tasks and how sensitive the results are to this value.","section":"Sec. 3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the proposed benchmark could be reused by the community. The self-citation overlap with GRAFT [26] and the prior evolution method [4] is not inherently problematic, but the authors should verify that the comparisons involving GRAFT-based zero-shot and the concept bottleneck baseline are apples-to-apples. The input-parity issue is the main blocker; I would like to see revised experiments with matched inputs and error bars before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read DiSciPLE. The core idea is solid: use LLM-driven evolutionary search to synthesize Python programs that interleave open-vocabulary segmentation with arithmetic, then simplify. That is a real extension of the authors' prior ECCV work and ViperGPT-style synthesis, and the ablations show each component (feature-set prediction, critic, simplifier) contributes. The three real-world benchmarks are a useful addition.\n\nThe main empirical claim — 35% lower error than the closest non-interpretable baseline — is not convincingly established because the paper does not state whether the deep baselines received the same auxiliary inputs. Section 4.2.1 gives DiSciPLE OpenStreetMaps masks for 42 land-use concepts; Sections 4.2.2 and 4.2.3 add temperature, precipitation, nightlight, and elevation primitives. Section 4.3 lists baselines but never says what they saw. If the deep models only got raw Sentinel-2 imagery, the comparison conflates input advantage with discovery method. This is exactly the kind of gap a referee should catch; it is the difference between 'learns state-of-the-art programs' and 'wraps precomputed geospatial features.' The stress-test note is on target.\n\nOther soft spots: no error bars or repeated-seed statistics; the 'no prior literature' phrasing is contradicted by the paper's own citations of population-mapping and poverty-mapping work (Metzger et al., Yeh et al.) — the claim should be 'no prior literature for interpretable program discovery on these tasks.' No code or benchmark link appears in the main text, though the project page exists. These are fixable in revision.\n\nCredit where due: the approach is novel, the ablations are informative, and the expert-human comparison is a nice touch. The paper is well written and the limitations section is candid. The central idea is worth pursuing.\n\nRecommendation: send it to review. A good referee will ask for input-parity details, error bars, and a more careful claim about prior literature. With those addressed, it could be a solid contribution.","headline":"Promising method for interpretable scientific program discovery, but the headline accuracy claim rests on an input-parity assumption the paper does not document.","tokens_in":14578,"tokens_out":2385,"would_cite":false,"duration_ms":21616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiSciPLE claims that an LLM-guided evolutionary search can discover Python programs that predict scientific quantities from satellite imagery with lower error than deep neural networks, and that the programs are interpretable and…","keywords":["interpretable machine learning","program synthesis","evolutionary search","large language models","satellite imagery","population density estimation","aboveground biomass","neuro-symbolic learning"],"falsifier":"Take the published tasks and give the deep baselines the exact same auxiliary inputs that DiSciPLE programs receive — OpenStreetMaps land-use masks for population density, and the temperature, precipitation, nightlight, and elevation features for poverty and biomass — then compare errors. If the gap shrinks to near zero, the central claim that evolutionary program search beats black-box models on these tasks is refuted.","tokens_in":13555,"feed_emoji":"🛰️","tokens_out":8470,"duration_ms":67773,"temperature":0.7,"pith_summary":"This paper tries to show that, with the right search procedure, a scientist can get both accuracy and interpretability from visual data. It introduces DiSciPLE, an evolutionary algorithm in which a large language model writes, crosses over, and mutates short Python programs that call an open-vocabulary image segmenter and combine the results with arithmetic. On three real-world tasks — population density, poverty, and aboveground biomass from satellite imagery — the discovered programs achieve lower error than the best non-interpretable deep-network baselines, and they keep their advantage on regions never seen during training. If the claim holds, domain experts would gain predictions that come with a readable mechanism they can inspect, edit, and test.","feed_headline":"LLM-evolved programs beat black-box nets on satellite imagery","feed_subtitle":"A method that turns satellite images into readable formulas, with lower error than black-box models.","key_machinery":"The central object is a Python program that interleaves calls to an open-vocabulary satellite-image segmenter (GRAFT) with arithmetic, logical, and distance-transform operations, and returns a list of features that a linear regressor maps to the target quantity. The search machinery is the evolutionary loop: an LLM (Llama-3-8B) writes the initial population, performs crossover and mutation given the parents' program texts and fitness scores, and the two proposed components — the critic and the simplifier — add fine-grained guidance. The critic partitions the data by land-use type and reports per-partition errors so the LLM can fix weak strata; the simplifier treats the program as an abstract syntax tree and repeatedly removes leaf nodes unreachable from the return statement, then cancels features whose regression weight is below 5% of the largest weight. Feature-set prediction is a third load-bearing design choice: the program outputs several features rather than a single number, so the linear layer, which is fitted conventionally, carries part of the model's expressiveness.","core_discovery":"The paper's claim is that LLM-guided evolutionary search can discover accurate, human-readable programs for scientific vision tasks that have no documented prior solution. DiSciPLE starts from zero-shot LLM programs, then over 15 generations applies LLM-based crossover and mutation, a critic that evaluates each program per land-use stratum and tells the LLM where it fails, and a simplifier that prunes dead code and low-weight features. The final program is a linear model over interpretable features computed by the program, such as distance to water, presence of residential or commercial buildings, and vegetation. On population density estimation, the best discovered program has 35% lower error than the closest non-interpretable baseline, and on out-of-distribution regions it outperforms all baselines, which the authors take as evidence that interpretable-by-design programs generalize better and overfit less.","pith_inferences":["One can test whether the core recipe transfers to other sensor modalities and scientific targets — crop yield, flood risk, or biodiversity from aerial or camera-trap imagery — since the domain-specific pieces are only the LLM prompt and the primitive library.","A clean way to determine whether the reported advantage comes from discovery rather than from extra inputs is to give the deep baselines the same land-use masks, temperature, precipitation, nightlight, and elevation inputs that DiSciPLE programs receive; the paper does not report such an input-fair comparison.","The critic's per-stratum feedback suggests a general principle for LLM-guided search: supplying the model with subgroup error signals instead of a single global fitness score may accelerate synthesis in any task with meaningful subgroups.","Extending simplification beyond the final feature list could yield even more compact programs; the paper notes that only the last layer is optimized and that a fully differentiable pipeline did not help, so this remains a conjecture rather than a demonstrated result."],"forward_implications":["Population density can be estimated from satellite imagery with lower error using a readable program than using a large ResNet, both on held-out and out-of-distribution regions.","A program for aboveground biomass learned from the northeastern United States predicts better in Washington state than every baseline tested, suggesting the discovered formulas capture mechanisms that transfer geographically.","The method stays accurate when training data shrinks: with only a few hundred observations, DiSciPLE's error degrades much less than deep models, so it can be used where labels are scarce.","Domain experts can read the discovered program, see which semantic concepts matter (for example, highway and residential building for population density), and edit or extend it, which enables a human-in-the-loop scientific workflow."],"supporting_citations":[{"why":"Supplies the Llama-3-8B-instruct LLM that DiSciPLE uses to generate, cross over, mutate, and critique programs.","marker":"[9]"},{"why":"Provides GRAFT, the open-vocabulary satellite-image segmenter that acts as the main visual primitive the evolved programs call.","marker":"[26]"},{"why":"Establishes the population-density estimation task from satellite imagery and the log-L2 error metric used as the fitness function.","marker":"[30]"},{"why":"Provides the SustainBench dataset of coordinates to wealth asset index values used for the poverty estimation task.","marker":"[43]"},{"why":"Provides the GEDI lidar observations of aboveground biomass used for the biomass task.","marker":"[8]"},{"why":"Supplies the OpenStreetMaps land-use masks used as input in the population-density experiment to disentangle segmentation from discovery.","marker":"[40]"},{"why":"Prior work evolving interpretable visual classifiers with LLMs, which DiSciPLE extends from bag-of-words bottlenecks to full programs.","marker":"[4]"},{"why":"Zero-shot program synthesis baseline, ViperGPT, that DiSciPLE compares against to show LLM prior knowledge alone is not enough on scientific tasks.","marker":"[37]"}],"fun_headline_variants":["LLM-evolved programs beat black-box nets on satellite data","Readable AI programs: LLM evolution wins on imagery","From pixels to formulas: LLM-guided evolution outperforms","Evolving readable programs: LLMs outdo opaque baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy comparisons assume the deep baselines saw the same auxiliary inputs (land-use masks for population density; temperature, precipitation, nightlight, and elevation for poverty and biomass) as the DiSciPLE programs, so the gap reflects the discovery method rather than an input advantage.","fun_headline_variants_meta":{"raw":{"variants":["LLM-evolved programs beat black-box nets on satellite data","Readable AI programs: LLM evolution wins on imagery","From pixels to formulas: LLM-guided evolution outperforms","Evolving readable programs: LLMs outdo opaque baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1312,"prompt_tokens":888,"completion_tokens":424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":504,"tokens_out":424,"duration_ms":4531,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:32:32.881742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the published tasks and give the deep baselines the exact same auxiliary inputs that DiSciPLE programs receive — OpenStreetMaps land-use masks for population density, and the temperature, precipitation, nightlight, and elevation features for poverty and biomass — then compare errors. If the gap shrinks to near zero, the central claim that evolutionary program search beats black-box models on these tasks is refuted.","supporting_citations":[{"cited_title":"Fine- grained population mapping from coarse census counts and open geodata","cited_arxiv_id":null,"evidence_quote":"Establishes the population-density estimation task from satellite imagery and the log-L2 error metric used as the fitness function."},{"cited_title":"The global ecosystem dynamics investigation: High-resolution laser ranging of the earth’s forests and topography","cited_arxiv_id":null,"evidence_quote":"Provides the GEDI lidar observations of aboveground biomass used for the biomass task."},{"cited_title":"Openstreetmap: Challenges and oppor- tunities in machine learning and remote sensing","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenStreetMaps land-use masks used as input in the population-density experiment to disentangle segmentation from discovery."},{"cited_title":"Evolving interpretable visual classifiers with large language models","cited_arxiv_id":null,"evidence_quote":"Prior work evolving interpretable visual classifiers with LLMs, which DiSciPLE extends from bag-of-words bottlenecks to full programs."},{"cited_title":"Vipergpt: Visual inference via python execution for reasoning","cited_arxiv_id":null,"evidence_quote":"Zero-shot program synthesis baseline, ViperGPT, that DiSciPLE compares against to show LLM prior knowledge alone is not enough on scientific tasks."}],"review_version":1}