{"id":"5f2ccb9c-b777-4003-812a-7d5b06362ab5","arxiv_id":"2411.11206","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM is prompted with correct ARC solutions in a redesigned DSL to produce comments, refactored functions, high-level tactics, and step descriptions, and the resulting data is released openly.","lead":"This paper uses a cheap commercial LLM (Gemini Flash) to comment, refactor, and explain known-good code solutions for ARC puzzles, then releases the resulting dataset. It does not solve ARC, but offers a new open resource for training or prompting local models that are eligible for the ARC Prize.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of downstream usability is unsupported: the paper demonstrates extraction of annotations, not that local LLMs can use them; no RAG, fine-tuning, or prompt-injection experiment appears anywhere in the text.","rationale":"The paper is an honest, open-source resource contribution: the DSL re-engineering is reproducible, all 400 training solutions are verified, and the dataset with generation code is released. These count for credit. However, the title and abstract claim 'Capturing' abstraction 'in a form that could be used'—that is a predictive empirical claim about downstream transfer. I could not find a single experiment in the text where a local LLM attempts an ARC task with and without the extracted artifacts. The reader's weakest_assumption points to the human-legibility proxy in Section 3 as load-bearing; I see that as one plausible failure mode, but the broader, more direct weakness is the missing transfer evaluation, which would subsume it. Therefore I recommend no change to the reader's CONDITIONAL verdict: the resource should be released, but the claimed downstream usefulness must be demonstrated before acceptance as a verified capability. The proposed RAG experiment is feasible with the released code and data and would settle the question.","tokens_in":12217,"tokens_out":4553,"duration_ms":41038,"concrete_test":"Run a retrieval-augmented generation experiment under ARC Prize constraints. Use the released jina-embeddings-v2-base-code model to embed (a) each of the 377 tasks' Part 2 refactored functions and Part 3 tactics, and (b) each ARC training task's input/output grid pair (serialized as tokens). For each held-out task, retrieve the top-k most similar artifact sets from the 377 training tasks. Prompt a local LLM within Kaggle GPU limits (e.g., Qwen2.5-7B-Instruct) to produce a candidate arc-dsl-llm solution using those retrieved artifacts, execute the candidates on the task's input grids, and measure exact-match accuracy on the output grids. Compare against (i) a no-retrieval baseline prompt and (ii) a retrieval condition that supplies raw arc-dsl-llm solutions without the LLM-generated comments/tactics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that 'Sparks of Abstraction' can be extracted 'in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize.' The word 'could' is doing real work: the paper never shows that any local LLM actually does better on ARC when given these artifacts. Section 5.1 states 'the overall system envisioned is still being brought on-line,' conceding no integrated demonstration. The only checks in Section 3.5 are syntactic: outputs have the required number of Parts and are valid Python/YAML. Section 4.3 uses clustering of tactic embeddings as evidence that high-level tactics were 'surfaced,' but clustering shows similarity among texts, not that a local LM can execute them. The one qualitative example in Figure 7 even contains a factual error ('no holes exist within objects'), which the paper acknowledges. If a local model cannot turn these comments, refactorings, or tactics into correct DSL solutions on held-out tasks, the dataset is a fluent annotation corpus but the claimed transferable 'sparks' do not exist. The Section 3 assumption that human-legibility proxies LLM-legibility is one step in the pipeline, but the decisive gap is that downstream utility is asserted, not measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces arc-dsl-llm, a renamed and type-augmented version of Hodel's arc-dsl for ARC tasks, and uses Gemini-Flash-002 to generate four kinds of annotations for 377 ARC training tasks: commented code, refactored code, high-level tactics, and generic solution steps. The artifacts are released as an open-source dataset together with the generation code. The central claim is that these annotations capture 'Sparks of Abstraction' in a form that could later be used by local LLMs in the ARC Prize environment via RAG or fine-tuning. The paper presents one worked example, a clustering visualization of tactic embeddings, and structural sanity checks; it does not report any downstream experiment with local LLMs.","tokens_in":12415,"tokens_out":5463,"duration_ms":51774,"significance":"If the extracted annotations prove reusable, the open-source DSL and dataset could provide a concrete bridge between commercial LLM reasoning and the resource-constrained local models allowed in the ARC Prize. The reproducible pipeline — a pinned Gemini model, open generation code, and validation of all 400 training solutions against their test examples — is a clear strength. Equally clear is the honest acknowledgment of limitations. However, the paper's central evidence is currently qualitative: one illustrative task, an unquantified clustering plot, and structural rather than semantic checks. The downstream utility that motivates the abstract is asserted, not measured. As a resource and position paper, the work is promising; as a demonstration of transferable 'sparks,' it is incomplete.","major_comments":[{"comment":"The central claim that the extracted 'sparks' are 'in a form that could be used in downstream tasks' is not supported by any experimental evidence in the manuscript. Section 5.1 explicitly states that 'the overall system envisioned is still being brought on-line,' and no RAG, prompt-injection, or fine-tuning experiment with a local LM is presented anywhere. Without at least a small controlled study — for example, using Part 4 steps as in-context prompts for a local LLM on held-out training tasks, or fine-tuning on Part 1 comments and measuring solution accuracy — the abstract overclaims: the paper demonstrates extraction of annotations, not that the annotations are usable. This is load-bearing for the paper's motivation.","section":"Abstract; Section 5.1"},{"comment":"The refactored code produced in Part 2 is not checked for semantic correctness. The sanity checks described (sub-functions cannot call each other; main function variable names are consistent) are purely syntactic and structural. Since the paper already has executable DSL code and input grids, the natural verification is to run each generated solver_virtual_chunked on the training examples and compare against the ground-truth output. The manuscript reports no such execution check. If a substantial fraction of refactorings are incorrect, the dataset cannot support downstream RAG or fine-tuning as claimed. Please add per-task execution results and release the evaluation script.","section":"Section 4.2"},{"comment":"The design rationale for arc-dsl-llm rests on the assumption that 'human-legibility can be used as a simple proxy for LLM-legibility.' This assumption is never tested. The paper motivates extensive function renaming (85 functions) and type-hinting changes by appeal to LLM familiarity, but no experiment shows that Gemini or any local LM understands arc-dsl-llm better than the original arc-dsl. A minimal validation would be to ask an LLM to perform a downstream comprehension task (e.g., explain each line, or predict intermediate values) on paired code samples from both DSLs and compare accuracy. Without this, the 'LLM-legible' property is asserted rather than established, and the foundation of the pipeline is unverified.","section":"Section 3, first paragraph; Section 3.2"},{"comment":"The clustering analysis is presented as evidence that the LLM has 'surfaced high-level tactics of the type required for the Self-Discover framework.' However, the figure shows only that sentence embeddings of tactic strings can be grouped by UMAP/HDBSCAN. There are no cluster-quality metrics (e.g., silhouette score), no manual inspection protocol, no task-distribution analysis, and no demonstration that a downstream system can select a relevant tactic from a cluster and execute it. The interpretive leap from 'similar texts form clusters' to 'useful, generalizable tactics exist' is not justified. Please provide a quantitative and/or human-validated evaluation of tactic quality.","section":"Section 4.3, Figure 6"}],"minor_comments":[{"comment":"The text says 'valid Python/Y AML as required'; this appears to be a typo for 'valid Python/YAML'.","section":"Section 3.5"},{"comment":"The sentence 'The ability of LLMs to reason was explore in Lee et al.' should read 'was explored in Lee et al.'.","section":"Section 2.4"},{"comment":"In the sentence 'One interesting observation is thatfind_internal_objects has been factored out,' a space is missing between 'that' and 'find_internal_objects'.","section":"Section 4.2"},{"comment":"The figure has no axis labels, no legend, and no cluster-size distribution; as printed it is not interpretable standalone, and the claim about 'the top 30 clusters' cannot be verified from the figure alone.","section":"Figure 6"},{"comment":"The phrase 'over-cautious in generalising' is not operationalized; consider giving examples or a heuristic for measuring caution, or remove the qualitative characterization.","section":"Section 4.4"},{"comment":"The remapping of COLOR constants to the range [1009..1019] and the statement that 'if an integer is > 20, remap as a COLOR constant' are confusing; clarify whether this applies only to LLM-facing rendering and how it interacts with the earlier claim that the COLOR-constant value assumptions were fixed.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about its exploratory nature, but the gap between the extraction results and the downstream-use claim is substantial. If the journal is open to resource/position papers, the open-source contributions may be sufficient after revision; if the expectation is a validated method, at least one downstream experiment is needed. I would support conditional acceptance after a careful revision addressing the missing correctness checks and the untested legibility assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a data/resource paper, not a method paper, and it should be read that way. The concrete deliverables are real: an LLM-legible rewrite of Hodel's arc-dsl, re-verified against all 400 ARC training tasks, plus a Gemini-generated corpus of comments, refactored functions, tactics, and solution steps, all released open-source. The re-engineering is thoughtful (renaming 85 DSL functions, separating color constants from integers, type-checked), and the author is upfront that the integrated ARC solver is not yet running.\n\nThe soft spot is exactly where the abstract oversells. 'Sparks of Abstraction can be extracted... in a form that could be used' – the 'could' is doing all the work. There is no RAG, fine-tuning, or prompt-injection experiment anywhere in the paper. The sanity checks are structural; the one worked example includes an acknowledged factual error; and the tactic clustering only shows that similar texts embed similarly, not that a local model can execute them. The human-legibility proxy for LLM-legibility is also untested. So the load-bearing claim of downstream value is unmeasured. To the author's credit, this is stated plainly in Section 5.1, but the abstract's framing is stronger than the evidence.\n\nI don't think this is a fatal flaw. As a dataset release, the paper is honest and the artifacts are reusable. The right fix is to either soften the abstract to 'we provide this resource and conjecture it will help' or add one small experiment: e.g., take a local LLM, give it the commented DSL or tactics for a held-out task, and see if program search improves. That would make the claim credible.\n\nWho is this for? Anyone building ARC solvers with local models or trying to distill abstraction from commercial LLMs. It's a legitimate resource, and a serious referee should get it – but the referee should demand a rewritten abstract and, ideally, a pilot downstream evaluation. Worth bringing to a reading group if you work on ARC or LLM-guided program synthesis.","headline":"A useful dataset and DSL release whose abstract overclaims: the extraction pipeline is real, but downstream usefulness is asserted, not measured.","tokens_in":12956,"tokens_out":3105,"would_cite":true,"duration_ms":25592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that giving an LLM known-good ARC solutions in a plainer DSL lets it produce reusable comments, refactored code, tactics, and solution steps that can be captured for local models.","keywords":["ARC Challenge","large language models","domain-specific language","code comprehension","program refactoring","Abstraction and Reasoning Corpus","retrieval-augmented generation","LLM-legible DSL"],"falsifier":"Take a local model that has never seen the released dataset, fine-tune it on the Gemini-generated comments, refactors, tactics, and steps, then test it on a held-out set of ARC tasks; if performance does not improve over an equivalent model trained on the raw DSL code or on nothing, the claim that these abstractions carry downstream value fails.","tokens_in":11987,"feed_emoji":"🧩","tokens_out":3728,"duration_ms":34377,"temperature":0.7,"pith_summary":"This paper argues that an LLM's understanding of ARC problems can be harvested if it is given complete, known-good solution code rather than just input and output grids. The author re-engineered an existing ARC domain-specific language to be more readable, fed the rewritten solutions for 400 training tasks to Gemini-Flash, and asked it to comment the code, refactor it into reusable functions, name high-level tactics, and describe generic solution steps. The claim is that the resulting dataset captures transferable 'sparks of abstraction' that a resource-limited local model — the kind allowed inside the ARC Prize environment — could consume via fine-tuning or retrieval. The evidence shown is qualitative: the example extracts reveal the LLM recognizing the intent behind convoluted code, and clustering of the tactics hints at reusable patterns. The paper's stated goal is to bring a strong commercial LLM's reasoning into a restricted environment without the commercial model being present at inference time.","feed_headline":"Feeding an LLM the right code captures reusable ARC abstractions","feed_subtitle":"A rewritten DSL plus Gemini-generated comments, refactors, and tactics could let small local models enter the ARC Prize.","key_machinery":"The central object is arc-dsl-llm, a re-engineered, type-safe version of the original arc-dsl with 85 function renames and corrected color constants, designed to be human-legible and therefore, the author assumes, LLM-legible. The machinery is the prompting pipeline built around it: known-good solutions written in arc-dsl-llm, a reformulated Core Knowledge rubric, DSL documentation, and input/output grids are fed to Gemini-Flash, which outputs the four parts (comments, refactored functions, tactics, solution steps) that become the dataset. The re-engineered DSL is what carries the argument, because it supposedly lets the LLM see the intentionality behind each line of code.","core_discovery":"The central discovery is that 'Sparks of Abstraction' — the LLM's grasp of the big picture of an ARC task — can be extracted when the LLM is shown complete code solutions expressed in an LLM-legible DSL. Given such code, Gemini-Flash produces four artifacts: line-by-line comments that name the goal and core knowledge behind each block; refactored code factored into independent, reusable sub-functions; high-level tactics phrased generically; and step-by-step task descriptions tied to both tactics and core-knowledge primitives. The paper argues these outputs carry abstract understanding that can be captured offline and later injected into local models eligible for the ARC Prize, either through fine-tuning or a simple retrieval-augmented generation system.","pith_inferences":["The human-legibility proxy is testable: one could compare extraction quality on the original arc-dsl versus arc-dsl-llm for the same tasks; if quality is equal, the expensive re-engineering was not the cause of the observed understanding.","If the dataset proves useful, the approach likely generalizes to any domain that has known-good programs and a legible DSL, not just ARC grids.","The real test of the pipeline — entering the ARC Prize competition — is still open; the paper's value hinges on whether the captured abstractions actually improve a local model's performance on held-out tasks.","The tactics clustering suggests a potentially small vocabulary of reusable ARC move types; an extension would be to measure how often tactics shared across tasks actually transfer to unseen tasks."],"forward_implications":["The released dataset of 377 sanity-checked tasks can be used to fine-tune local LLMs inside the ARC Prize's compute-limited container, giving them commented code examples and high-level goals.","A simple retrieval-augmented generation system can inject the extracted tactics and refactored functions into live prompts on test-set problems, without calling a commercial API at inference time.","The refactored sub-functions, being independent and consistently commented, could seed code denoising, program search, or library-learning approaches inside the container.","The comparison between the original arc-dsl and arc-dsl-llm code suggests that DSL naming and type clarity are practical bottlenecks for LLM program understanding.","The clustering result across the dataset indicates that the LLM is surfacing a finite set of reusable high-level tactics that a self-discovering system could select from."],"supporting_citations":[{"why":"Defines the ARC dataset and the Core Knowledge rubric that the whole extraction pipeline is built around.","marker":"Chollet (2019)"},{"why":"Supplied the 400 ARC training-set solutions in arc-dsl that this work rewrites into arc-dsl-llm and feeds to the LLM.","marker":"Hodel (2023)"},{"why":"Provides the Self-Discover framework that the high-level tactics (Part 3) are intended to feed.","marker":"Zhou et al. (2024)"},{"why":"The LARC dataset is explored as an alternative source of human descriptions and found lacking, motivating automated extraction from code.","marker":"Acquaviva et al. (2023)"},{"why":"DreamCoder motivates the refactoring/chunking goal that Part 2 attempts to produce cheaply outside the container.","marker":"Ellis et al. (2020)"},{"why":"Provides evidence that LLMs struggle to generate new ARC code from scratch, supporting the decision to start from known-good code.","marker":"Greenblatt (2024)"},{"why":"Found that LLMs lack logical coherence and compositionality in ARC reasoning, supporting the strategy of extracting higher-level thinking from human-written code rather than generation.","marker":"Lee et al. (2024)"}],"fun_headline_variants":["LLM extracts reusable ARC abstractions from given code","Sparks of abstraction captured by showing LLM the code","Feeding LLM code yields abstract ARC tactics for local models","Code-to-abstraction: How LLMs grasp ARC tasks","ARC abstractions from LLM code comments and refactors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline rests on the assumption that making the DSL easier for a person to read also makes it easier for an LLM to genuinely understand; if the LLM is only producing fluent paraphrases without transferable comprehension, the captured sparks have no downstream value.","fun_headline_variants_meta":{"raw":{"variants":["LLM extracts reusable ARC abstractions from given code","Sparks of abstraction captured by showing LLM the code","Feeding LLM code yields abstract ARC tactics for local models","Code-to-abstraction: How LLMs grasp ARC tasks","ARC abstractions from LLM code comments and refactors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1219,"prompt_tokens":929,"completion_tokens":290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":207}},"tokens_in":545,"tokens_out":290,"duration_ms":3270,"temperature":1.0,"reasoning_tokens":207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:47:11.713157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a local model that has never seen the released dataset, fine-tune it on the Gemini-generated comments, refactors, tactics, and steps, then test it on a held-out set of ARC tasks; if performance does not improve over an equivalent model trained on the raw DSL code or on nothing, the claim that these abstractions carry downstream value fails.","supporting_citations":[],"review_version":1}