{"id":"34ca1ffd-e1f0-4f43-8496-e3f270b642ad","arxiv_id":"2411.18158","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A knowledge-graph-based abductive symbolic solver predicts ARC output grid size and color set with reported accuracies of 90.5% and 74.75%, but without trivial baselines or a disclosed evaluation split.","lead":"This paper presents a symbolic solver for ARC visual reasoning puzzles that converts each puzzle into a knowledge graph, extracts features repeated across the examples, and searches over hand-defined transformation rules to predict the output grid's height, width, and colors. The authors report that the knowledge graph raises accuracy on these partial targets from about 80% to 91% for size and from 40% to 75% for colors on 400 tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The KG vs no-KG comparison is confounded by simultaneous changes to object extraction and the Specifier, so the accuracy gap does not isolate the knowledge graph's contribution.","rationale":"After reading the paper, the central claim is H1 as stated in Section 4.2. For that claim to hold, the comparison between 'with KG' and 'without KG' must isolate the knowledge graph as the causal factor. It does not: the without-KG condition also omits the Specifier and the object-level node extraction described in Sections 3.1-3.2, and applies transformations directly to the raw grid. This confound is more directly fatal to H1 than the task-selection issue the Reader raised, because even if the 400 tasks are fair and out-of-sample, the reported gain still cannot be attributed to the graph per se. The paper's own Figure 2 and Figure 8 show two pipelines that differ in multiple components. I credit the authors for a clear internal comparison and for not overclaiming content prediction; however, the absence of code, task list, or trivial baselines further weakens the evidential value. The proposed ablation would settle whether the graph structure itself adds value. If the no-edges variant performs the same, then the KG is an implementation detail rather than the mechanism, and the title's emphasis on abductive symbolic solving via KG would need revision. This concern does not require rejecting the paper; it requires a revised experiment and is consistent with the Reader's CONDITIONAL verdict.","tokens_in":10879,"tokens_out":5760,"duration_ms":48154,"concrete_test":"Run the following three conditions on the same 400 tasks: (a) the paper's full KG+Specifier solver; (b) a 'no-edges' variant that keeps identical Pnode/Onode/Gnode node construction and Specifier candidate selection but removes all edges (or replaces graph traversal with flat feature matching), so only the graph's edge structure is ablated; (c) the paper's current no-KG baseline. Also compute a trivial baseline that predicts output height, width, and color set equal to the input's values. If (a) ≈ (b) > (c), the gain is due to object-centric constraints and not the KG; if (a) > (b), the graph edges contribute. The trivial baseline then calibrates whether the absolute accuracies are meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 describes two conditions that differ in more than the knowledge graph. The 'without KG' condition not only skips graph construction; it also removes the Specifier and applies Transformation DSLs directly to raw grid elements. The 'with KG' condition constructs Pnode/Onode/Gnode/Vnode layers (Section 3.1.2), extracts objects, and uses Specifier to select candidate nodes from the graph (Section 3.2). Consequently, the accuracy differences in Figure 9 and Table 2 (e.g., HWC 66.5% vs. 32.3%) could arise from object-centric representation and candidate pruning rather than from the graph's edges or structure. This is not a minor implementation detail: Section 3.3.2 states that Specifier's constraint extraction is inseparable from the KG traversal, so the no-KG baseline is denied the very mechanism being tested. H1 ('knowledge graphs effectively enhance performance') is therefore not actually tested by this comparison. The absolute numbers also lack calibration: a trivial baseline that copies input height, width, and color set to the output may already achieve high accuracy on ARC metadata, which would change the interpretation of 'nearly perfect' (90.5% HW) and the 74.75% color accuracy. The absence of code or the full 400-task list prevents checking whether these confounds or in-sample DSL design occurred. This concern is load-bearing because the paper's central empirical claim rests on this single comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an abductive symbolic solver for the Abstraction and Reasoning Corpus (ARC). The pipeline converts each input-output example into a four-layer knowledge graph (ARCKG) using Property DSLs, extracts repeated constraints with a module called Specifier, and searches for combinations of Transformation DSLs with a Synthesizer. The reported experiments compare a KG-based setup against a no-KG setup on 400 ARC tasks, predicting grid height, width, and color set, and also compare two Synthesizer sizes (10 vs. 5 DSLs). The paper claims that knowledge graphs effectively enhance performance (H1) and that larger Transformation DSL inventories improve accuracy (H2).","tokens_in":11205,"tokens_out":3370,"duration_ms":32001,"significance":"If the central claims are validated, the framework would be a useful interpretable and neuro-symbolic contribution to ARC research, with a concrete graph-based representation, an explicit constraint-extraction mechanism, and a falsifiable evaluation on metadata prediction. The paper is clearly written in terms of its pipeline components, and the choice to predict height, width, and color set is a reasonable first step. However, the current experimental design does not isolate the effect of the knowledge graph, lacks trivial baselines and statistical calibration, and leaves open the possibility that the DSL inventory was tailored to the evaluation tasks. The significance of the results is therefore conditional on substantial additional evidence.","major_comments":[{"comment":"The KG vs. no-KG comparison confounds several changes simultaneously. The 'without KG' condition omits graph construction, object extraction, and the Specifier, applying Transformation DSLs directly to raw grid elements. The 'with KG' condition uses Pnode/Onode/Gnode/Vnode layers and Specifier-based candidate selection. The accuracy gap in Figure 9 (e.g., HWC 66.5% vs. 32.3%) could therefore be caused by object-centric representation or candidate pruning rather than by graph edges. Since Section 3.3.2 describes Specifier's constraint extraction as inseparable from KG traversal, the no-KG baseline is not a valid ablation of the knowledge graph. The authors should add a controlled condition that keeps the Specifier and object extraction but removes graph edges, or otherwise vary only the KG component.","section":"§4.1, Figure 8, Section 3.2"},{"comment":"The selection of the 400 ARC tasks is underspecified. The text says only that tasks were selected 'ensuring a diverse range of grid sizes and color sets,' with no list, random seed, or protocol. It is also not stated whether these tasks were used when designing the 22 Property DSLs and the Transformation DSLs. Given that the DSL inventory in Figure 3 includes functions that directly return evaluation targets (get_height, get_width, get_number_of_colorset), the high accuracy may be partly attributable to in-sample DSL design. The authors should release the task list and the DSL design process, or otherwise demonstrate that the DSLs were not fitted to these 400 tasks.","section":"§4.1"},{"comment":"No trivial baseline or statistical significance test is reported. For the chosen metadata targets, a baseline that copies the input grid's height, width, and color set to the output could already achieve substantial accuracy, especially for tasks with unchanged dimensions. Without such a baseline and without error bars or a significance test, the statement that 'nearly perfect accuracy' supports H1 is uncalibrated. The authors should add at least an input-copying baseline and report confidence intervals or paired tests across tasks.","section":"§4.2, Table 2"},{"comment":"The comparison between Synthesizer-10 and Synthesizer-5 does not cleanly support H2 because the exact membership of TS10 and TS5 is not given. Figure 4 only says TS5 is a subset of TS10; it does not specify which DSLs are included, nor how search depth and computational budget are controlled. If the additional DSLs include directly target-relevant functions, the performance gain is expected by construction. The authors should list the DSLs in each set and report search effort or runtime.","section":"§4.2, Table 2, Figure 4"}],"minor_comments":[{"comment":"Line 13 uses the variable name 'node_list_pair', but the algorithm constructs a single 'node_list'; the naming should be aligned for readability.","section":"Algorithm 1"},{"comment":"The DSL name is written both as 'get_number_of_colorset' and 'get_number_of_colors'; please standardize the name and ensure the figure and text match.","section":"Figure 3 and Section 3.3.2"},{"comment":"The caption introduces 'Transformation Selection 10 (TS10)' and 'TS5' but does not define them; connect these labels explicitly to Synthesizer-10 and Synthesizer-5 used in Section 4.2.","section":"Figure 4"},{"comment":"The sentence describing the 400-task selection would benefit from concrete information about the distribution of grid sizes, color counts, and task sources, since the current description is too vague to assess representativeness.","section":"Section 4.1"},{"comment":"The phrase 'According to Pan Lu et al.' is informal for a journal report; please provide a more precise statement of the relevant finding and its context.","section":"Section 1 and Reference [1]"}],"recommendation":"major_revision","confidential_remarks":"The central empirical comparison is confounded, and the DSL inventory concern is substantial because the evaluation targets appear as DSL functions. The paper would need a controlled ablation, trivial baselines, significance testing, and disclosure of the task/DSL relationship before I could support acceptance. In its current form, the contribution is closer to a workshop-level system description than a fully supported journal claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful framework paper, not yet a convincing empirical one. The four-layer ARCKG plus the Specifier is a genuine combination I haven't seen in the cited work, and the authors are honest that they are only predicting height, width, and color set, not grid content. That is a legitimate first step, and the abductive framing fits the few-shot nature of ARC.\n\nWhat the paper does well: the pipeline is transparent, the DSL taxonomy is clearly laid out, and the Synthesizer-size ablation (10 vs 5) moves in the expected direction. The internal comparison is directionally consistent with the idea that object-centric structure helps, and the related work is cited fairly.\n\nBut the central empirical claim is not actually tested. The \"without KG\" condition also drops the Specifier and object extraction, so the accuracy gap in Figure 9 cannot be attributed to the knowledge graph per se. That is a load-bearing confound. On top of that, there is no trivial baseline. For ARC metadata, copying input height, width, and color set might already achieve high accuracy, and with 400 tasks selected by the authors without a stated split, we have no way to know whether the DSLs were tailored to these exact tasks. The DSL inventory includes get_height, get_width, get_number_of_colorset—the evaluation targets—so at least part of the accuracy is baked into the language. No code or task list is provided, so none of this can be checked.\n\nThe math and citation pattern are fine; the framework is coherent. The soft spots are in the evaluation, not the conception. The paper would be much stronger with a proper copy-input-metadata baseline, a no-KG condition that keeps object extraction and Specifier-style filtering while removing only the graph structure, and a public release of the 400-task split.\n\nWho this is for: ARC solver researchers and people designing interpretable program-synthesis pipelines. It does not address the hard part of ARC, as the authors implicitly acknowledge by ignoring grid content.\n\nI would send this to peer review. The ideas deserve referee time, but the empirical claims need major revision before publication. If the confound is fixed and baselines are added, this could become a solid contribution.","headline":"A plausible symbolic pipeline for ARC metadata prediction, but the KG-vs-no-KG comparison is confounded and the DSL inventory makes the reported accuracies look partly built-in rather than measured.","tokens_in":11710,"tokens_out":1947,"would_cite":false,"duration_ms":19493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Representing ARC puzzles as knowledge graphs and extracting repeated 'core knowledge' lets a symbolic solver predict output grid height, width, and color set with 91.5%, 91.25%, and 74.75% accuracy on 400 tasks.","keywords":["Abstraction and Reasoning Corpus","knowledge graph","abductive reasoning","domain-specific language","program synthesis","symbolic reasoning","visual reasoning","core knowledge extraction"],"falsifier":"Re-run the pipeline on the full ARC evaluation set (400 held-out tasks) and compare against a trivial baseline that returns the input grid's height, width, and color set as the prediction. If the baseline matches or exceeds the reported 91.5%/91.25%/74.75% accuracies, or if the KG-based solver fails to beat the non-KG baseline on this held-out set, the central claim that knowledge graphs enhance performance would be falsified.","tokens_in":10686,"feed_emoji":"🧩","tokens_out":6920,"duration_ms":51900,"temperature":0.7,"pith_summary":"The paper argues that solving Abstraction and Reasoning Corpus (ARC) puzzles — visual reasoning tasks that require inferring a transformation from a few input-output pairs — benefits from representing each pair as a knowledge graph before searching for a program. The authors build a four-layer graph (pixels, objects, grids, and input-output pairs) and extract 'core knowledge' from repeated features across examples, which then constrains a DSL-based synthesizer that searches over transformation primitives. On 400 ARC tasks, this knowledge-graph pipeline predicts output grid height, width, and color set at 91.5%, 91.25%, and 74.75% accuracy, compared to 80.5%, 80.5%, and 40.5% without the graph. The paper also reports that a larger set of transformation DSLs more than triples accuracy on the combined height-width-color target. If correct, the work shows that symbolic, interpretable reasoning can outperform the plain grid-transition approach on ARC metadata and that abductive core-knowledge extraction is a key ingredient.","feed_headline":"Knowledge graphs lift ARC solver's color-set accuracy to 74.75%","feed_subtitle":"Adding a knowledge graph narrows the symbolic search space, lifting grid height and width past 90% and color-set prediction to 74.75%.","key_machinery":"The load-bearing mechanism is the ARC Knowledge Graph (ARCKG) together with the Specifier–Synthesizer pair. ARCKG is a four-layer graph in which pixels (Pnode), objects (Onode), grids (Gnode), and input-output pairs (Vnode) are nodes, and edges are drawn by Property DSLs that capture relations such as same color, containment, and adjacency. The Specifier inspects all example graphs, counts features that appear in every pair, and outputs 'core knowledge' — the minimal set of candidate objects and their properties that consistently appear. These constraints are then fed to the Synthesizer, which performs a brute-force search over a hand-defined set of Transformation DSLs (e.g., get_height, get_width, get_number_of_colorset, linear transformations) to find a path from an input node to the output node. The core-knowledge constraints prune the search tree, making the synthesis tractable and interpretable.","core_discovery":"The central claim is that converting ARC tasks into knowledge graphs and extracting core knowledge via a Specifier module materially improves a symbolic solver's ability to predict the output grid's height, width, and color set. Formally, the paper states that with the knowledge graph the solver achieves 'nearly perfect accuracy' for height, width, and their combination, with measured accuracies of 91.5%, 91.25%, and 90.5% on a 400-task selection; color-set accuracy reaches 74.75%, and the combined height-width-color accuracy 66.5%. Without the knowledge graph, the same synthesizer drops to 80.5%, 80.5%, 79.3%, 40.5%, and 32.3% respectively. The authors take this as support for their hypothesis H1 that knowledge graphs effectively encapsulate symbolic knowledge. They also report that Synthesizer-10, using ten transformation DSLs, outperforms Synthesizer-5 using five, with combined HWC accuracy rising from 21% to 66.5%, supporting H2 that DSL count is positively correlated with performance.","pith_inferences":["The color-set accuracy gap (74.75%) compared to height/width (>91%) suggests that the current Property DSLs capture spatial structure better than color semantics; expanding the graph with color-relation edges (e.g., color adjacency or frequency) might close that gap.","The 400-task selection may be biased toward tasks whose solutions are expressible in the hand-defined DSLs; a truly neutral test would measure performance on tasks specifically generated to be out-of-distribution for the DSL inventory.","The core-knowledge extraction is essentially a frequency filter — features appearing in all examples are kept. A more robust abduction, such as selecting the minimal feature set that distinguishes the target object, might generalize better to tasks with noisy or contradictory examples.","The framework currently predicts only grid metadata (height, width, colors). The same knowledge-graph constraints could be extended to synthesize full grid content by adding transformation DSLs for pixel-level operations, which the paper itself anticipates."],"forward_implications":["If knowledge-graph extraction is genuinely the cause of the accuracy jump, then any ARC solver that converts grids into relational structures before synthesis should see similar gains, not just this particular DSL implementation.","The reported scaling with DSL count suggests that investing in a richer library of transformation primitives is a direct path to higher accuracy on ARC, potentially extending from grid metadata to full grid content prediction.","The interpretable (constraints, path) hypotheses generated by the framework could be used to produce human-readable explanations of ARC solutions, not just answers.","The framework's two-phase abductive process — extract repeated features, then search under those constraints — could be adapted to other few-shot visual reasoning benchmarks."],"supporting_citations":[{"why":"Supplies the ARC dataset and the four core knowledge priors that motivate the DSL design.","marker":"[2]"},{"why":"Provides the DSL taxonomy (property and transformation DSLs) that the framework adapts for graph construction and synthesis.","marker":"[7]"},{"why":"Supplies the abductive reasoning setup (observation and explanation) that the Specifier's constraint extraction follows.","marker":"[11]"},{"why":"Shows graph abstractions for ARC; the four-layer ARCKG extends this to multi-layer features.","marker":"[15]"},{"why":"Demonstrates a DSL-based solver that the paper's approach builds on for synthesis.","marker":"[6]"}],"fun_headline_variants":["Knowledge graphs push ARC solver's grid-shape accuracy past 90%","ARC solver with knowledge graph: 91.5% height, 74.75% color set","Abductive symbolic solver gains from knowledge graphs on ARC","Knowledge graph narrows ARC solution space, lifting accuracy","Knowledge graph lifts ARC height prediction to 91.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation set of 400 ARC tasks and the hand-crafted DSL inventory are independent, so that the DSLs were not designed with these tasks' solutions in mind and no simple baseline that copies input size and colors would reach the reported accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graphs push ARC solver's grid-shape accuracy past 90%","ARC solver with knowledge graph: 91.5% height, 74.75% color set","Abductive symbolic solver gains from knowledge graphs on ARC","Knowledge graph narrows ARC solution space, lifting accuracy","Knowledge graph lifts ARC height prediction to 91.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000685,"raw_usage":{"total_tokens":3097,"prompt_tokens":926,"completion_tokens":2171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":542,"tokens_out":2171,"duration_ms":14511,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:27:11.518186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the full ARC evaluation set (400 held-out tasks) and compare against a trivial baseline that returns the input grid's height, width, and color set as the prediction. If the baseline matches or exceeds the reported 91.5%/91.25%/74.75% accuracies, or if the KG-based solver fails to beat the non-KG baseline on this held-out set, the central claim that knowledge graphs enhance performance would be falsified.","supporting_citations":[{"cited_title":"Hodel, arc-dsl, https://github.com/michaelhodel/arc-dsl, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the DSL taxonomy (property and transformation DSLs) that the framework adapts for graph construction and synthesis."},{"cited_title":"Visual Abductive Reasoning","cited_arxiv_id":"2203.14040","evidence_quote":"Supplies the abductive reasoning setup (observation and explanation) that the Specifier's constraint extraction follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates a DSL-based solver that the paper's approach builds on for synthesis."}],"review_version":1}