{"id":"2bf040fc-1cad-41e1-b700-4648a218c8b9","arxiv_id":"2505.07859","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An open-source LLM pipeline with augmentation-aware training, probability-thresholded depth-first search, and product-of-experts scoring reaches 71.6% on the public ARC-AGI evaluation set.","lead":"A method combining test-time training, varied data augmentations, and a product-of-experts score lets an 8B open-source language model solve 71.6% of public ARC-AGI reasoning tasks, the best public result so far. It matters because it shows strong abstract reasoning is achievable with transparent, low-cost open models rather than massive proprietary systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline score is internally inconsistent: 286.5/400 'solved tasks' cannot arise from the paper's binary per-task definition; the metric or count must be clarified before the SOTA claim is interpretable.","rationale":"The paper's central claim is the 71.6% headline. For that claim to mean what it says, 286.5/400 must be a count of solved tasks under the definition in Section 2. It cannot be, so the headline is either mislabeled or measured with an unreported averaging/partial-credit rule. This is more immediate than the reader's weakest_assumption about probability ranking: the paper actually supplies direct evidence for ranking quality (Figure 4, Table 3, and the Sudoku experiment, where correct candidates are selected 100% of the time when sampled), so that assumption is at least partially tested. It is also more specific than the general in-sample-tuning concern: the ConceptARC result with identical hyperparameters partially addresses tuning worries, but no auxiliary experiment can interpret an internally inconsistent headline number. I recommend keeping the reader's CONDITIONAL verdict: the score may survive unchanged if the official scorer gives an integer near 287, but the paper should state the exact metric and release per-task predictions before the SOTA claim is accepted.","tokens_in":15633,"tokens_out":15912,"duration_ms":181633,"concrete_test":"Download the released prediction files and run them through the official ARC-AGI scoring script (arc-agi scorer / Kaggle metric) with the paper's Section 2 rule: a task is solved iff either of the two guesses exactly matches the expected output. Check whether the resulting sum over the 400 tasks is 286, 287, or another integer. If it is 286 or 287, the '286.5' is a typo or an unreported average, and the SOTA claim is likely intact; if the official binary count is materially lower and 286.5 comes from half-credit for second-guess successes, recompute Table 1 comparisons under a common metric. Also report the number of test outputs per task to rule out a per-output scoring convention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is the score 71.6% = 286.5/400 on the public ARC-AGI evaluation set. Section 2 defines a task as solved if 'the model produces the accurate output within a maximum of two attempts', which is a binary per-task outcome; a count over 400 tasks must therefore be an integer. The abstract reports 286.5 solved tasks, but the paper nowhere defines partial credit, averaging over randomized runs, or multi-test-input tasks. If the actual protocol is a correct-first-guess = 1, correct-second-guess = 0.5 convention (or an average over seeds), then '286.5/400 solved tasks' is not comparable to the binary per-task human average (60.2%, LeGris et al.) or to the baselines in Table 1, all of which are reported as per-task solved rates. The same number is the basis for the 'state-of-the-art among publicly available approaches' claim, so the anomaly is not cosmetic: it determines whether the central claim is measured on the same scale as its comparators. The ambiguity can be resolved from the released predictions without re-running inference.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for solving ARC-AGI tasks with an 8B open-source LLM. The model is initially fine-tuned on RE-ARC (generated from the official training tasks), then test-time fine-tuning is performed on each task's examples. Candidate solutions are generated by a depth-first search with a probability threshold T, and the best candidates are selected by a product-of-experts score computed across 16 augmented views of the problem. The main claim is 71.6% (286.5/400) two-guess accuracy on the public ARC-AGI evaluation set, which would be state of the art among open-source approaches and above the average human score. Additional results are reported on ConceptARC and a Sudoku benchmark.","tokens_in":15843,"tokens_out":13040,"duration_ms":125112,"significance":"If the headline result is reproducible, it is a substantial advance for open-source LLMs on ARC-AGI: an 8B model with roughly two cents of inference per task outperforms cited open-source baselines and the average human. The paper's strengths include a systematic ablation (Table 2), a comparison of sampling and selection strategies (Table 3), external validation on ConceptARC and Sudoku, and a simple theoretical justification for product-of-experts ranking. The authors state that code and the final model are publicly available. However, the central number's internal inconsistency and the evaluation-set tuning of the DFS threshold currently prevent the paper from being taken at face value.","major_comments":[{"comment":"The reported result \"71.6% (286.5/400 solved tasks)\" in the Abstract is incompatible with the binary per-task definition of \"solved\" given in Section 2, where a task is considered solved when the model produces the accurate output within a maximum of two attempts. A count over 400 tasks must be an integer under that definition, unless partial credit or averaging over randomized runs is used, but the paper nowhere defines such a scoring rule. This matters because Table 1 compares the 71.6% against baselines and the human average that are computed on a binary per-task basis. Please clarify the exact scoring protocol and, if a partial-credit or averaged metric is used, recompute the comparisons on the binary metric; alternatively, correct the count.","section":"Abstract and Section 2"},{"comment":"The DFS probability threshold T is selected by inspecting the public evaluation set (\"We found that values between T = 5% to T = 20% provided a reasonable compromise... Our final results are calculated using T = 9%\"). Since the reported 71.6% is measured on that same set, the headline result is affected by evaluation-set tuning. This is a methodological concern for the claimed state-of-the-art comparison and for the statement that the method generalizes to ConceptARC, where the same hyperparameters are used. Please disclose this explicitly and, if possible, report the model's performance with T chosen on a development set (e.g., a held-out subset of the ARC training tasks) or show that the result is robust across the reported T values.","section":"Section 5.3"}],"minor_comments":[{"comment":"The sentence \"This function is monotonically increasing in T\" is reversed relative to the data in Table 3: as T increases from 0.5% to 20%, the fraction of candidate sets containing the correct solution decreases from 83.5% to 73.5%. Please correct the direction or rephrase to describe the -log(prob) threshold used in Figure 4.","section":"Section 5.3"},{"comment":"In Step 2 of the proof of Theorem C.1, the line reading \"Z = ∑_s ∑_j (1/m) \\hat P_j(s) = 1\" should be written as an inequality (Z ≤ ... ≤ 1); as typeset, the equality is a mathematical error that could confuse readers about the argument.","section":"Appendix C"},{"comment":"The paper states that the final model and code are publicly available on GitHub but does not provide a URL or repository identifier anywhere in the text; please add the link so the reproducibility claim is actionable.","section":"General"},{"comment":"The abstract uses \"2ct per task\" and the introduction uses \"0.02$ per task\"; please use a consistent currency notation and include the explicit conversion to cents.","section":"Abstract and Introduction"},{"comment":"The \"Open Source\" column has a question mark for \"Avg. Human\", which is not a meaningful open/proprietary classification; consider marking it as N/A or removing the column for that row.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and the ablations are well-structured, but the fractional solved count and the evaluation-set tuning of T must be addressed before the headline result can be accepted as stated. If the 286.5 is simply a rounding artifact or a typo, the paper can be corrected quickly; if it reflects a different scoring protocol, the comparisons in Table 1 need to be redone. I recommend a major revision with clear reporting of the scoring metric and hyperparameter selection procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv:2505.07859. The headline result—71.6% on the public ARC-AGI eval with an open 8B model—looks real. The ablations in Tables 2 and 3 are monotone and coherent, the method is transparent, and the per-task cost is tiny. That's a legitimate advance over the previous open best of 62.8%, and it beats the average human score. The paper also deserves credit for checking on ConceptARC and Sudoku, and for actually releasing code.\n\nThe main thing to know: the stress-test note is right. The paper defines a task as solved if either of two guesses is correct—binary per task—so '286.5/400 solved tasks' is not an integer and cannot be the count. It's likely 71.6% times 400 rounded to one decimal, but the abstract presents it as a count. That's a reporting error or an unexplained partial-credit metric, and it needs to be fixed or clarified before the SOTA claim is fully comparable to the baselines, which all report binary per-task rates. I'd guess the actual number is 286 or 287 tasks, and the conclusion doesn't change, but the paper should say so.\n\nSoft spots are proportionate. The DFS threshold T is indeed tuned on the public eval set (Section 5.3), so the 71.6% is not a strictly held-out number. No multi-seed variance over the randomized augmentations either. These are real but not fatal; the method's components are standard, the gains are consistent across ablations, and the ConceptARC result with the same hyperparameters provides some out-of-distribution confirmation.\n\nThe PoE theory is a known identity, not a new theorem, and the paper says so—fine. The citation pattern looks honest; they cite prior ARC work properly. The claim about 'state-of-the-art among publicly available approaches' is credible given the table.\n\nBottom line: this is a solid engineering contribution with a transparent evaluation, one clear reporting inconsistency, and one tuning caveat. I'd send it to peer review and ask the authors to fix the metric reporting and add seed variance. If the numbers hold, it's a useful advance for the open-source ARC community.","headline":"Strong open-source ARC-AGI result, but the fractional solved-task count needs clarification before the headline is fully interpretable.","tokens_in":16416,"tokens_out":2476,"would_cite":true,"duration_ms":21638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source 8B LLM, used as both generator and scorer, solves 71.6% of ARC-AGI tasks.","keywords":["ARC-AGI","abstraction and reasoning","product of experts","test-time training","depth-first search","data augmentation","LLM reasoning","geometric-mean ensemble"],"falsifier":"Run the pipeline on a held-out set of puzzles where the correct answer is known, sample $C_{p,T}$ with $T=9\\%$ so the correct solution is almost always present, and check whether the product-of-experts score picks the correct candidate. If a substantial share of tasks have a wrong candidate with higher aggregate score than the correct one, the ranking assumption fails; an even sharper test is to use augmentations that are not true symmetries (e.g., color remappings that break a task's semantics) and show that PoE then selects incorrect candidates.","tokens_in":15410,"feed_emoji":"🧩","tokens_out":6561,"duration_ms":57359,"temperature":0.7,"pith_summary":"This paper argues that a single open-source 8-billion-parameter language model, fine-tuned on synthetically generated ARC-AGI training puzzles, can reach state-of-the-art performance on the benchmark by being used twice: first as a generator of candidate answers through threshold-pruned depth-first search, and then as a scorer that re-ranks candidates by the product of their probabilities across many augmented views of the problem. The method reports 71.6% accuracy on the public ARC-AGI evaluation set, above the 60.2% average human score and above prior open-source approaches. This matters because it suggests that abstract reasoning on ARC does not require massive proprietary models: the same dual-role trick, test-time training, and cheap hardware yield competitive results at roughly two cents of inference per task.","feed_headline":"Open-source LLM pipeline solves 71.6% of ARC-AGI tasks","feed_subtitle":"An 8B model used as generator and scorer beats the average human at about 2 cents per task.","key_machinery":"The load-bearing object is the geometric-mean ensemble of LLM likelihoods over augmentation-preserved views of a puzzle, called a product of experts. For each candidate solution $s$, the model computes $\\hat{P}(\\phi_j(s)|\\phi_j(p))$ under every transformation $\\phi_j$ and multiplies them, forcing a candidate to be plausible from every perspective at once and suppressing spurious high-confidence errors. The companion mechanism is DFS-based candidate generation: a depth-first search over solution tokens prunes any partial path whose accumulated log-probability falls below a threshold $T$, producing a small, diverse set $C_{p,T}$ of high-probability candidates cheaply. A 64-token vocabulary (one token per grid-cell color plus delimiters) and test-time fine-tuning on each task's examples supply the conditions under which the likelihood signal is trustworthy.","core_discovery":"The central discovery is that next-token likelihood under a fine-tuned LLM is a usable ranking signal for ARC solutions, provided it is aggregated across transformations. The paper defines a candidate set $C_{p,T}$ of solutions pruned by a probability threshold $T$ via depth-first search, then scores each candidate with $\\text{score}_{\\text{agg}}(s)=\\prod_{\\phi_j\\in\\Phi}\\hat{P}(\\phi_j(s)|\\phi_j(p))$, a product-of-experts (geometric-mean) ensemble over D8 symmetries, color permutations, and example orderings. It proves that the KL divergence of this ensemble from the true solution distribution is bounded by the average single-augmentation KL divergence plus a non-positive log-normalization term, so the ensemble is in expectation at least as good as a randomly chosen augmentation. On the public ARC-AGI evaluation set this pipeline solves 286.5 of 400 tasks (71.6%), generalizes to 73.3% on ConceptARC, and reaches 53% on a Sudoku benchmark; in the reported experiments, when the correct Sudoku solution is sampled, the ensemble selects it 100% of the time.","pith_inferences":["The augmentation-product trick should transfer to any reasoning domain with cheap semantic-preserving transformations, such as shuffled program syntax, paraphrased math word problems, or rotated visual grids; disagreement across views is what makes the product filter false positives.","The KL bound suggests that the ensemble's benefit grows exactly when the LLM is inconsistent across augmentations, so deliberately diverse problem representations could be engineered rather than taken as a given.","Because the whole pipeline leans on next-token probability as a quality proxy, calibrating those probabilities (e.g., with temperature or labeled validation tasks) could further close the gap to the upper-bound coverage line in Figure 4.","A natural extension the paper leaves implicit is applying the DFS+PoE scheme to induction-style solutions (inferred programs) instead of only transduction-style grid outputs, potentially combining both routes on the same puzzle."],"forward_implications":["An open-source 8B model with LoRA fine-tuning, test-time training, and the generate-then-rescore pipeline matches or beats the average human on ARC-AGI public tasks.","The same method transfers to other ARC-like benchmarks (ConceptARC) and structured domains (Sudoku), suggesting a recipe rather than a benchmark-specific patch.","Inference cost drops to about $0.02 per task on a single Nvidia RTX 4090, versus the reported $17 per task for the closed-source o3 system.","Selecting by product of experts outperforms selecting by mean probability, max probability, or single-augmentation probability in all reported sampling regimes.","Threshold $T$ controls the coverage/selectivity tradeoff: lower $T$ finds more correct solutions but floods the candidate set with false positives; moderate $T$ around 9% gives the best two-guess accuracy."],"supporting_citations":[{"why":"Supplies the RE-ARC generative dataset used for initial fine-tuning, the only external training data in the pipeline.","marker":"Hodel (2024)"},{"why":"Introduces the test-time training recipe that more than doubles baseline performance in the ablations.","marker":"Akyürek et al. (2024)"},{"why":"Provides the NeMo-Minitron-8B base model that the authors fine-tune for all main results.","marker":"Sreenivas et al. (2024)"},{"why":"The product-of-experts formulation whose log-pooling identity underlies Theorem 4.1.","marker":"Hinton (1999; 2002)"},{"why":"Provides the 60.2% average-human and 97.8% solved-by-at-least-one-participant baselines used for comparison.","marker":"LeGris et al. (2024)"},{"why":"Documents the o3 score of 82.8% and $17-per-task compute cost used as the closed-source comparison.","marker":"arcprize.org (2025)"},{"why":"Introduces ARC-AGI and defines the benchmark the paper targets.","marker":"Chollet (2019)"}],"fun_headline_variants":["LLM product-of-experts hits 71.6% on ARC-AGI","Open-source LLM solver reaches 71.6% on ARC-AGI","LLM as scorer: 71.6% on ARC-AGI at 2¢ per task","Product-of-experts LLM achieves 71.6% on ARC-AGI","LLM ensemble scores 71.6% on ARC-AGI for ~2¢ each"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's next-token probabilities rank solution quality reliably, so that DFS pruning at threshold $T$ and the product-of-experts score select the correct output; the paper does not separately calibrate or test this ranking assumption apart from the final benchmark score.","fun_headline_variants_meta":{"raw":{"variants":["LLM product-of-experts hits 71.6% on ARC-AGI","Open-source LLM solver reaches 71.6% on ARC-AGI","LLM as scorer: 71.6% on ARC-AGI at 2¢ per task","Product-of-experts LLM achieves 71.6% on ARC-AGI","LLM ensemble scores 71.6% on ARC-AGI for ~2¢ each"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00128,"raw_usage":{"total_tokens":5243,"prompt_tokens":964,"completion_tokens":4279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":4163}},"tokens_in":580,"tokens_out":4279,"duration_ms":31267,"temperature":1.0,"reasoning_tokens":4163,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:11:53.751035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a held-out set of puzzles where the correct answer is known, sample $C_{p,T}$ with $T=9\\%$ so the correct solution is almost always present, and check whether the product-of-experts score picks the correct candidate. If a substantial share of tasks have a wrong candidate with higher aggregate score than the correct one, the ranking assumption fails; an even sharper test is to use augmentations that are not true symmetries (e.g., color remappings that break a task's semantics) and show that PoE then selects incorrect candidates.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The product-of-experts formulation whose log-pooling identity underlies Theorem 4.1."}],"review_version":1}