Pith. sign in

REVIEW 3 major objections 6 minor 34 references

Neuro-symbolic AI collapses at strategic reasoning, new benchmark shows

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 00:23 UTC pith:OS4GYRGL

load-bearing objection Benchmark with a genuinely useful three-axis framework, but the difficulty calibration has a real gap that undermines the central procedural-composition claim. the 3 major comments →

arxiv 2607.05185 v1 pith:OS4GYRGL submitted 2026-07-06 cs.AI cs.SC

ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

Mahnoor Shahid , Hannes Rothe This is my paper
classification cs.AI cs.SC
keywords benchmarkcompositionalstrategiesabilityagentclassicclassiclogiccomplex
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces ClassicLogic, a benchmark built from four classic logic puzzles—Sudoku, KenKen, Kakuro, and Futoshiki—whose central innovation is a hierarchical knowledge base where every solving strategy is formally defined as a composition of simpler ones. The authors use this structure to separate compositional generalization into three distinct layers: Entity Composition (perceiving handwritten digits as symbolic entities), Relational Composition (following game rules like uniqueness or inequality constraints), and Procedural Composition (chaining simple strategies into novel multi-step plans). By generating puzzles with mathematically validated difficulty tied to the compositional depth of required strategies, the benchmark can test whether a model has learned to compose reasoning steps rather than merely memorize patterns. The authors train three neuro-symbolic architectures—SATNet, Neural Module Networks, and Logical Neural Networks—on easy and moderate puzzles, then test them on hard puzzles that demand strategy compositions not seen during training. All three models drop below 25% success on these hard puzzles, despite performing reasonably on perceptual and rule-following tasks. The paper's core claim is that current neuro-symbolic systems can learn static rules but cannot compose them into dynamic, multi-step plans, and that this gap is diagnosable precisely because the benchmark's knowledge base makes the composition structure explicit.

Core claim

The paper's central discovery is a sharp qualitative separation between relational and procedural composition in neuro-symbolic models. Models that learn to respect game rules (relational composition) and parse visual inputs (entity composition) still fail catastrophically when required to chain known strategies into novel multi-step sequences (procedural composition), with success rates below 25% across all tested architectures. The hierarchical knowledge base is the mechanism that makes this separation visible: because each complex strategy is formally defined as a composition of atomic ones, and puzzle difficulty is calibrated by the compositional depth of required strategies, the authors

What carries the argument

Hierarchical Knowledge Base

Load-bearing premise

The paper assumes that its compositional depth metric—how many layers of simpler strategies a strategy is built from—correctly measures reasoning difficulty, and that its knowledge base captures all strategies needed to solve the generated puzzles. This is validated only on Sudoku by correlating depth with solver time and strategy count, without confirming that the strategy hierarchy is complete or that the depth metric generalizes to the other three games.

What would settle it

A model that achieves above 50% success on hard puzzles across all four games while trained only on easy and moderate puzzles would directly challenge the claim that procedural composition is a catastrophic failure point for current architectures.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark's difficulty calibration is sound, then any AI system achieving high success on hard puzzles would demonstrate genuine procedural composition—the ability to construct novel multi-step strategies from known components—a capability currently lacking in tested neuro-symbolic architectures.
  • The three-layer decomposition of compositional generalization (entity, relational, procedural) could become a standard diagnostic axis for evaluating reasoning systems, allowing researchers to pinpoint whether a model fails at perception, rule-following, or strategic planning.
  • If the procedural composition gap is real and not an artifact of benchmark design, it suggests that architectural innovations beyond current neuro-symbolic approaches are needed for multi-step strategic reasoning—possibly mechanisms for explicit planning, search, or program synthesis.
  • The benchmark's modular design means new puzzle types can be added with their own strategy hierarchies, potentially expanding the diagnostic to domains like graph-coloring or temporal logic puzzles.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The procedural composition failure might partly reflect the benchmark's training protocol: models trained only on easy and moderate puzzles may not have sufficient exposure to the compositional structure needed to generalize to hard puzzles. Testing with curriculum learning or explicit strategy composition training could reveal whether the gap is architectural or pedagogical.
  • The difficulty validation is performed only on Sudoku; if the compositional depth metric does not correlate with computational hardness for the other three games, the procedural composition results on those games could conflate genuine reasoning failure with miscalibrated difficulty.
  • The knowledge base is acknowledged as non-exhaustive; if alternative solution paths exist using uncatalogued strategies, some hard puzzles might be solvable by simpler means the benchmark does not recognize, which would inflate the apparent procedural composition gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ClassicLogic, a benchmark suite of four classic logic puzzles (Sudoku, KenKen, Kakuro, Futoshiki) with a hierarchical knowledge base (KB) of solving strategies, designed to evaluate three forms of compositional generalization: entity composition (perception), relational composition (rule-following), and procedural composition (strategic planning). The benchmark features procedural puzzle generation with a two-stage template-based approach, a formal compositional depth metric δ(s) for strategies, and guaranteed unique solutions. Experiments evaluate three neuro-symbolic models (SATNet, NMN, LNN) on zero-shot transfer and in-domain procedural composition tasks.

Significance. The hierarchical KB, where complex strategies are explicitly defined as compositions of simpler ones, is a genuinely useful contribution that enables fine-grained diagnostic evaluation. The three-axis taxonomy of compositional generalization (entity/relational/procedural) is well-motivated and fills a gap left by existing NLP-centric benchmarks. The open-source release with automated setup scripts and the strategy-driven template generation pipeline are commendable for reproducibility. The falsifiable prediction that current NeSy models fail specifically at procedural composition (not just general difficulty) is the kind of claim that, if properly validated, would make this a valuable diagnostic tool.

major comments (3)
  1. Algorithm 1 (§3.4.1) implements a one-directional difficulty check: the LimitedSolver verifies that a puzzle CAN be solved with strategies of depth ≤ D_target, but does not verify that strategies of depth < D_target are INSUFFICIENT. The paper defines Σ_req as 'the pre-analyzed, minimal set of strategies required to solve any valid instantiation of this template' (§3.4.1), but the algorithm provides no mechanism to establish this minimality. Consequently, a 'Hard' puzzle (D_target=3) could in principle be solvable using only Easy (depth=1) strategies applied more extensively. This is load-bearing because the central experimental claim in §4.4—that 'Hard puzzles are guaranteed to require composed strategies in sequences not seen during training'—depends on Hard puzzles genuinely requiring depth-D_target strategies. Without a bidirectional check (i.e., verifying that a LimitedSolver with K
  2. §4.1: The empirical validation of the difficulty metric D(P) covers only Sudoku (99 puzzles, 33 per difficulty). The other three games (KenKen, Kakuro, Futoshiki) have no empirical validation that the difficulty categories correlate with computational hardness. The paper states this Sudoku validation 'provides confidence that the difficulty scaling is a robust feature of the entire benchmark suite,' but this extrapolation is unsupported given the different constraint structures (arithmetic cages, cross-sum logic, ordinal constraints) of the other games. Since the procedural composition results in §4.4 span all four games, the lack of difficulty validation for three of them weakens the cross-game claims.
  3. §4.4 and Figure 4: The procedural composition results are presented only as a bar chart without a corresponding table of exact numbers, standard deviations, or sample sizes. The text states success rates are 'below 25% across the board' and mentions LNN at '20-24%,' but precise per-game, per-model numbers are not reported. Additionally, the training protocol states models were trained on 'Easy' and 'Moderate' puzzles and tested on 'Hard' puzzles, but the number of training and test instances per game is not specified. This makes it difficult to assess the statistical reliability of the claimed 'catastrophic failure.'
minor comments (6)
  1. Table 3 caption states 'The results show that models are generalizing well across other games,' which directly contradicts the body text in §4.3 stating 'their accuracy dropped significantly on the other games' and 'a critical failure to generalize abstract logical principles.' The caption should be corrected.
  2. §3.3: The composition notation is inconsistent. The KB uses sc := L_{i=1}^{k} s_i for composition, but §3.5 uses s_new := s_k ∘ ... ∘ s_1, and the Sudoku example uses s_np := s_find_pair ⊕ s_cp. Three different operators (L, ∘, ⊕) are used for the same concept.
  3. Figure 2 is referenced as showing the strategy hierarchy for Sudoku but the text mentions strategies (Naked Pair, X-Wing, Swordfish) without clearly mapping them to the depth values δ(s) used in the difficulty metric. A table listing each strategy with its δ(s) value would clarify the formal definition.
  4. §4.3: The training protocol states 'all models were trained exclusively on the Sudoku environment,' but it is unclear whether the models were trained on puzzles of all difficulty levels or only specific ones for the entity/relational composition experiment.
  5. The paper does not discuss hyperparameter selection for the three baseline models. Given that only three models are tested, reporting whether they were tuned (and how) would strengthen the fairness of the comparison.
  6. §3.4.1, line 8 of Algorithm 1: 'K_limit ← {s ∈ K | δ(s) ≤ D_target}' uses ≤, but the text in §3.4.1 says 'less than or equal to the target difficulty D_target.' This is consistent, but the LimitedSolver is described as 'equipped only with strategies from the Knowledge Base whose compositional depth δ(s) is less than or equal to' — clarify whether δ(s) ≤ D_target or δ(s) < D_target, as this affects which strategies are available at each difficulty tier.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee identifies three substantive issues: (1) Algorithm 1 verifies sufficiency but not necessity of depth-D_target strategies, (2) difficulty validation covers only Sudoku, and (3) procedural composition results lack exact numbers, standard deviations, and sample sizes. We agree with all three points and will revise accordingly.

read point-by-point responses

  1. Referee: Algorithm 1 implements a one-directional difficulty check: LimitedSolver verifies a puzzle CAN be solved with strategies of depth <= D_target, but does not verify that strategies of depth < D_target are INSUFFICIENT. The claim that Hard puzzles require composed strategies depends on this minimality, which is not established.

    Authors: The referee is correct. Algorithm 1 verifies that a puzzle is solvable using strategies up to depth D_target, but it does not verify that lower-depth strategies are insufficient. This is a genuine gap between what the algorithm guarantees and what the experimental claims in Section 4.4 require. We will address this in two ways. First, we will add a bidirectional check to the template generation pipeline: after confirming solvability with strategies of depth <= D_target, we will additionally run LimitedSolver with strategies of depth <= D_target - 1 and require that it FAILS to solve the puzzle. Only templates passing both checks will be retained as difficulty-D_target templates. Second, we will revise the language in Section 4.4 to accurately reflect what is guaranteed: that Hard puzzles are designed to require strategies at depth D_target, with the bidirectional check ensuring minimality. We will also add a note clarifying that this minimality holds at the template level and that individual instantiations inherit this property because Stage 2 instantiation preserves the logical structure of the template. revision: yes

  2. Referee: The empirical validation of the difficulty metric D(P) covers only Sudoku (99 puzzles, 33 per difficulty). The other three games have no empirical validation that difficulty categories correlate with computational hardness. The extrapolation to the full suite is unsupported given the different constraint structures of KenKen, Kakuro, and Futoshiki.

    Authors: The referee is right that extrapolating from Sudoku alone to all four games is unsupported, particularly given the structurally different constraint types (arithmetic cages, cross-sum logic, ordinal constraints) in the other three puzzles. We will extend the empirical validation to all four games. Specifically, we will generate 99 puzzles per game (33 per difficulty category) and run both the glass-box solver (measuring strategy count) and the black-box CSP solver (measuring wall-clock time), following the same protocol as the existing Sudoku validation. We will report Spearman's rho for each game individually. We will also revise the claim in Section 4.1 to remove the unsupported extrapolation and instead present per-game validation results. If any game shows a weaker correlation, we will report this transparently and discuss possible reasons. revision: yes

  3. Referee: The procedural composition results in Section 4.4 and Figure 4 are presented only as a bar chart without exact numbers, standard deviations, or sample sizes. The training protocol does not specify the number of training and test instances per game, making it difficult to assess the statistical reliability of the claimed catastrophic failure.

    Authors: This is a fair criticism. We will add a detailed table reporting per-game, per-model success rates with standard deviations and sample sizes for both training and test sets. Specifically, we will report: (1) the number of training instances (Easy + Moderate puzzles) per game, (2) the number of test instances (Hard puzzles) per game, (3) mean success rates with standard deviations across multiple runs, and (4) the number of random seeds used. We will also add confidence intervals to Figure 4 or replace it with a table-plus-figure combination that provides both visual and precise numerical information. This will allow readers to assess the statistical reliability of the procedural composition failure we report. revision: yes

Circularity Check

1 steps flagged

Mild circularity in the glass-box validation (same KB used for both difficulty metric and solver), but independent black-box CSP validation and the actual model evaluation results carry independent content.

specific steps
  1. fitted input called prediction [§4.1, Table 2, Figure 3 — glass-box solver validation]
    "Glass-Box Solver: Our implemented baseline agent that greedily applies strategies from the Knowledge Base K. We measured the total number of distinct strategies applied to reach a solution. [...] The number of strategies applied by the glass-box solver and the solve time for the black-box CSP solver both increase substantially with each difficulty category"

    The difficulty metric D(P) is derived from the KB's compositional depth δ(s), and the glass-box solver also applies strategies from the same KB. Correlation between D(P) and the glass-box solver's strategy count is partly tautological: both quantities are computed from the same strategy hierarchy. However, this is mitigated by the independent black-box CSP solver (ρ=0.98), which is unaware of the KB and provides genuine external validation. The circularity is therefore partial and non-load-bearing for the central claims.

full rationale

The paper's central experimental claims (model failure rates on Hard puzzles, zero-shot transfer failures) are evaluated against externally trained models (SATNet, NMN, LNN) and have independent content regardless of the difficulty metric's provenance. The main concern raised by the reader — that Algorithm 1's LimitedSolver only checks sufficiency (puzzle solvable with depth ≤ D_target) but not necessity (puzzle requires depth = D_target) — is a correctness risk (the paper claims Σ_req is 'minimal' without verifying minimality), not a circularity issue. The difficulty metric is defined in terms of the KB structure, not in terms of the LimitedSolver's output, so there is no self-definitional loop. The glass-box validation in §4.1 is mildly circular because both the metric and the solver draw from the same KB, but the independent black-box CSP solver correlation (ρ=0.98) provides external support. No self-citations are present in the reference list. Overall, the derivation is largely self-contained with one minor circular validation step that is not load-bearing for the paper's conclusions.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The benchmark's core machinery depends on several internally defined constructs (δ(s), LimitedSolver, the three-axis taxonomy) whose validity is checked primarily through internal consistency rather than external benchmarks. The Sudoku validation provides one external check (CSP solver correlation), but the other three games lack equivalent validation. The KB completeness assumption is acknowledged as a limitation but is structurally load-bearing for the generation algorithm.

free parameters (3)
  • D_target (target difficulty) = Easy/Moderate/Hard (ordinal)
    The difficulty categories are defined by the KB's compositional depth and used to partition puzzles. The mapping from depth to category is a design choice, not derived from external data.
  • Strategy depth thresholds = Not explicitly stated
    The boundaries between Easy/Moderate/Hard in terms of δ(s) are not numerically specified. The paper states δ(s) is recursive but does not give the actual depth values for each strategy.
  • MNIST digit rendering parameters = Not specified
    Puzzles are rendered with MNIST digits, but the specific digit samples, sizing, and noise parameters used for rendering are not stated.
axioms (4)
  • domain assumption The hierarchical KB for each game captures all strategies needed to solve any valid puzzle instance.
    The LimitedSolver and difficulty metric depend on the KB being complete. The paper acknowledges in §6 that the KB is not claimed to be exhaustive, yet the generation algorithm (Alg. 1, line 9) requires the LimitedSolver to succeed, which assumes KB completeness for solvability.
  • domain assumption Compositional depth δ(s) is a valid proxy for puzzle difficulty and compositional reasoning complexity.
    The entire difficulty calibration rests on this. Validated empirically only for Sudoku (§4.1) via correlation with CSP solver time, not for the other three games.
  • domain assumption Training on Easy/Moderate puzzles exposes models to all base and moderate strategies, and Hard puzzles require compositions not seen during training.
    The procedural composition experiment (§4.4) depends on this separation. The paper does not verify that the specific strategy sequences required by Hard puzzles are genuinely absent from the training set.
  • ad hoc to paper The three-axis taxonomy (entity/relational/procedural composition) is a complete partition of compositional generalization for these tasks.
    The paper claims this is 'the full stack of compositional reasoning' but does not justify why these three axes are exhaustive or independent. Other formulations (e.g., systematic vs. productive vs. substitutive composition) exist in the cognitive science literature cited.
invented entities (3)
  • Compositional depth function δ(s) no independent evidence
    purpose: Defines the recursive depth of a strategy in the KB hierarchy, used as the difficulty metric D(P).
    δ(s) is defined by the paper's own KB structure. Its validity as a difficulty measure is checked only via correlation with CSP solver time on Sudoku. No external standard for 'compositional depth' is referenced.
  • LimitedSolver no independent evidence
    purpose: A constrained solver that uses only KB strategies up to a given depth, used in template generation to validate puzzle difficulty.
    The LimitedSolver is the paper's own construction. Its correctness depends on the KB being complete. No comparison with independent difficulty ratings (e.g., human difficulty ratings or published Sudoku difficulty databases) is provided.
  • Three-axis composition taxonomy (Entity/Relational/Procedural) no independent evidence
    purpose: Framework for evaluating and disentangling forms of compositional generalization.
    Introduced by this paper without prior validation. The axes are motivated by examples but not formally derived from cognitive theory or prior compositional generalization literature.

pith-pipeline@v1.1.0-glm · 15537 in / 3564 out tokens · 142093 ms · 2026-07-08T00:23:55.579257+00:00 · methodology

0 comments
read the original abstract

Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.

Figures

Figures reproduced from arXiv: 2607.05185 by Hannes Rothe, Mahnoor Shahid.

Figure 1
Figure 1. Figure 1: Visual examples from the ClassicLogic benchmark, showcasing the diversity of reasoning challenges. The top row (a-c) displays the initial puzzle states for Sudoku, KenKen, and Futoshiki, presented with perceptual MNIST digits. The bottom row (d-f) shows their corresponding, unique logical solutions. 3.4 Procedural Generation and Difficulty Calibration A central requirement for a robust benchmark is the abi… view at source ↗
Figure 2
Figure 2. Figure 2: An example illustration of the strategy hierarchy in the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Correlation between the difficulty categories and empirical measures. The bars (left axis) show the increase in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Procedural composition results across all games. Success rate defines the accurate formation and application [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · 7 internal anchors

  1. [1]

    Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016). Neural module networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2016

  2. [2]

    Berthier, D. (2013). Pattern-based constraint satisfaction and logic puzzles.arXiv preprint arXiv:1304.1628

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  3. [3]

    and Sandholm, T

    Brown, N. and Sandholm, T. (2019). Superhuman AI for multiplayer poker.Science, 365(6456):885–890

    work page 2019

  4. [4]

    Cao, P., Men, T., Liu, W., Zhang, J., Li, X., Lin, X., Sui, D., Cao, Y ., Liu, K., and Zhao, J. (2025). Large language models for planning: A comprehensive and systematic survey.arXiv preprint arXiv:2505.19683

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    and Tooby, J

    Cosmides, L. and Tooby, J. (1997). The modular nature of human intelligence.The origin and evolution of intelligence, pages 71–101

    work page 1997

  6. [6]

    Dong, H., Mao, J., Lin, T., Wang, C., Lih, L., and Zhou, D. (2019). Neural logic machines. InInternational Conference on Learning Representations (ICLR)

    work page 2019

  7. [7]

    Fodor, J. A. and Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis.Cognition, 28(1-2):3–71

    work page 1988

  8. [8]

    Garcez, A. d. and Lamb, L. C. (2023). Neurosymbolic ai: The 3 rd wave.Artificial Intelligence Review, 56(11):12387– 12406

    work page 2023

  9. [9]

    Giadikiaroglou, P., Lymperaiou, M., Filandrianos, G., and Stamou, G. (2024). Puzzle solving using reasoning of large language models: A survey.arXiv preprint arXiv:2402.11291

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    and Sarker, M

    Hitzler, P. and Sarker, M. K. (2022). Neuro-symbolic artificial intelligence: The state of the art

    work page 2022

  11. [11]

    Hudson, D. A. and Manning, C. D. (2019). GQA: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2019

  12. [12]

    L., and Girshick, R

    Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. (2017). CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910

    work page 2017

  13. [13]

    Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., et al. (2019). Measuring compositional generalization: A comprehensive method on realistic data. arXiv preprint arXiv:1912.09713

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  14. [14]

    Keysers, D., Schärli, N., Kale, N., Cer, D., Firat, O., Mourad, A., Riesa, J., Bapna, A., Caswell, I., and Hassan, H. (2020). Measuring compositional generalization: A comprehensive method on realistic data. InInternational Conference on Learning Representations (ICLR)

    work page 2020

  15. [15]

    and Linzen, T

    Kim, N. and Linzen, T. (2020). COGS: A compositional generalization challenge based on semantic interpretation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9107–9123

    work page 2020

  16. [16]

    Lake, B. M. and Baroni, M. (2018). Generalization without systematicity: On the compositional skills of sequence-to- sequence recurrent networks. InInternational Conference on Machine Learning (ICML), pages 2873–2882

    work page 2018

  17. [17]

    Liang, B., Wang, Y ., and Tong, C. (2025). Ai reasoning in deep learning era: From symbolic ai to neural–symbolic ai. Mathematics, 13(11):1707

    work page 2025

  18. [18]

    Manhaeve, R., Dumanˇci´c, S., Kimmig, A., Demeester, T., and De Raedt, L. (2018). DeepProblog: Neural probabilistic logic programming. InAdvances in Neural Information Processing Systems (NeurIPS)

    work page 2018

  19. [19]

    Marcus, G. (2018). Deep learning: A critical appraisal.arXiv preprint arXiv:1801.00631. 12 ClassicLogic: BENCHMARK OFCLASSICPUZZLEGAMES FORCOMPOSITIONALGENERALIZATION

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  20. [20]

    and Baral, C

    Mitra, A. and Baral, C. (2015). Learning to automatically solve logic grid puzzles. InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1023–1033

    work page 2015

  21. [21]

    Advancing Reasoning in Large Language Models: Promising Methods and Approaches

    Patil, A. and Jadon, A. (2025). Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03671

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    J., and Browne, C

    Piette, C., Piette, E., Stephenson, M., Soemers, D. J., and Browne, C. (2019). Ludii and xcsp: playing and solving logic puzzles. In2019 IEEE Conference on Games (CoG), pages 1–4. IEEE

    work page 2019

  23. [23]

    Qefalija, E., Snopce, H., and Dermaku, A. (2024). Literature review on constraint satisfaction problems solving. In 2024 8th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), pages 1–6. IEEE

    work page 2024

  24. [24]

    Y ., Yarkoni, T., Koco, Z., and Mvelase, P

    Riegel, R., Gray, A., Luus, F., Khan, N., Makondo, N., Akhalwaya, I. Y ., Yarkoni, T., Koco, Z., and Mvelase, P. (2020). Logical neural networks

    work page 2020

  25. [25]

    Ruis, L., Andreas, J., Baroni, M., Dagan, I., and Goldberg, Y . (2020). A benchmark for systematic generalization in grounded language understanding. InInternational Conference on Machine Learning (ICML)

    work page 2020

  26. [26]

    Russell, S. J. and Norvig, P. (2010).Artificial Intelligence: A Modern Approach. Pearson Education, 3rd edition

    work page 2010

  27. [27]

    Sakr, C., Hitzler, P., and Sheth, A. (2022). Neuro-symbolic ai star: A tale of two worlds.AI Magazine, 43(4):406–419

    work page 2022

  28. [28]

    Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. (2016). Mastering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489

    work page 2016

  29. [29]

    Sternberg, R. J. (1984). Toward a triarchic theory of human intelligence.Behavioral and Brain Sciences, 7(2):269–287

    work page 1984

  30. [30]

    Wang, P.-W., Donti, P., Wilder, B., and Kolter, Z. (2019). SATNet: Bridging deep learning and logical reasoning using a differentiable MAX-SAT solver. InInternational Conference on Machine Learning (ICML)

    work page 2019

  31. [31]

    Wang, W., Yang, Y ., and Wu, F. (2024). Towards data-and knowledge-driven ai: a survey on neuro-symbolic computing. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2024

  32. [32]

    Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. (2023). Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36:6941–6960

    work page 2023

  33. [33]

    Yi, K., Gan, C., Li, Y ., Torralba, A., Kohli, P., and Tenenbaum, J. (2020). CLEVRER: Collision events for video representation and reasoning. InInternational Conference on Learning Representations (ICLR)

    work page 2020

  34. [34]

    A Survey of Large Language Models

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2). 13

    work page internal anchor Pith review Pith/arXiv arXiv 2023