Pith. sign in

REVIEW 3 major objections 6 minor 72 references

Label accuracy does not equal rule recovery: current AI agents fail at active visual concept induction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 10:54 UTC pith:JF344I4D

load-bearing objection Clean, useful benchmark that finally puts visual perception, rule induction, and active experiment design in one loop; the three headline dissociations are real and well-supported. the 3 major comments →

arxiv 2607.08233 v1 pith:JF344I4D submitted 2026-07-09 cs.AI cs.CV

Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

Sophia Koehler , Antonia W\"ust , Inga Ibs , Wasu Top Piriyakulkij , Wolfgang Stammer , Constantin Rothkopf , Kevin Ellis , Kristian Kersting This is my paper
classification cs.AI cs.CV
keywords active visual inductionZendoWorldhypothesis testingvision-language modelsneuro-symbolic AIexperiment designscientific discoveryrule recovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Building agents that can see a scene, form a hypothesis about a hidden rule, and design experiments to test that hypothesis is still unsolved as a single loop. The authors introduce ZendoWorld, an interactive visual game environment inspired by the inductive logic game Zendo, in which an agent receives labeled 3D scenes, proposes new scenes, and must eventually state a rule that is logically equivalent to a hidden ground-truth rule. Across pure vision-language models, Bayesian particle filters, vision-language program synthesizers, and a privileged neuro-symbolic oracle, they show that high accuracy at labeling observed examples does not imply recovery of the true rule, that perception and induction fail in different ways for different agent classes, and that VLM-based agents almost never propose experiments that reduce hypothesis uncertainty. Human players outperform the agents, especially on complex and out-of-distribution rules. The benchmark therefore supplies a controlled stress test for active scientific discovery in vision.

Core claim

On ZendoWorld, current agents can correctly label many observed scenes while still failing to recover the hidden rule, and VLM-based agents systematically propose near-uninformative experiments that leave the hypothesis space essentially unreduced. Perception and induction are separable bottlenecks: removing vision helps some agents a lot and others little, and human players close the gap mainly on harder rules.

What carries the argument

ZendoWorld, a closed interactive loop that combines raw 3D visual scenes, a formal Prolog-style rule DSL, agent-proposed experiments, and equivalence-checked hypothesis submission, so that perception, induction, and experimentation can be measured separately.

Load-bearing premise

That whether a guessed rule matches the hidden rule can be decided reliably by DSL canonicalization or, when needed, by an LLM translator and judge.

What would settle it

Re-run the same agent suite with a fully symbolic, non-LLM equivalence checker (or human adjudication of every natural-language hypothesis) and check whether win rates, structural-F1 trajectories, and the label-accuracy-versus-rule-recovery dissociation remain essentially unchanged.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ZendoWorld, an interactive 3D visual benchmark that jointly requires perception of rendered scenes, induction of a hidden logical rule from a Prolog-style DSL, and active experiment design by proposing new scenes. Agents observe labeled seed scenes, propose experiments with predicted labels, and may submit rule hypotheses when predictions are correct; episodes end on logical equivalence to the ground-truth rule or after 30 observations. The authors evaluate four agent classes (Oracle neuro-symbolic upper bound, end-to-end VLM, Bayesian LLM-SMC, and Vision-Language Programs) over 22 games spanning first-order, second-order, compositional, and one OOD rule, with multi-seed runs, symbolic-input ablations, backbone ablations, structural-F1 learning curves, and an expected-information-gain (EIG) metric. A human study (19 participants, 10 points per game on a 6-game subset) provides an external reference. The central empirical claims are that high label accuracy does not imply rule recovery, that perception and induction are distinct agent-dependent bottlenecks, and that VLM-based agents propose near-uninformative experiments.

Significance. If the results hold, ZendoWorld fills a clear gap in the taxonomy of benchmarks (Table 1) by combining grounded 3D visual input, explicit rule recovery, and active experimentation in one closed loop—capabilities that matter for AI scientists and inductive agents. The dissociation between labeling accuracy and rule recovery (e.g., Bayesian Agent 75.2% label accuracy vs 13.6% wins), the symbolic ablation isolating perception (Table 4), and the near-zero late-game EIG of VLM proposals (Fig. 6) are concrete, falsifiable findings that go beyond passive few-shot visual reasoning. Strengths include multi-seed evaluation, released code and data, a formal DSL with algorithmic equivalence checking for most guesses, and human comparison. These make the work a useful testbed and diagnostic for active visual concept induction rather than only another leaderboard.

major comments (3)
  1. [§4.3 / Fig. 4 / Table 8] §4.1.1, §4.3, Fig. 4, and Table 8: Human–agent comparisons (including the claim of a gap that widens with rule complexity) rest on only 6 of the 22 games, with uneven coverage of complexity classes and only one OOD rule. The aggregate human win rate of 73.3% and the complexity-stratified plot therefore over-generalize from a non-representative subset. Either expand the human study to a stratified sample of all complexity classes or restrict the comparative claims (and Fig. 4) to the six shared games with explicit caveats.
  2. [§3.1 / App. D] §3.1 and App. D: Win decisions and counterexample generation for natural-language hypotheses fall back to an LLM judge when DSL translation fails. Although fallback rates are low (1.8–6.3%), the VLM Agent’s wins are the most exposed to this path, and the same LLM family is used both as player and as judge. For the primary win metric, report a sensitivity analysis that treats LLM-adjudicated wins as uncertain, or require dual independent judges / human adjudication for those cases, so that finding (1) and the VLM win rate are not partly circular.
  3. [§4.3 Q4 / Fig. 6 / App. C] §4.3 Q4 and Fig. 6: The EIG analysis that underpins finding (3)—that VLM-based agents propose near-uninformative experiments—is conducted only in the symbolic-input setting with a top-K=20 PCFG posterior. The main visual results do not report an analogous information-gain measure. Either compute EIG (or a proxy) on the visual trajectories, or clearly limit the claim about “near-uninformative experiments” to the symbolic ablation rather than the full visual loop.
minor comments (6)
  1. [Table 3 / §4.3] Table 3 reports human label accuracy (53.9%) below several agents while humans win more games; a short discussion of why humans may under-predict labels yet recover rules (e.g., different exploration strategy or risk preference) would help readers interpret the dissociation.
  2. [§4.2 Metrics / Fig. 5] Structural F1 is defined via permutation-invariant matching of commutative subexpressions (Q3 / Fig. 5) but is not fully formalized in the main text; a brief definition or pointer to an appendix equation would improve reproducibility.
  3. [App. B / §4.3] App. B backbone ablations show large variance (e.g., VLM Agent wins 4.5–45.5% across models). The main text should note more prominently that headline VLM numbers are backbone-dependent rather than architecture-invariant.
  4. [Fig. 1 / Table 1] Figure 1’s Venn-style taxonomy is useful but the caption and Table 1 slightly disagree on which prior environments count as “experimentation”; align the two.
  5. [§3.1] Minor notation: |D| ≤ 30 is used both as budget and as termination; clarify whether the 30th observation can still trigger a final rule guess.
  6. [Throughout] Typos / consistency: “ZENDOWORLD” vs “ZendoWorld” capitalization varies; “RA VEN” spacing in related work; arXiv ID in the header is 2607.08233 while the abstract block is fine—standard copy-edit pass recommended.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark measurements against independent ground-truth rules and PCFG enumeration do not reduce to inputs by construction.

full rationale

ZendoWorld is an empirical evaluation paper that defines a new interactive environment, runs multiple agent classes (Oracle, VLM, Bayesian, VLP) plus humans under a fixed protocol (|D|<=30, 22 games, multi-seed), and reports win rates, label accuracy, structural F1 trajectories, and expected information gain (EIG) relative to an independent top-K heap-search posterior over a uniform PCFG. The three headline findings—(1) label accuracy dissociated from rule recovery (e.g., Bayesian 75.2% label acc vs 13.6% wins), (2) perception vs induction bottlenecks (symbolic ablation Table 4), (3) near-zero EIG of VLM proposals (Fig. 6)—are direct comparative measurements against ground-truth rules r* and counter-examples generated by rejection sampling; none is obtained by fitting a free parameter and then re-predicting a quantity forced by that fit. The Oracle’s DSL is hand-crafted from the same Looney Labs card set used to sample rules, but this is explicitly disclosed as privileged access, the Oracle is treated only as an approximate upper bound, and the comparative patterns among non-privileged agents survive its removal. Equivalence checking falls back to an LLM judge in only 1.8–6.3% of guesses (App. D) and is not load-bearing for the reported metrics. No self-definitional loop, fitted-input-as-prediction, uniqueness theorem imported from overlapping authors, or ansatz smuggled via self-citation appears in the derivation of the claims. The paper is therefore self-contained against its own external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

As a benchmark paper the load-bearing commitments are design choices rather than free physical parameters. The central claims rest on the fidelity of the Blender–Prolog scene generator, the completeness of the hand-crafted DSL relative to the intended rule space, the reliability of LLM translation for natural-language hypotheses, and the representativeness of the 22 selected games (including one deliberately OOD rule). No continuous parameters are fitted to produce the headline numbers; experimental knobs (max 30 examples, 5 seeds, top-K=20 for EIG) are disclosed.

free parameters (3)
  • max observations |D| ≤ 30
    Hard episode budget chosen by the authors; affects win rates and turn counts but is held fixed across agents.
  • EIG top-K = 20 hypotheses
    Number of programs retained for the expected-information-gain calculation (App. C); changes the numerical EIG values but not the qualitative ranking of agents.
  • Bayesian particle count / resampling size (25)
    Support size of the sequential Monte Carlo posterior; a free design choice of the Bayesian agent.
axioms (4)
  • domain assumption Logical equivalence of two rules can be decided by syntactic identity after canonicalization of DSL programs, or by an LLM judge when natural-language hypotheses appear.
    Stated in Sec. 3.1; fallback rates reported in App. D. Underpins every win/loss decision.
  • domain assumption The hand-crafted Prolog DSL (App. E.2.1) is an adequate formalization of the intended Zendo rule space for the 22 evaluation games.
    Rules are drawn from Looney Labs cards and compiled into this DSL; the Oracle’s single failure is the deliberately OOD SAME_AMOUNT rule.
  • domain assumption Rejection sampling from the Blender–Prolog pipeline yields unbiased positive/negative/counter-example scenes.
    Sec. 3.1; ensures feedback is not adversarially chosen.
  • ad hoc to paper Structural F1 over rule trees (permutation-invariant matching of commutative subexpressions) is a valid continuous measure of hypothesis quality.
    Introduced for the learning-trajectory plots in Fig. 5; not a standard metric outside this work.
invented entities (2)
  • ZendoWorld environment independent evidence
    purpose: Controlled interactive testbed that jointly requires 3D visual perception, logical rule induction and active experiment design.
    The central contribution; defined by the interaction loop of Fig. 2, the DSL, and the 22-game suite.
  • Expected Information Gain (EIG) metric for proposed scenes independent evidence
    purpose: Quantify how much a player-proposed example reduces posterior uncertainty over the top-K consistent programs.
    Defined in App. C; used to show VLM proposals are near-uninformative (Fig. 6).

pith-pipeline@v1.1.0-grok45 · 30947 in / 3067 out tokens · 32595 ms · 2026-07-10T10:54:30.401215+00:00 · methodology

0 comments
read the original abstract

A central challenge in building intelligent systems is enabling agents to jointly perceive complex inputs, form hypotheses about hidden patterns, and design informative experiments to test them. To study this problem, we propose ZendoWorld, a controlled interactive environment in which agents must infer a logical rule about visual game observations, acquire information by proposing new scenes, and refine their hypotheses based on feedback from the game environment. We evaluate several agents spanning pure VLM reasoning, Bayesian particle filtering, dynamic concept discovery, and neuro-symbolic methods. Our main findings are: (1) high accuracy in predicting labels for observed examples does not imply recovery of the underlying rule; (2) perception and induction are distinct bottlenecks for different agent classes; and (3) VLM-based agents propose near-uninformative experiments, failing to actively reduce hypothesis uncertainty. To compare these results, we collect human data on the task, which reveals a gap in inductive reasoning, particularly for more complex rules. Overall, ZENDOWORLD takes an important step toward evaluating intelligent agents and identifies concrete avenues for improvement, particularly in domains like scientific discovery.

Figures

Figures reproduced from arXiv: 2607.08233 by Antonia W\"ust, Constantin Rothkopf, Inga Ibs, Kevin Ellis, Kristian Kersting, Sophia Koehler, Wasu Top Piriyakulkij, Wolfgang Stammer.

Figure 1
Figure 1. Figure 1: ZENDOWORLD combines percep￾tion, induction, and experimentation in a con￾trolled visual environment. The ability to form, test, and revise hypotheses in light of new evidence is a hallmark of human intelligence, re￾flecting the interplay between perception and inductive reasoning. Cognitive science has long emphasized this process as central to learning, showing how humans ac￾tively explore hypothesis spac… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the ZENDOWORLD setup. (A) The agent is initialized with a set of labeled seed scenes that are consistent with a hidden rule. (B) In the experimentation phase, the agent proposes new scenes, predicts their labels, and earns additional guess attempts for each correct prediction. (C) When the agent decides to commit, it submits a rule hypothesis. If incorrect, a counter example is revealed and the… view at source ↗
Figure 3
Figure 3. Figure 3: Scene examples for the rule "odd number of pieces". Each scene has both a visual representation (image) and a symbolic representation. T ∈ N. Throughout the episode, the agent accumulates a set of labeled observations D = {(xi , yi)}, each consisting of a visual scene xi ∈ R H×W×3 and a binary label yi ∈ {0, 1}, where yi = 1 if the scene satisfies r ∗ and yi = 0 otherwise. The episode is bounded by the amo… view at source ↗
Figure 4
Figure 4. Figure 4: Win rate per complexity class. Hu￾mans excel over AI Agents, the gap becomes es￾pecially large for more challenging tasks. Lines show SEM across seeds and tasks. Human participants perform better than the VLM-based agents in terms of win rate and require a similar amount of turns but achieve lower label accuracy (53.9%). Notably, 7 out of 10 participants that played the OOD task solved it, whereas no agent… view at source ↗
Figure 5
Figure 5. Figure 5: Smoothed structural F1 score by number of observed examples at each rule guess [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean expected information gain (EIG, bits) by example position of experi￾ment within the game, averaged across tasks and seeds. Shaded bands show ±1 SE. The data is from the secondary experiment using symbolic input instead of images. Interestingly, the Oracle Agent (Random) achieves the second-highest EIG, suggesting that diverse scene pro￾posals can reduce the search space more effectively than hypothesi… view at source ↗
Figure 7
Figure 7. Figure 7: Overview of the generation pipeline and example model output. Object Detection Data: We generated 54,252 training images at 640 × 480 resolution, each annotated with per-object attributes (color, shape, orientation, bounding box) and relations (pointing, touching, on_top_of). Object encodings follow the vector format: [ID, COLOR, SHAPE, ORIENTATION, left, right, front, back, top, bottom, pointing, xmin, ym… view at source ↗
Figure 8
Figure 8. Figure 8: Overview of study UI. G.1 Participants We collect data from 19 participants of which five played all six games, two played 4 games, two played three games, six played two games and four played just one game. Participants were recruited from within the institution. The participants play time varied between 10 to 40 minutes per game. G.2 System Architecture The human study is delivered through a purpose-buil… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 72 canonical work pages · 6 internal anchors

  1. [1]

    ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

    ARC Prize Foundation. ARC-AGI-3: A new challenge for frontier agentic intelligence.arXiv preprint arXiv:2603.24621, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Do ai models perform human-like abstract reasoning across modalities?arXiv preprint arXiv:2510.02125, 2025

    Claas Beger, Ryan Yi, Shuhao Fu, Kaleda Denton, Arseny Moskvichev, Sarah W Tsai, Sivasankaran Rajamanickam, and Melanie Mitchell. Do ai models perform human-like abstract reasoning across modalities?arXiv preprint arXiv:2510.02125, 2025

    work page arXiv 2025

  3. [3]

    Bongard.Pattern Recognition

    Mikhail M. Bongard.Pattern Recognition. Spartan Books, 1970

    work page 1970

  4. [4]

    Grounding compositional hypothesis generation in specific instances

    Neil R Bramley, Anslem Rothe, Joshua B Tenenbaum, Fei Xu, and Todd M Gureckis. Grounding compositional hypothesis generation in specific instances. InProceedings of the Annual Meeting of the Cognitive Science Society, volume 40, 2018

    work page 2018

  5. [5]

    Science-gym: a simple testbed for ai-driven scientific discovery.Machine Learning, 115(1):16, 2026

    Mattia Cerrato, Lennart Baur, Jannis Brugger, Sajjad Shumaly, Nicholas Schmitt, Edward Finkelstein, Selina Jukic, Lars Münzel, Felix Peter Paul, Pascal Pfannes, et al. Science-gym: a simple testbed for ai-driven scientific discovery.Machine Learning, 115(1):16, 2026

    work page 2026

  6. [6]

    On the Measure of Intelligence

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  7. [7]

    Goodman, and Laura E

    Claire Cook, Noah D. Goodman, and Laura E. Schulz. Where science starts: Spontaneous experiments in preschoolers’ exploratory play.Cognition, 120(3):341–349, 2011. ISSN 0010-

    work page 2011

  8. [8]

    Probabilistic models of cognitive development

    work page

  9. [9]

    Dreamcoder: growing gener- alizable, interpretable knowledge with wake–sleep bayesian program learning.Philosophical Transactions of the Royal Society A, 381(2251):20220050, 2023

    Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing gener- alizable, interpretable knowledge with wake–sleep bayesian program learning.Philosophical Transactions of the Royal Society A, 381(2251):20220050, 2023

    work page 2023

  10. [10]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge.Advances in Neural Information Processing Systems, 35:18343–18362, 2022

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge.Advances in Neural Information Processing Systems, 35:18343–18362, 2022

    work page 2022

  11. [11]

    Feist and Michael E

    Gregory J. Feist and Michael E. Gorman. The psychology of science: Review and integration of a nascent discipline.Review of General Psychology, 2:3 – 47, 1998

    work page 1998

  12. [12]

    Scaling neural program synthesis with distribution-based search

    Nathanaël Fijalkow, Guillaume Lagarde, Théo Matricon, Kevin Ellis, Pierre Ohlmann, and Akarsh Nayan Potta. Scaling neural program synthesis with distribution-based search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6623–6630, 2022

    work page 2022

  13. [13]

    SLR: Automated synthesis for scalable logical reasoning

    Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer, and Kristian Kersting. SLR: Automated synthesis for scalable logical reasoning. InProceedings of the 64th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL 2026). Association for Computational Lingu...

    work page 2026

  14. [14]

    Douglas R Hofstadter.Fluid concepts and creative analogies: Computer models of the funda- mental mechanisms of thought.Basic books, 1995

    work page 1995

  15. [15]

    Bongard-HOI: Benchmarking few-shot visual reasoning for human-object interactions

    Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-HOI: Benchmarking few-shot visual reasoning for human-object interactions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    work page 2022

  16. [16]

    Image analysis through the lens of chatgpt-4.Journal of Applied Artificial Intelligence, 4(2):31–46, 2023

    Olanrewaju Victor Johnson, Osamah Mohammed Alyasiri, Dua’a Akhtom, and Olabisi Esher Johnson. Image analysis through the lens of chatgpt-4.Journal of Applied Artificial Intelligence, 4(2):31–46, 2023

    work page 2023

  17. [17]

    The robot scientist adam.Computer, 42(8):46–54, 2009

    Ross D King, Jem Rowland, Wayne Aubrey, Maria Liakata, Magdalena Markham, Larisa N Soldatova, Ken E Whelan, Amanda Clare, Mike Young, Andrew Sparkes, et al. The robot scientist adam.Computer, 42(8):46–54, 2009

    work page 2009

  18. [18]

    The nethack learning environment.Advances in Neural Information Processing Systems, 33:7671–7684, 2020

    Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment.Advances in Neural Information Processing Systems, 33:7671–7684, 2020

    work page 2020

  19. [19]

    just another tool for online studies

    Kristian Lange, Simone Kühn, and Elisa Filevich. " just another tool for online studies”(jatos): An easy solution for setup and management of web servers supporting online studies.PloS one, 10(6):e0130834, 2015

    work page 2015

  20. [20]

    Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis

    Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  21. [21]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    Chris Lu, Cong Lu, R. Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery.ArXiv, abs/2408.06292, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Reasoning limitations of multi- modal large language models

    Mikołaj Małki´nski, Szymon Pawlonka, and Jacek Ma´ndziuk. Reasoning limitations of multi- modal large language models. a case study of bongard problems. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  23. [23]

    Visual agentic ai for spatial reasoning with a dynamic api

    Damiano Marsili, Rohun Agrawal, Yisong Yue, and Georgia Gkioxari. Visual agentic ai for spatial reasoning with a dynamic api. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19446–19455. IEEE, 2025

    work page 2025

  24. [24]

    Bongard- logo: A new benchmark for human-level concept learning and reasoning.Advances in neural information processing systems, 33:16468–16480, 2020

    Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongard- logo: A new benchmark for human-level concept learning and reasoning.Advances in neural information processing systems, 33:16468–16480, 2020

    work page 2020

  25. [25]

    A rational analysis of the selection task as optimal data selection.Psychological review, 101(4):608, 1994

    Mike Oaksford and Nick Chater. A rational analysis of the selection task as optimal data selection.Psychological review, 101(4):608, 1994

    work page 1994

  26. [26]

    Bongard-rwr+: Real-world representations of fine-grained concepts in bongard problems.arXiv preprint arXiv:2508.12026, 2025

    Szymon Pawlonka, Mikołaj Małki´nski, and Jacek Ma ´ndziuk. Bongard-rwr+: Real-world representations of fine-grained concepts in bongard problems.arXiv preprint arXiv:2508.12026, 2025

    work page arXiv 2025

  27. [27]

    Cambridge university press, 2009

    Judea Pearl.Causality. Cambridge university press, 2009

    work page 2009

  28. [28]

    Basic Books, 2018

    Judea Pearl.The book of why: The new science of cause and effect. Basic Books, 2018

    work page 2018

  29. [29]

    Doing experiments and revising rules with natural language and probabilistic reasoning.Advances in Neural Information Processing Systems, 37:53102–53137, 2024

    Top Piriyakulkij, Cassidy Langenfeld, Tuan Anh Le, and Kevin Ellis. Doing experiments and revising rules with natural language and probabilistic reasoning.Advances in Neural Information Processing Systems, 37:53102–53137, 2024

    work page 2024

  30. [30]

    α ilp: thinking visual scenes as differentiable logic programs.Machine Learning, 112(5):1465–1497, 2023

    Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. α ilp: thinking visual scenes as differentiable logic programs.Machine Learning, 112(5):1465–1497, 2023

    work page 2023

  31. [31]

    Object centric concept bottlenecks.Advances in Neural Information Processing Systems (NeurIPS), 2025

    David Steinmann, Wolfgang Stammer, Antonia Wüst, and Kristian Kersting. Object centric concept bottlenecks.Advances in Neural Information Processing Systems (NeurIPS), 2025. 11

    work page 2025

  32. [32]

    Hypothesis search: Inductive reasoning with language models

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. Hypothesis search: Inductive reasoning with language models. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  33. [33]

    Benchmarking World-Model Learning with Environment-Level Queries

    Archana Warrier, Dat Nguyen, Michelangelo Naim, Moksh Jain, Yichao Liang, Karen Schroeder, Cambridge Yang, Joshua B Tenenbaum, Sebastian V ollmer, Kevin Ellis, et al. Benchmarking world-model learning.arXiv preprint arXiv:2510.19788, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games

    Nicholas R Waytowich, Devin White, MD Sunbeam, and Vinicius G Goecks. Atari-gpt: Benchmarking multimodal large language models as low-level policies in atari games.arXiv preprint arXiv:2408.15950, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Bongard-OpenWorld: Few-shot reasoning for free-form visual concepts in the real world

    Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, and Yizhou Wang. Bongard-OpenWorld: Few-shot reasoning for free-form visual concepts in the real world. InInternational Conference on Learning Representations, 2024

    work page 2024

  36. [36]

    Pix2code: Learning to compose neural visual concepts as programs

    Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Pix2code: Learning to compose neural visual concepts as programs. InUncertainty in Artificial Intelligence, pages 3829–3852. PMLR, 2024

    work page 2024

  37. [37]

    Rothkopf, and Kristian Kersting

    Antonia Wüst, Tim Tobiasch, Lukas Helff, Inga Ibs, Wolfgang Stammer, Devendra Singh Dhami, Constantin A. Rothkopf, and Kristian Kersting. Bongard in wonderland: Visual puzzles that still make AI go mad? InInternational Conference on Machine Learning, 2025

    work page 2025

  38. [38]

    Synthesizing visual concepts as vision-language programs.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026

    Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Devendra Singh Dhami, Lukas Helff, and Kristian Kersting. Synthesizing visual concepts as vision-language programs.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026

    work page 2026

  39. [39]

    How Far Are AI Scientists from Changing the World?

    Qiujie Xie, Yixuan Weng, Minjun Zhu, Fuchen Shen, Shulin Huang, Zhen Lin, Jiahui Zhou, Zilan Mao, Zijie Yang, Linyi Yang, et al. How far are ai scientists from changing the world? arXiv preprint arXiv:2507.23276, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    RA VEN: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RA VEN: A dataset for relational and analogical visual reasoning. InConference on Computer Vision and Pattern Recognition (CVPR), 2019

    work page 2019

  41. [41]

    TOUCHING

    Weijie Zhou, Xuantang Xiong, Yi Peng, Manli Tao, Chaoyang Zhao, Honghui Dong, Ming Tang, and Jinqiao Wang. Physvlm-avr: Active visual reasoning for multimodal large language models in physical environments. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 12 A Environmental Footprint For the primary experiment, we colle...

    work page 2025

  42. [42]

    A required checkbox confirms informed consent before the study begins

    Instructions and consent.Participants read a study information sheet explaining the purpose, procedure, expected duration (≈45 minutes), data collected, storage, and their right to withdraw at any time. A required checkbox confirms informed consent before the study begins

    work page

  43. [43]

    Tutorial data are excluded from analysis

    Tutorial.Participants complete one full guided game in which on-screen annotations explain the interface elements and game rules. Tutorial data are excluded from analysis

    work page

  44. [44]

    Figure 8 shows the three screens the user sees during the game

    Study games.Participants play a subset of the 22 ZENDOWORLDepisodes following the same protocol as the AI Agents of the main paper. Figure 8 shows the three screens the user sees during the game. The first image (Figure 8a) is during the experimentation phase where the user can build the scene, the second image (Figure 8b) shows the user the scene and ask...

    work page

  45. [45]

    Build a scene to test your hypothesis

    work page

  46. [46]

    Guess whether it follows the rule (YES/NO)

    work page

  47. [47]

    there must be a red block touching a pyramid

    If correct, optionally guess the rule; otherwise a counter-example is shown. Ending.The game ends when the rule is found or after 30 labelled scenes. Tips. • Compare YES and NO scenes. • Change one feature at a time. • Track previous guesses. H Human Study Results The results of the human study are reported in Table 8 combined with the Agent results per t...

    work page

  48. [48]

    and" or

    You may combine short rules using "and" or "or" if needed, but **if a single predicate works, use it.**

    work page

  49. [49]

    The rule must be as short and simple as possible but still accurate

    work page

  50. [50]

    Return **only** the rule - no explanation, no formatting, no extra text

    work page

  51. [51]

    if", "when

    Do not use any conditionals ("if", "when", "only if", etc.) or any text outside the rule itself. ### Output Format: Return **only** a single rule in natural language. Do **not** include explanations or extra text. 26 J.2 VLP Agent The following prompts were used in the beginning of the game to discover predicates over the images: Object Discovery: You are...

    work page

  52. [52]

    **Examine all images carefully** - Look for properties that apply to the relevant objects across the image set

    work page

  53. [53]

    **Identify important properties** - Focus on significant, clearly observable properties that meaningfully describe objects

    work page

  54. [54]

    **Consider property variation** - Properties that vary across images or objects may be particularly noteworthy

    work page

  55. [55]

    **Prioritize meaningful properties** - Choose properties that help distinguish or characterize objects (e.g., color, size, position, orientation, state)

    work page

  56. [56]

    **Return exactly {n} properties** - If fewer notable properties exist, return as many as available

    work page

  57. [57]

    red" rather than

    **Use descriptive names** - Name properties clearly and specifically (e.g., " red" rather than "colored", "horizontal" rather than "oriented") ## Relevant Objects The objects to consider are: {objects} ## Property Categories - **Visual attributes**: colors (red, green, blue, yellow etc.) - **Geometric attributes**: orientations (upright, flat, upside_down...

    work page

  58. [58]

    **Examine all images carefully** - Look for relations that apply to the relevant objects across the image set

    work page

  59. [59]

    **Identify important relations** - Focus on significant, clearly observable actions that meaningfully describe what objects are doing

    work page

  60. [60]

    **Consider relation variation** - Relations that vary across images or objects may be particularly noteworthy (contrasting actions)

    work page

  61. [61]

    **Return exactly {n} relations** - If fewer notable relations exist, return as many as available

    work page

  62. [62]

    running

    **Use descriptive names** - Name relations clearly and specifically (e.g., " running" rather than "moving", "sitting" rather than "positioned", "touching" rather than "close") ## Relevant Objects The objects to consider are: {objects} ## Output Requirements - Return a Python list assigned to variable ‘actions‘ - Include only the Python code, no explanatio...

    work page

  63. [63]

    Only use objects/properties from the provided lists

    work page

  64. [65]

    car", "person

    No explanations or additional text ## Output Format ‘‘‘python objects = [ [’object_name’, ’property1’, ’property2’, ...], [’object_name’, ’property1’], 28 ... ] ‘‘‘ **If no valid objects:** ‘objects = [[]]‘ ## Examples **Example 1** - Objects: ["car", "person", "tree"] - Properties: ["red", "tall", "small", "standing"] - Image: Red car under tall tree wit...

    work page

  65. [66]

    Only use properties from the provided lists

    work page

  66. [67]

    Return empty list if no valid objects found

    work page

  67. [68]

    car", "person

    No explanations or additional text ## Output Format ‘‘‘python properties = [ [’property1’, ’property2’, ...], 29 [’property1’], ... ] ‘‘‘ **If no valid objects:** ‘objects = [[]]‘ ## Examples **Example 1** - Objects: ["car", "person", "tree"] - Properties: ["red", "tall", "small", "standing"] - Image: Red car under tall tree with small standing person ‘‘‘...

    work page

  68. [69]

    Only use values from the provided lists for each position

    work page

  69. [70]

    Each triple must have exactly 3 elements: [subject, action, object] 30

    work page

  70. [71]

    If the same interaction occurs multiple times in the image, include one entry per occurrence

    work page

  71. [72]

    Return empty list if no valid triples found

    work page

  72. [73]

    block",

    No explanations or additional text ## Output Format ‘‘‘python actions = [ [’subject1’, ’action1’, ’object1’], [’subject2’, ’action2’, ’object2’], ... ] ‘‘‘ **If no valid interactions:** ‘actions = [[]]‘ ## Examples **Example 1** - Objects: ["block", "pyramid", "wedge"] - Properties: ["blue", "red"] - Actions: ["touching", "grounded"] - Image: Two blue blo...

    work page