{"id":"fc3c5310-0b34-47f3-9e77-68a677f73445","arxiv_id":"2411.11672","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A thesis arguing that autonomous symbol interpretation is the key missing capability for artificial scientists, demonstrated through Olivaw, Explanatory Learning on Odeen, the training-free ASIF model, and the Symbol Interpretation Task.","lead":"This PhD thesis compiles four published AI research projects that together argue machines can become 'artificial scientists' only if they learn to interpret and explain symbols autonomously. It retraces a path from a self-taught Othello agent, to a new 'explanatory learning' framework and benchmark, to a training-free multimodal model, to a benchmark showing LLMs fail at symbol interpretation.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Odeen 'scientist problem' assumes a representative 32-observation D0 (Sec 3.2.1), and Sec 3.7 disclaims modeling how D0 is acquired; the 77.7% discovery result is therefore contingent on a favorable starting condition, undercutting the thesis that autonomous symbol interpretation is the…","rationale":"Good-faith reading: the thesis is a well-structured compilation of four peer-reviewed works. The individual chapters are credible; Olivaw's Othello results, ASIF's zero-shot retrieval, and the SIT benchmark are plausible and largely published. The CRN versus empiricist gap on Odeen is large, and the qualitative examples (e.g., 40% vs 4% on 'exactly two' rules) are striking. Nevertheless, the central thesis — that an artificial scientist must learn its own interpreter and that this is the keystone problem — is an extrapolation from a toy environment. The most defensible version of the claim would be: in the EL formalization, learning an interpreter beats using a fixed one, and this is a useful design principle for machine discovery. The stronger 'only if' claim about what an artificial scientist requires is not established by the Odeen experiment. The specific load-bearing weakness is the representativity assumption. It is not hidden: Sec 3.7 states the limitation. But because it is stated, it does not rescue the broad conclusion; it delimits it. The reader's weakest_assumption identifies exactly this. No other concern is equally load-bearing: the lack of error bars in Table 3 is unlikely to change the 55-point gap; Table 4's learned-versus-hardcoded interpreter difference, while small, is directionally supportive; the ASIF ImageNet-validation tuning is a disclosed methodological caveat, not a threat to the EL chapter's central comparison. Therefore the verdict should remain CONDITIONAL: the internal claims are plausible, but the thesis-level conclusion requires either a demonstration with actively acquired D0 or a more cautious statement of scope.","tokens_in":51091,"tokens_out":7360,"duration_ms":72898,"concrete_test":"Use the trained CRN and interpreter from Sec 3.5 (or retrain identically) and re-run the test protocol of Table 3 with D0 of size k=32 drawn uniformly at random from U, instead of the curated representative set of App A.1. Track NRS; if it drops substantially (e.g. below the empiricist baseline or by more than 10 points), the headline result depends on the representativity guarantee. As a second check, measure the fraction of random 32-element D0 that satisfy the representativity definition for a sample of test phenomena; if this fraction is low, the Odeen test set is not representative of unassisted scientific discovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical pillar of the thesis is Table 3: CRNs reach 77.7% NRS on 1132 unseen Odeen phenomena from k=32 labelled observations, versus 22.5% for the best empiricist baseline. This result is obtained inside the 'scientist problem' defined in Sec 3.2.1, which requires two guarantees: P0 is explainable in L, and D0 is representative for P0, i.e. for every other explainable phenomenon P there is an xi in D0 with 1_P0(xi) != 1_P(xi). Without representativity the problem is ill-posed: many rules fit D0, and 'discovery' is underdetermined. The appendix (A.1) shows D0 is not random: the first 10 structures are chosen by searching for near-miss pairs (similar structures, different labels), and the rest are selected 'to ensure the lack of ambiguity on the board'. So the 77.7% number measures performance under a deliberately favorable sampling distribution. The thesis itself acknowledges in Sec 3.7 that it does not model how the scientist actively acquires D0 and leaves that to 'future work'. The central claim — that a true artificial scientist requires autonomously learned symbol interpretation — is inferred from this success. The inference is an overreach if the same method cannot discover explanations when D0 must be actively obtained, as in real experimental science. This is a concern about external validity and scope, not an internal inconsistency: the formal EL framework is coherent, but the broad 'artificial scientist' conclusion is not supported by the narrow, favorable-data demonstration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD thesis argues that realizing an artificial scientist requires a machine to learn its own interpreter—the mapping between explanatory symbols and observations—rather than relying on a fixed, human-provided interpreter. The argument is developed across four empirical chapters: (1) Olivaw, an AlphaGo-Zero-like Othello agent that reaches world-class play with minimal search but cannot communicate its knowledge; (2) the Explanatory Learning (EL) framework, the Odeen benchmark, and Critical Rationalist Networks (CRNs), which pair a conjecture generator with a learned interpreter and report 77.7% Nearest Rule Score on 1,132 unseen phenomena from 32 observations each, versus 22.5% for the best empiricist baseline (Table 3); (3) ASIF, a training-free procedure that couples frozen unimodal encoders with roughly 1.6M image-text pairs, reaching CLIP-competitive zero-shot accuracy at a fraction of the multimodal data budget; and (4) the Big-Bench Symbol Interpretation Task, on which LLMs score at chance while the best human raters reach perfection. The thesis concludes that autonomous symbol interpretation is the keystone open problem for machine-assisted scientific discovery.","tokens_in":51423,"tokens_out":20314,"duration_ms":193535,"significance":"If the results hold, the manuscript makes a substantive contribution: it articulates Explanatory Learning as a distinct formalization (explanation-observation pairs with no given interpreter), releases a parametric benchmark (Odeen) with a controlled compositional split (rules containing 'exactly two' are withheld from training and tested), and shows on that benchmark that a two-stage rationalist architecture beats end-to-end empiricist baselines of similar size. The finding that a learned interpreter can match or beat the ground-truth hardcoded interpreter on Odeen (Table 4) is genuine, non-obvious, and directly supports the thesis's central intuition. ASIF provides a simple, reproducible, transparent baseline for multimodal alignment with a disclosed data budget, and the SIT results are a clean, falsifiable negative result for LLM symbol interpretation. The manuscript ships unusually complete supporting material: training hyperparameters for Olivaw, cost analyses for CRNs (Tables 9-10), backbone specifications for ASIF, and released code for the main claims.","major_comments":[{"comment":"The Odeen scientist problem is defined by two guarantees (Sec. 3.2.1): P0 is explainable in L, and D0 is representative for P0, i.e., every other explainable phenomenon differs from P0 on at least one item of D0. This representativity guarantee is load-bearing: the CRN's test-time selection (Sec. 3.4.1) picks the conjecture with the highest hit rate on D0, and that procedure is well-posed precisely because D0 pins down P0 uniquely among the roughly 24k rules. The headline result of Table 3 (CRN NRS 0.777 vs. Emp-C 0.225) is therefore measured under a favorable data-generation protocol: Appendix A.1 states that the first 10 structures of each D0 are chosen by searching for near-miss pairs (similar structures, different labels) and the remaining 22 are selected to ensure the lack of ambiguity on the board. No sensitivity analysis is reported: with a uniformly random D0 of size 32, the representativity guarantee will generally fail (many rules agree with any 32 observations), and the measured discovery rate should be expected to drop, possibly sharply. Because the thesis's central claim is inferred from the magnitude of the CRN advantage, this point is load-bearing, and the thesis itself concedes in Sec. 3.7 that acquisition of D0 is not modeled. I ask the authors to either (a) add experiments that degrade the D0 guarantee (uniformly random D0, smaller k, noisy labels) and report CRN-vs-empiricist performance under those conditions, or (b) visibly narrow the claims in the Abstract and Introduction to the conditional setting in which representative observations are supplied. This is a scope/external-validity concern, not an internal inconsistency; the EL formalism itself is coherent.","section":"Sec. 3.2.1, Sec. 3.7, Appendix A.1"},{"comment":"The keystone claim is stated unconditionally: 'a true artificial scientist can only emerge when a machine can autonomously interpret symbols' (Sec. 1.1), and the Abstract presents the Odeen success as establishing it. The evidence actually supports a conditional statement: within the EL framework on the Odeen benchmark, a learned interpreter matches or slightly outperforms the ground-truth hardcoded interpreter (Table 4, 0.813 vs. 0.801 at n = 1438 rules), and CRNs beat the empiricist baselines (Table 3) — with the caveat that the learned-versus-hardcoded advantage reverses at n = 500 rules (Table 4: 0.354 vs. 0.377 at 10K structures), so the benefit is regime-dependent. No experiment in the thesis connects the Odeen results to the ASIF or SIT results in a way that tests whether interpreter learning is the limiting factor in those settings; the inference from three independent toy-domain results to 'the central open problem of machine-assisted scientific discovery' is an inductive leap. I recommend reformulating the keystone claim to be conditional on the EL setting and on the representative-D0 assumption raised in the previous comment, or extending the evidence to at least one environment with substantially different structure so that the universality claim is tested rather than asserted.","section":"Abstract, Sec. 1.1, Sec. 6.1"}],"minor_comments":[{"comment":"The Abstract says SIT is 'fully solved by humans,' but Table 6 reports an average human score of 36.8% (chance 20%) with only the best raters reaching 100%; suggest 'solved by the best human raters, with average human performance well above chance.'","section":"Abstract and Sec. 5.2"},{"comment":"The claim that SIT shows 'the largest performance gap between human participants and LLMs' should specify that this holds for gaps in favor of humans, since several Big-Bench Lite tasks in the same table show larger absolute gaps in favor of models (e.g., auto_debugging: PaLM 540B 38.2 vs. human 13.7).","section":"Sec. 5.2 / Table 6"},{"comment":"Typo: 'hradcoded' should read 'hardcoded'; the same error appears in the prose accompanying the table.","section":"Table 3 caption"},{"comment":"The disclosure that k = 800 and p = 8 were tuned on a subset of the ImageNet validation set appears only in a table footnote; it deserves a sentence in the main text, with a pointer to the robustness analysis in Appendix B.3.","section":"Sec. 4.4.2 / Table 5"},{"comment":"The Odeen results are reported without variance estimates; since the training sampler and the conjecture generator are stochastic, please report means and standard deviations over at least a few training seeds for the central Table 3/Table 8 comparisons.","section":"Sec. 3.5 / Table 3"},{"comment":"Several spacing and typographical artifacts should be cleaned up, e.g., 'O livaw', 'T ransformer', 'VITb16', 'mutimodal' (Sec. 4.5.4), and 'sensibility' in the Appendix B.3 heading, which should be 'sensitivity.'","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a compilation of previously published results (IEEE ToG 2022, ICML 2023 workshop, NeurIPS 2023, TMLR 2023), and the editorial question is whether it is evaluated as an archival synthesis (in which case the D0-sensitivity and claim-scope issues in the major comments should be addressed) or as a thesis-style document in which prior peer review carries the experimental burden. The representativity concern is not new to this version — it is citable from the published EL paper — and the thesis discloses it in Sec. 3.7, but the Abstract omits the caveat; the same is true of the 'fully solved by humans' phrasing, which overstates the average human score in Table 6. Since the Abstract is what will be quoted, these two phrasings are more than cosmetic. On scope: for a cs.AI venue the thesis-level narrative is in scope, but if the venue expects new technical results, most of the technical content has already appeared elsewhere and the novel synthesis is the part that needs the revisions above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a PhD thesis that assembles four already-published papers (Olivaw, Explanatory Learning, ASIF, and the Big-Bench SIT task) into a narrative about building an artificial scientist. The individual chapters are credible and mostly well-executed; the overarching claim that autonomous symbol interpretation is the missing capability is programmatic and overreaches the evidence presented.\n\nWhat is new here is the framing, not the results. Relative to the cited papers, the thesis adds an integrative story: from Olivaw's silent competence, to the EL formalization, to ASIF's transparent interpreter, to the SIT result that LLMs fail to interpret novel symbols. The EL framework and the Odeen benchmark are genuinely useful formalizations, and the CRN-versus-empiricist comparison (77.7% vs 22.5% NRS) is a striking demonstration that a rationalist approach can beat end-to-end learning when the language is learnable. ASIF is a strong cheap baseline, and SIT is a real contribution to Big-Bench. Credit is also due for the unusually clear writing and for the explicit acknowledgment of limitations.\n\nThe soft spots are real but not fatal. The most important is flagged by the author himself in Sec 3.7: the \"scientist problem\" assumes a representative D0 of 32 observations per phenomenon, with a guarantee that it disambiguates the target rule. The 77.7% discovery rate is measured under this deliberately favorable sampling (Appendix A.1 confirms the first 10 structures are chosen as near-miss pairs). The thesis does not model how the scientist actively obtains D0, which is the essence of experiment design in real science. So the central \"artificial scientist\" conclusion rests on a narrower demonstration than the rhetoric suggests. Second, Table 3 has no error bars or repeated seeds; this is addressable, but a reviewer would want them. Third, ASIF's hyperparameters were tuned on ImageNet validation; this is disclosed and the appendix shows mild sensitivity, but it slightly undercuts the \"no training\" framing.\n\nWho this is for: readers interested in explainable AI, rationalist versus empiricist learning, or benchmarks for symbol grounding. It deserves a serious referee—probably as a monograph or a position piece—but the author should be pushed to either temper the artificial-scientist claim or add experiments with active data acquisition. I would not cite the thesis itself; I would cite the underlying papers.","headline":"A well-written thesis compiling four solid papers whose central 'artificial scientist' claim outruns the evidence, mainly because the flagship Odeen result assumes a favorable, pre-given data set.","tokens_in":51999,"tokens_out":3201,"would_cite":false,"duration_ms":30639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An artificial scientist must learn to interpret symbols, not inherit an interpreter.","keywords":["artificial scientist","explanatory learning","symbol interpretation","Critical Rationalist Networks","Odeen benchmark","multimodal alignment","zero-shot classification","large language models"],"falsifier":"Run the Odeen scientist problem twice on the same test phenomena, once with the standard representative 32-observation sets and once with sets deliberately chosen so that each D0 is ambiguous between the target rule and at least one other rule in the grammar; if Nearest Rule Score stays near 0.777 in the ambiguous condition, the representativity premise is not load-bearing, and if it collapses toward the empiricist baseline, the discovery claim depends on the favourable starting point.","tokens_in":50823,"feed_emoji":"🔬","tokens_out":6368,"duration_ms":62780,"temperature":0.7,"pith_summary":"Across three connected studies, the thesis tries to establish what a machine needs in order to make original, communicable scientific discoveries. Its central claim is that the missing piece is autonomous symbol interpretation: the machine must learn the mapping from explanation strings to observations from paired examples, rather than being given a rigid interpreter for its explanation language. The empirical anchor is the Odeen benchmark, a small universe of 117,649 structures and roughly 24,794 grammar-generated rules, where Critical Rationalist Networks discover the correct explanation for 77.7% of unseen phenomena using only 32 labelled observations per phenomenon, against 22.5% for the best empiricist baseline. The thesis goes on to argue that the same interpretation problem appears in multimodal models and in large language models, whose chance-level performance on the Symbol Interpretation Task marks the largest human-machine gap in a large multi-model benchmark study.","feed_headline":"Machines must invent their own meanings to become scientists","feed_subtitle":"Learning to read explanation symbols beats receiving an interpreter—and today's AI fails that test.","key_machinery":"The load-bearing mechanism is the learned interpreter, a transformer encoder that maps a rule string and a structure to a label, trained only on paired explanation-observation data. Around it, a Critical Rationalist Network adds a stochastic Conjecture Generator that proposes candidate explanations from the 32 observations of a new phenomenon; at test time the network generates t conjectures, scores each with the interpreter on the labelled observations, and tags unseen structures with the best-scoring explanation. The second machinery, used for multimodal interpretation, is the relative-representation 'as if' step: represent an image by its similarities to a set of image-text pairs and then treat that vector as if it were the representation of the image's ideal caption, which aligns frozen unimodal encoders without training. The Odeen environment supplies the testbed: six-slot structures over a small alphabet, a grammar of roughly 24,794 rules, and a semantics encoded by a hardcoded interpreter that the learner is never allowed to use.","core_discovery":"The central claim is that an artificial scientist is possible only when the machine can acquire an interpreter for the language of explanations from data. In the Explanatory Learning formulation, explanations are free strings with no grammar or interpreter supplied, and the learner must build the map from symbols to phenomena out of explanation-observation pairs. The thesis reports that on Odeen, Critical Rationalist Networks—a Conjecture Generator paired with a learned Interpreter—achieve a Nearest Rule Score of 0.777 on 1,132 unseen phenomena with 32 observations each, whereas the strongest end-to-end empiricist model reaches only 0.225 and rarely produces a usable explanation (R-Acc 0.035). It further reports that a learned interpreter can beat the hardcoded ground-truth interpreter in a CRN, because it tolerates ambiguous or ungrammatical conjectures, and that large language models scored no better than random chance on the companion Symbol Interpretation Task.","pith_inferences":["The thesis's 77.7% discovery rate assumes the 32 initial observations are representative, i.e., they distinguish the target phenomenon from every other language-explainable phenomenon; the thesis itself flags that it does not model how the scientist actively acquires those observations, so transfer to uncurated settings is the main open risk.","A direct extension would turn the scientist problem into an active-learning loop: let a CRN choose which structures to observe and test whether the discovery rate rises or the required number of observations falls; this would also operationalize curiosity, which the conclusion names as missing from current models.","ASIF's success suggests that much of 'interpretation' in multimodal models is stored in the training pairs rather than in the weights, predicting that ASIF performance tracks the semantic distance between coupled dataset and downstream task—a measurable quantity.","The Symbol Interpretation Task offers a cheap falsifier for claims about LLM reasoning: rerun it on newer models; the thesis's position implies they should still perform at chance unless they incorporate an explicit multi-step uncertainty or symbol-redefinition mechanism."],"forward_implications":["Separating conjecture generation from conjecture testing by a learned interpreter gives a concrete architecture for few-shot scientific explanation, with time-varying computation controlled by the number of generated conjectures.","A learned interpreter can outperform the ground-truth one because it assigns meaning to malformed hypotheses; discovery is therefore not tied to syntactic correctness of the explanation language.","Multimodal understanding comparable to contrastively trained models can be assembled from frozen unimodal encoders plus a small set of coupled pairs, making interpretation editable and transparent—adding or removing knowledge is just adding or deleting stored embeddings.","Because large language models fail the Symbol Interpretation Task at random chance while humans solve it, scaling current architectures is not by itself a path to the symbol malleability an artificial scientist needs."],"supporting_citations":[{"why":"Supplies the AlphaGo Zero paradigm that Olivaw replicates cheaply, motivating the discovery-without-communication problem that the thesis then addresses.","marker":"Silver et al. 2017"},{"why":"Provides the CLIP model as the key instance of a learned interpreter built from image-text pairs, the main baseline and target for ASIF.","marker":"Radford et al. 2021"},{"why":"Supplies the earlier formalization of the scientist problem that Explanatory Learning generalizes by removing the fixed interpreter.","marker":"Angluin 1987"},{"why":"Provides the critical-rationalist epistemology that structures the CRN design: hypotheses are conjectures to be accepted or rejected, not parameters to be tuned.","marker":"Popper 1935"},{"why":"Gives LiT, the locked-image-encoder baseline whose data and evaluation setup ASIF matches with far fewer multimodal pairs.","marker":"Zhai et al. 2022"},{"why":"Reports the Big-Bench results, including the Symbol Interpretation Task with the largest human-machine gap that the thesis uses to test LLM symbol interpretation.","marker":"Srivastava et al. 2023"}],"fun_headline_variants":["AI scientists must invent their own symbols","Learned interpreters beat fixed ones for AI discovery","For AI to do science, it must build its own meanings","LLMs fail at symbol interpretation—AI needs a new route","Artificial scientists need to craft their own interpretations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learner is handed 32 labelled observations of the new phenomenon that are guaranteed to distinguish it from every other explainable phenomenon, rather than having to find or design those observations itself.","fun_headline_variants_meta":{"raw":{"variants":["AI scientists must invent their own symbols","Learned interpreters beat fixed ones for AI discovery","For AI to do science, it must build its own meanings","LLMs fail at symbol interpretation—AI needs a new route","Artificial scientists need to craft their own interpretations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1297,"prompt_tokens":981,"completion_tokens":316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":597,"tokens_out":316,"duration_ms":3970,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:15:30.818760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Odeen scientist problem twice on the same test phenomena, once with the standard representative 32-observation sets and once with sets deliberately chosen so that each D0 is ambiguous between the target rule and at least one other rule in the grammar; if Nearest Rule Score stays near 0.777 in the ambiguous condition, the representativity premise is not load-bearing, and if it collapses toward the empiricist baseline, the discovery claim depends on the favourable starting point.","supporting_citations":[],"review_version":1}