Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

From Tool Calling to Symbolic Thinking: LLMs in a Persistent Lisp Metaprogramming Loop

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can grow their own tools in a live Lisp loop

desk verdict A clearly written conceptual proposal for an LLM + Lisp REPL loop, but the core pause-and-resume injection mechanism is underspecified and the paper has no implementation or evaluation. read the letter →

arxiv 2506.10021 v1 pith:YR3BCZNR submitted 2025-06-08 cs.PL cs.AI

Jordi de la Torre This is my paper
classification cs.PLcs.AI
keywords largelanguagemodelsLispREPLmiddlewaremetaprogrammingtoolcreationpersistentmemorysymbolicAIself-extension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an architecture in which a large language model writes Lisp expressions inside its own output stream, a middleware layer executes them in a persistent Lisp REPL, and the results are fed back so generation continues as if the computation had been part of the response. The intended consequence is that tools, functions, and macros defined by the model survive across conversation turns, letting the model build an evolving toolkit rather than relying on fixed, predefined APIs. The paper argues Lisp's homoiconicity and metaprogramming make it the right substrate, and it lays out the design principles for such a system. This is a conceptual contribution; the author states it has not yet been implemented or empirically validated.

What carries the argument

The load-bearing mechanism is the middleware layer: a stream-aware proxy that watches the LLM's token output, detects a `<lisp>...</lisp>` block, pauses generation, sends the enclosed expression to a persistent Lisp REPL, and inserts the evaluation result back into the generation stream. This single interception loop converts the model from a text emitter into a control loop that can compute, reflect, and accumulate state while producing language. Lisp's homoiconicity, meaning code is represented uniformly as data, is the enabling property because it makes the embedded expressions parseable, manipulable, and extensible through macros.

What would settle it

Take the proposed system with a representative LLM and run a fixed battery of prompts that require defining a Lisp function in one turn and calling it in a later turn. If any meaningful fraction of the generated `<lisp>...</lisp>` blocks fails to parse or evaluate, or if the generation stream cannot be resumed cleanly after an error, then the claimed persistent self-tooling property does not hold.

Watch

Extended reading notes

Core claim

The central claim is that the boundary between an LLM and a programming environment can be made permeable: whatever code the model emits inside a reserved tag, such as `<lisp>...</lisp>`, is treated not as text but as a live computation. The middleware pauses token generation, evaluates the expression in a persistent Lisp process, and injects the resulting value back into the stream, after which generation resumes. Because the REPL keeps state, the model can define a function in one turn and call it in a later turn, and can inspect, debug, and redefine its own definitions. The paper presents this as a framework for self-extending agents in which the model is an active participant in programming rather than a code generator.

Load-bearing premise

The whole design rests on the assumption that a current LLM can produce syntactically valid, semantically meaningful Lisp inside self-generated tags reliably enough for the middleware to evaluate without breaking the response flow.

Editorial extensions

If this is right

  • Functions and macros defined in one turn persist, so the model can accumulate a growing library of reusable tools over a session.
  • The model can query its own environment, inspect function source, and redefine procedures, enabling real-time debugging and self-aware generation.
  • Macros and higher-order functions let the model create domain-specific languages and custom control structures suited to its tasks.
  • A live, evaluable REPL could serve as a grounded feedback source for reinforcement learning on reasoning, letting the model test hypotheses and refine strategies.
  • The same persistent state could later connect to external tools such as search engines or embedding encoders, extending memory and contextual grounding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit is that generation becomes a blocking interactive process: every tag insertion adds latency, so practical systems need policies for deciding when to evaluate eagerly versus defer.
  • One could benchmark this architecture against conventional function calling on a multi-turn task where the model must build and reuse tools; if the success rate is not higher, persistence alone would not justify the added complexity.
  • The same middleware design could be ported to any homoiconic or scriptable runtime, so the Lisp-specific claims are one instance of a more general pattern of giving models a persistent executable workspace.
  • Allowing self-modification introduces failure modes such as infinite loops and destructive redefinition, making sandboxing a core correctness issue rather than an optional safety feature.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a conceptual architecture for coupling a large language model with a persistent Common Lisp REPL via a middleware layer that intercepts <lisp> expressions in the model's output stream, evaluates them, and injects the results back into the generation stream. The authors argue that this enables the model to build persistent tools, introspect and redefine its environment, and engage in metaprogramming. The paper explicitly states that it is a conceptual framework and reports no experimental results or implementation.

Significance. If the proposed mechanism could be made precise and workable, the architecture would offer a concrete way to combine neural generation with symbolic, stateful computation, potentially enabling long-lived self-extending agents. The paper is clearly written and honestly scoped, and the background on Lisp and symbolic AI is competently assembled. However, as written, the central mechanism--pause-and-resume injection of evaluated results--is not a well-defined operation in autoregressive decoding, and the paper provides no specification or trace that would make it implementable. The claimed capabilities therefore remain at the level of aspiration rather than demonstrable design.

major comments (2)
  1. [Section 4, paragraph 2] The description of the middleware 'pauses the generation process', inserts the evaluated result 'in place of the original tag', and resumes 'as if the result had been part of the model's initial output' is underspecified with respect to autoregressive decoding. In a transformer, the model conditions on all previously generated tokens; if the middleware stops exactly at the closing tag, appends the result, and then continues sampling, the model's earlier decision to emit the Lisp block was made without knowledge of the result, so the 'as if' semantics is false. If, instead, the middleware lets generation proceed past the tag and then replaces the block, the subsequent tokens were generated without conditioning on the result. The paper does not state which of these protocols is intended, nor does it explain how the key/value cache or the input context is updated. This is load-bearing because the central claim that the LLM can 'define, invoke, and evolve' tools through live execution depends on this loop being well-defined. Please provide a concrete protocol (e.g., stop at the closing tag, append the result as text, then continue generating; or generate a placeholder and do a second pass) and justify that the described semantics is achievable.
  2. [Section 5 and Section 6] The paper asserts that the architecture yields 'stateful tools', 'reflective' capabilities, 'metaprogramming', 'generative self-extension', and integration with reinforcement learning, but these capabilities are not demonstrated or even illustrated with a single complete interaction trace. Because the underlying injection mechanism is undefined, the capabilities remain restatements of Lisp's well-known properties rather than features of a working design. Provide at least one end-to-end example--from a user prompt, through Lisp code generation, evaluation, result injection, and final response--showing how a specific capability, such as defining a persistent function that is used in a later turn, is realized. Without such a trace, the architecture is not sufficiently specified to guide implementation, which undermines the paper's stated purpose of providing 'architectural principles to guide future implementations'.
minor comments (5)
  1. [Section 4, first paragraph] There are missing spaces in 'The middleware layeroperates' and 'thelanguage model backend'; please fix these typesetting issues.
  2. [Section 7] The sentences 'it also safety concerns' and 'To mitigate this risks' contain grammatical errors; they should read 'it also raises safety concerns' and 'To mitigate these risks.'
  3. [Section 5, fourth paragraph] The phrase 'Oneparticularlyforward-lookingcapability' is missing spaces; also, the discussion of 'generative self-extension' is marked speculative but still presented as a capability of the system, so it should be more explicitly separated from the concrete claims.
  4. [Section 3 and Section 3.1] The paper uses 'Lisp' and 'Common Lisp' almost interchangeably; clarify whether the proposed architecture relies on Common Lisp-specific features (such as CLOS or the condition system) or would work with any Lisp dialect.
  5. [Section 2] The historical background is disproportionately long relative to the paper's contribution; consider condensing the survey to focus on the aspects directly relevant to the proposed architecture.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an explicitly conceptual architecture proposal with no fitted parameters, no self-citation chain, and no derivation that reduces to its own inputs.

full rationale

This paper is a design-position essay, not an empirical study. It contains no fitted parameters, no quantitative predictions, and no equations whose output is defined by an input. The load-bearing architectural claims (LLM generates <lisp> blocks; middleware evaluates them in a persistent REPL; results are re-inserted into the generation stream) are specified constructively in Section 4 and are not derived from any prior result. The 'capabilities and benefits' in Section 5 (stateful tools, reflection, metaprogramming, self-extension) are transparent consequences of the design choice to use a persistent, homoiconic Lisp REPL, as argued in Sections 3 and 3.1; this is a design rationale, not a circular derivation. The paper contains no self-citations by the author (all references are external classics), so no self-citation chain is load-bearing. Two passages explicitly assert the limitations of the work: the footnote on page 1 ('This paper presents a conceptual framework intended to guide future implementations rather than report experimental results') and Section 8 ('this design remains conceptual and has yet to be implemented or empirically validated'). These statements reduce the evidential weight of the claimed capabilities but do not indicate circularity; an architecture that has not been built cannot be accused of deriving its own output from a fitted input. The skeptical concern about Section 4 (whether autoregressive decoding can 'resume as if the result had been part of the model's initial output') is a genuine architectural underspecification, but it is a completeness/correctness issue, not a circularity issue, and under the hard rules it is not scored here. Overall, there is no step in the paper where a prediction reduces by construction to a fitted parameter, a self-citation, or a definition.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on unverified assumptions about LLM code-generation reliability, middleware behavior, and Lisp's superiority. No free parameters are fit and no invented entities are introduced; the proposed components are standard software architecture.

assumptions (3)
  • domain assumption LLMs can reliably generate well-formed Lisp code in their output.
    Section 4 depends on the model producing syntactically correct expressions within <lisp> tags; no evidence is provided, and failure cases are not analyzed.
  • domain assumption Streaming middleware can detect and execute complete <lisp> blocks without breaking generation.
    Section 4 assumes token-stream parsing can recognize tag boundaries and pause/resume generation cleanly, which is nontrivial and untested.
  • domain assumption Lisp's homoiconic syntax and macro system offer practical advantages for LLM tool-building.
    Section 3 argues Lisp is uniquely suited based on historical and syntactic grounds, but no comparative evaluation supports this claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Tool Calling to Symbolic Thinking: LLMs in a Persistent Lisp Metaprogramming Loop." pith.science (2026). https://pith.science/paper/YR3BCZNR

@misc{pith2026250610021,
  author       = {Pith},
  title        = {Pith review of: From Tool Calling to Symbolic Thinking: LLMs in a Persistent Lisp Metaprogramming Loop},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YR3BCZNR}},
  note         = {Machine review of arXiv:2506.10021}
}
read the original abstract

We propose a novel architecture for integrating large language models (LLMs) with a persistent, interactive Lisp environment. This setup enables LLMs to define, invoke, and evolve their own tools through programmatic interaction with a live REPL. By embedding Lisp expressions within generation and intercepting them via a middleware layer, the system allows for stateful external memory, reflective programming, and dynamic tool creation. We present a design framework and architectural principles to guide future implementations of interactive AI systems that integrate symbolic programming with neural language generation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Workflow as Knowledge: Semantic Persistence for LLM-Mediated Workflows

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A conceptual model proposes representing LLM workflow definitions, instances, and inference records as persistent typed objects in a shared knowledge substrate, distinguishing deterministic derive from LLM-mediated infer.

Reference graph

Works this paper leans on

25 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Programs with common sense

    John McCarthy. Programs with common sense. Technical report, MIT, 1959. Presented at the Teddington Conference on Mechanisation of Thought Processes

    work page 1959

  2. [2]

    Recursive functions of symbolic expressions and their computation by machine, part i.Communications of the ACM, 3(4):184–195, 1960

    John McCarthy. Recursive functions of symbolic expressions and their computation by machine, part i.Communications of the ACM, 3(4):184–195, 1960

    work page 1960

  3. [3]

    Procedures as a representation for data in a computer program for understanding natural language

    Terry Winograd. Procedures as a representation for data in a computer program for understanding natural language. Technical Report AI-TR-235, MIT Artificial Intelligence Laboratory, 1971

    work page 1971

  4. [4]

    A framework for representing knowledge

    Marvin Minsky. A framework for representing knowledge. Technical Report Memo 306, MIT AI Lab, 1974

    work page 1974

  5. [5]

    North-Holland, 1984

    Henk P Barendregt.The Lambda Calculus: Its Syntax and Semantics. North-Holland, 1984

    work page 1984

  6. [6]

    An introduction to lambda calculi for computer scientists.Electronic Notes in Theoretical Computer Science, 125:1–57, 2005

    Chris Hankin. An introduction to lambda calculi for computer scientists.Electronic Notes in Theoretical Computer Science, 125:1–57, 2005

    work page 2005

  7. [7]

    Review of: Rule-based expert systems: The mycin experiments of the stanford heuristic programming project.Artificial Intelligence, 1985

    William R Swartout. Review of: Rule-based expert systems: The mycin experiments of the stanford heuristic programming project.Artificial Intelligence, 1985. Book review of Buchanan and Shortliffe (1984), Addison-Wesley

    work page 1984

  8. [8]

    Digital Press, 1990

    Guy L Steele.Common LISP: The Language. Digital Press, 1990

    work page 1990

Show all 25 references
  1. [9]

    The perceptron: a probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386, 1958

    Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386, 1958

    1958

  2. [10]

    MIT press, 1992

    Hubert L Dreyfus.What computers still can’t do: A critique of artificial reason. MIT press, 1992

    1992

  3. [11]

    Learning representations by back-propagating errors.nature, 323(6088):533–536, 1986

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors.nature, 323(6088):533–536, 1986

    1986

  4. [12]

    Reducing the dimensionality of data with neural networks.science, 313(5786):504–507, 2006

    Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks.science, 313(5786):504–507, 2006

    2006

  5. [13]

    Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    2012

  6. [14]

    Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems, 30, 2017

    2017

  7. [15]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018. OpenAI Technical Report

    2018

  8. [16]

    Language models are unsupervised multitask learners.OpenAI Blog, 1(8):1–9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI Blog, 1(8):1–9, 2019

    2019

  9. [17]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. 7

    2001 arXiv

  10. [18]

    Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877– 1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877– 1901, 2020

    1901

  11. [19]

    Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems, 30, 2017

    2017

  12. [20]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  13. [21]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730...

    2022

  14. [22]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  15. [23]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  16. [24]

    Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207, 2021

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207, 2021

    arXiv

  17. [25]

    Converging paradigms: The synergy of symbolic and connectionist ai in llm-empowered autonomous agents.arXiv preprint arXiv:2407.08516, 2024

    Haoyi Xiong, Zhiyuan Wang, Xuhong Li, Jiang Bian, Zeke Xie, Shahid Mumtaz, Anwer Al-Dulaimi, and Laura E Barnes. Converging paradigms: The synergy of symbolic and connectionist ai in llm-empowered autonomous agents.arXiv preprint arXiv:2407.08516, 2024. 8

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.