{"id":"6d39d16d-b35b-428e-8b10-9daaa406f038","arxiv_id":"2507.15815","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The LLM Economist framework couples persona-conditioned worker agents with an in-context RL planner to search US-bracket tax schedules, yet its Saez benchmark is derived from the planner's own solution and its headline welfare gain is contradicted by Figure 5.","lead":"This paper simulates 100 AI workers with US Census-style incomes who react to tax rates set by a second AI, all through text prompts. It claims the AI tax planner approaches or beats standard optimal-tax formulas, but those formulas were tuned using the AI's own proposals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claim contradicted by own experiments: LLM planner underperforms Saez in both scenarios, and Saez baselines are derived from the LLM solution.","rationale":"The strongest_claim is the abstract's assertion that the planner converges near Stackelberg equilibria that improve aggregate social welfare relative to Saez solutions. For this to hold, the LLM planner's terminal policy must yield higher SWF than the Saez schedule in at least one main scenario. The paper's own Section 4.3 shows the opposite: in the seven-bracket bounded case the welfare-maximizing grid-searched Saez schedule is 10% above the LLM policy, and in the three-bracket isoelastic case the Saez-regression schedule outperforms the LLM policy. The interpretation paragraph concedes the LLM lands within 10–35% of the Saez optimum and that Saez retains an edge, so the headline is internally contradicted. The circularity compounds this: Appendix D states that a true analytical solution is not possible and that Saez rates require a solution from the LLM Economist to be locally perturbed, and Section 4.3 says the isoelastic Saez is solved analytically 'given a good starting point, the LLM Economist solution.' The comparison is therefore between the LLM and a local optimizer initialized at the LLM's answer; the local optimizer's edge is unsurprising and does not validate the LLM. A concrete, decisive check is to run the released code and compare terminal SWF values from Figure 5; if the LLM SWF is lower than the Saez SWF in either scenario, the central claim is false. This is an internal-consistency issue independent of whether the LLM agents truly maximize the mathematical utilities, and it is thus the most load-bearing concern. The reader's rationale explicitly mentions the contradiction and the circular Saez baseline, so the verdict of REJECT is supported; however, the reader's formal weakest_assumption (utility maximization) is a different concern, hence partial agreement.","tokens_in":19261,"tokens_out":8441,"duration_ms":77835,"concrete_test":"Run the released code (github.com/sethkarten/LLM-Economist) for the seven-bracket bounded-utility scenario and print the terminal SWF for the LLM Economist schedule and for the grid-searched Saez schedule. If SWF(Saez) > SWF(LLM), as the paper reports, the abstract's claim that the planner improves aggregate social welfare relative to Saez solutions is false. Repeat for the three-bracket isoelastic scenario with the Saez-regression schedule; any instance where the LLM SWF is lower than the Saez SWF refutes the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that the planner converges to equilibria that improve aggregate social welfare relative to Saez solutions is directly contradicted by the paper's own experiments. In the seven-bracket bounded scenario, the grid-searched Saez schedule improves SWF by 10% over the LLM Economist policy (Fig. 5a); in the three-bracket isoelastic scenario, the Saez-regression schedule outperforms the LLM Economist (Sec. 4.3, Fig. 5b). The interpretation section concedes that the in-context planner 'lands within 10–35% of the Saez optimum' and that Saez 'retains an edge.' Moreover, the Saez baselines are not independent: in the bounded case they are obtained by perturbing the LLM's terminal schedule and grid-searching, and in the isoelastic case the Saez regression is solved 'given a good starting point, the LLM Economist solution' (Sec. 4.3; App. D). Thus the headline comparison is both internally inconsistent and circular: the LLM does not improve on Saez; it falls short of a benchmark seeded from its own output.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the LLM Economist, a two-tier agent-based simulation in which LLM-based worker agents choose labor hours from persona-conditioned prompts and an LLM planner proposes piecewise-linear marginal tax schedules via in-context reinforcement learning. The authors claim that the planner converges to Stackelberg equilibria that improve aggregate social welfare relative to Saez-optimal schedules, and that periodic persona-level voting further improves outcomes. Experiments use Llama-3.1-8B with populations up to 100 agents and compare against statutory U.S. tax schedules and Saez-derived baselines.","tokens_in":19549,"tokens_out":4759,"duration_ms":48060,"significance":"If the claims were supported, the framework would be a valuable testbed for mechanism design with heterogeneous, bounded-rational agents, and the Census-calibrated population construction is a genuinely useful ingredient. The paper's strengths include the public code release, the GB2 fit to ACS data, and explicit ablations of prompt design and tax-year length. However, the central welfare comparison is invalid and contradicted by the paper's own figures, and the Stackelberg-equilibrium interpretation is not evidenced. The main evaluation therefore does not establish the paper's headline contribution.","major_comments":[{"comment":"The abstract's claim that the planner 'improves aggregate social welfare relative to Saez solutions' is contradicted by the manuscript's own results. In the seven-bracket bounded scenario, the grid-searched Saez schedule improves SWF by 10% over the LLM Economist policy (Fig. 5a); in the three-bracket isoelastic scenario, the Saez-regression schedule outperforms the LLM Economist (Fig. 5b). The Interpretation paragraph concedes that the planner 'lands within 10–35% of the Saez optimum' and that 'Saez retains an edge.' The headline claim should be corrected or replaced by experiments that actually show an improvement.","section":"Abstract; Sec. 4.3, Fig. 5"},{"comment":"The Saez baselines are not independent benchmarks. In the bounded case, the Saez schedule is obtained by perturbing the LLM Economist's terminal schedule and grid-searching; in the isoelastic case, the Saez regression is solved 'given a good starting point, the LLM Economist solution' (Sec. 4.3). Appendix D states that 'Saez tax rates require a solution from the LLM Economist to be locally perturbed before finding the optimal policy.' Because the elasticity is estimated from the same simulation, the comparison cannot establish that the LLM Economist improves on Saez; it only shows that a local search around the LLM's own solution can improve welfare.","section":"Sec. 4.3; App. D"},{"comment":"The interpretation of the results as convergence to a Stackelberg equilibrium is not supported. Section 2 asserts that stationary optimal responses are 'without loss of generality' and defines equilibrium via exact maximization of the utilities in Eqs. (3)-(4) and Eq. (1). Section 3, however, describes workers and the planner as producing JSON text actions from prompts, with no evidence that these actions solve the stated maximization problems. Without a validation that worker labor choices and planner schedules actually maximize the relevant objectives, the paper cannot claim that the simulated trajectories approximate Stackelberg equilibria.","section":"Sec. 2, Sec. 3"},{"comment":"The voting experiments claim to reproduce 'tyranny of the masses' and welfare-enhancing turnover, but these claims are supported only by qualitative descriptions of two runs. Figure 6 shows no quantitative welfare measures, no statistical summary across seeds, and no comparison to a non-voting control. The statement that 'electoral exploration can outperform static optimal taxation' is therefore not established by the evidence presented.","section":"Sec. 4.4, Fig. 6"}],"minor_comments":[{"comment":"The text says 'the dissatisfaction penalty ψ appearing in Eq. (3)' but in Eq. (3) ψ is the labor-disutility scale; the dissatisfaction penalty is φ in Eq. (4).","section":"Sec. 4.2"},{"comment":"The caption mentions 'grey not shown in the three-bracket case,' but no grey curve is identified for the seven-bracket case either; please clarify which baseline is orange and what the grey reference denotes.","section":"Fig. 5 caption"},{"comment":"The first two rows of Table 2a display '8 310' and '16 600,' which appear to be misaligned or missing separators; the total-step entries should be formatted consistently with the later rows.","section":"Table 2a"},{"comment":"The pseudocode refers to a variable 'two_timescale' that is not defined in the text; the planner-update interval should be specified explicitly or aligned with the tax-year length K.","section":"Appendix B, Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The abstract and the contribution list in the introduction disagree with each other: contribution (ii) says 'similar social welfare' while the abstract claims 'improve aggregate social welfare relative to Saez solutions.' The body of the paper sides with the weaker claim. Even the weaker claim is not credible because the Saez baseline is constructed from the LLM's own terminal schedule and elasticity. A revision that uses an independently computed Saez benchmark and tones down the equilibrium claims might produce a publishable descriptive study, but as it stands the central results do not support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: the paper builds something real—a two-level LLM tax simulation with Census-calibrated personas, in-context RL for both workers and planner, and a voting layer—but the selling point is broken. The abstract claims the planner improves social welfare relative to Saez solutions; the experiments show the opposite in both settings, and the Saez benchmark is constructed from the LLM's own terminal schedule. So the headline is both factually wrong and methodologically circular.\n\nWhat's genuinely new and well done: the combination of persona-conditioned workers, a planner using in-context RL over tax brackets, and periodic democratic voting is not something I've seen in one package. The setup is concrete, the code is released, and the ablations on tax-year length and prompt design are careful. The GB2 skill prior fitted to ACS data is a nice touch, and the paper is honest enough to include an interpretation section that concedes the LLM planner lands within 10–35% of Saez and that Saez retains an edge. That honesty makes the abstract overclaim more puzzling.\n\nThe soft spots are load-bearing. Figure 5a shows the grid-searched Saez schedule improves SWF by 10% over the LLM Economist; Figure 5b shows the Saez-regression schedule outperforming it in the isoelastic case. Yet the abstract says the planner improves welfare relative to Saez. The circularity is explicit in Appendix D: Saez tax rates require a solution from the LLM Economist to be locally perturbed before finding the optimal policy. So the baseline cannot serve as an independent check. There's also a softer, but real, concern: the claim that workers maximize their mathematical utilities via in-context RL is asserted rather than demonstrated. Nothing in the prompts or results proves the language model finds maximizing labor hours; it may simply follow prompt patterns. That doesn't kill the simulation-as-testbed idea, but it does mean the 'Stackelberg equilibrium' language is stronger than the evidence.\n\nOverall, I'd take this paper seriously as a first step. The engineering is solid and the question—can LLM simulacra serve as a policy testbed—is important. But the central comparison needs to be rebuilt with an independent Saez baseline and the abstract corrected. If the authors fix those, it could be a real contribution. I'd send it to peer review expecting major revision, not desk-reject it, because the framework is novel and the failure mode is instructive rather than trivial.","headline":"A genuinely novel LLM simulacra framework whose headline claim is contradicted by its own experiments: the planner underperforms a Saez baseline that is itself seeded from the LLM's solution.","tokens_in":20047,"tokens_out":2218,"would_cite":false,"duration_ms":26751,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-based planner, optimized only through text prompts, converges near optimal tax schedules in a simulated 100-agent economy, beating the U.S. statutory baseline.","keywords":["in-context reinforcement learning","mechanism design","optimal taxation","Stackelberg equilibrium","agent-based simulation","large language model agents","social welfare","generative simulacra"],"falsifier":"Take the same 100-agent setup and keep every prompt component except the utility numbers, utility formulas, and instructions to maximize utility; if average labor hours and social welfare barely move, the utility-maximization interpretation is unsupported. Separately, replace the planner's in-context search with random bracket shifts of the same magnitudes; if welfare is comparable, the language model's role in the reported gains is not established.","tokens_in":19089,"feed_emoji":"🏛️","tokens_out":8694,"duration_ms":87845,"temperature":0.7,"pith_summary":"This paper argues that a society of language-model agents can be used to design tax policy before any real policy is enacted. It builds a two-tier economy in which a planner agent proposes marginal tax brackets and one hundred worker agents, each with a distinct persona and skill level, respond by choosing how many hours to work. Both sides improve through in-context reinforcement learning: the agents read their own histories and rewards in the prompt and adjust their next text output, with no gradient updates. The paper reports that the planner converges close to a Stackelberg equilibrium whose social welfare exceeds the statutory U.S. tax schedule and reaches the neighborhood of the Saez benchmark, the standard static optimal-tax solution. It also shows that letting workers periodically vote to replace the planner can either entrench a majority or raise welfare, depending on the population.","feed_headline":"LLM tax planner nears optimal tax rates in a 100-agent economy","feed_subtitle":"Prompt-driven agents bargain over tax brackets and approach welfare levels that static formulas miss.","key_machinery":"The object that carries the argument is two-level in-context reinforcement learning inside a Stackelberg game. In-context reinforcement learning means the model updates its behavior from text prompts containing scalar rewards—here, social welfare for the planner and utility for each worker—rather than from weight updates. The planner acts on a piecewise-linear marginal tax schedule over seven federal brackets, the workers act on weekly labor hours, and a lump-sum rebate keeps the budget balanced. Two design choices make the equilibrium reachable: a long enough 'tax year' for workers to finish adapting before the planner moves, and prompt cues that first encourage exploration of new schedules and then lock in the best one. A Census-calibrated skill distribution and persona text supply the heterogeneity that makes the result a population model rather than a representative-agent model.","core_discovery":"The central claim is that purely text-prompted optimization can reproduce the classic equity-efficiency trade-off of optimal tax theory. The planner's prompt contains income and utility histograms, recent social-welfare values, and the best tax schedule seen so far; the worker prompts contain pre-tax income, post-tax income, marginal rate, rebate, and a history window. From these, the planner learns to shift seven piecewise-linear marginal rates and each worker learns a labor choice, and the pair is interpreted as a stationary Stackelberg equilibrium. The paper reports that this equilibrium reaches 90% of the best observed social welfare, lands within 10-35% of the Saez benchmark's welfare in the two evaluated scenarios, and improves welfare by 93% over the U.S. federal schedule in the seven-bracket bounded case. In the democratic variant, majority voting over planner candidates produces either stable majority exploitation in a three-agent society or nearly annual leadership turnover that sometimes outperforms static optimal taxation in the 100-agent society.","pith_inferences":["An extension the paper does not run would be a control planner that searches the same bracket space with random or grid disturbances; absent that baseline, some of the reported welfare gain could come from the search-and-replay scaffolding rather than from the language model's economic judgment.","The persona population is sampled from Census marginals rather than joint demographic distributions, so correlations such as age, occupation, and income are only approximated; re-running with a joint-calibrated sampler is a direct test of how much the equilibrium schedule depends on that approximation.","The democratic-voting result leaves open whether voters are persuaded by the substance of platforms or by prompt style; measuring vote outcomes against platform content versus platform tone would separate persuasion from policy preference.","A stronger claim, that the learned schedules transfer to a freshly sampled population, is not tested; showing transfer would turn the framework from a fitted simulacrum into a reusable mechanism-design engine."],"forward_implications":["If the central claim is right, tax-policy experiments can be run on a text-prompted population before being tried on humans or deployed in automated economies.","The planner's search implicitly re-estimates the behavioral elasticities that the static Saez formula takes as fixed, which removes a known practical limitation of using Saez rates directly.","The voting results imply that the framework can evaluate not only tax schedules but also institutional rules such as election frequency and candidate platforms.","Because the same 8-billion-parameter model handles 100-agent runs and scales to 1,000 agents, the testbed is cheap enough to use for repeated policy audits."],"supporting_citations":[{"why":"Supplies the Saez formula used as the static optimal-tax baseline the planner's welfare is compared against.","marker":"[60]"},{"why":"Defines the Mirrleesian utility-based optimal-tax objective that motivates the social-welfare function.","marker":"[52]"},{"why":"Provides the 2023 Census microdata used to fit the skill distribution that generates worker personas.","marker":"[66]"},{"why":"Supplies the in-context reinforcement-learning method that lets both planner and workers adapt from text prompts with scalar rewards.","marker":"[35]"},{"why":"Provides the isoelastic utility form and closed-form tax benchmarks used for comparability in the three-bracket scenario.","marker":"[48]"},{"why":"Frames the large-population-model approach used to create demographically realistic agent populations.","marker":"[8]"}],"fun_headline_variants":["LLM tax planner negotiates to near-optimal rates in 100-agent economy","Prompt-driven agents find welfare gains over static tax formulas","Democratic voting improves LLM-designed tax policies in simulations","Text-based AI economists reach 93% better welfare than US tax code","LLM simulacra bargain over taxes, approaching Stackelberg equilibria"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a worker-agent that is told to maximize its utility actually produces the labor choice that maximizes the mathematical utility in its prompt; if the model is only following the prompt's wording or examples, the converged tax schedule is not a genuine economic equilibrium.","fun_headline_variants_meta":{"raw":{"variants":["LLM tax planner negotiates to near-optimal rates in 100-agent economy","Prompt-driven agents find welfare gains over static tax formulas","Democratic voting improves LLM-designed tax policies in simulations","Text-based AI economists reach 93% better welfare than US tax code","LLM simulacra bargain over taxes, approaching Stackelberg equilibria"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1666,"prompt_tokens":976,"completion_tokens":690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":599}},"tokens_in":592,"tokens_out":690,"duration_ms":7996,"temperature":1.0,"reasoning_tokens":599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:22:54.637835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 100-agent setup and keep every prompt component except the utility numbers, utility formulas, and instructions to maximize utility; if average labor hours and social welfare barely move, the utility-maximization interpretation is unsupported. Separately, replace the planner's in-context search with random bracket shifts of the same magnitudes; if welfare is comparable, the language model's role in the reported gains is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Saez formula used as the static optimal-tax baseline the planner's welfare is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Mirrleesian utility-based optimal-tax objective that motivates the social-welfare function."},{"cited_title":"Census Bureau","cited_arxiv_id":null,"evidence_quote":"Provides the 2023 Census microdata used to fit the skill distribution that generates worker personas."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the isoelastic utility form and closed-form tax benchmarks used for comparability in the three-bracket scenario."},{"cited_title":"Large Population Models","cited_arxiv_id":"2507.09901","evidence_quote":"Frames the large-population-model approach used to create demographically realistic agent populations."}],"review_version":1}