{"id":"0922031a-2d6b-4057-9b44-1e6b7b8abe83","arxiv_id":"2605.30094","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"PokerSkill enables LLMs to reach near-GTO poker performance by grounding decisions in an expert-designed skill library retrieved via a deterministic context engine, without training or solvers.","lead":"PokerSkill combines LLMs with a human-designed library of poker skills and a context engine that retrieves relevant rules for the current game state, enabling competitive play without any training or solver access. A smart generalist might read it to understand whether structured human knowledge can let general-purpose models handle complex strategic decisions that previously required massive specialized computation.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The human-expert skill library's comprehensiveness and the deterministic context engine's retrieval accuracy for every game state remain unverified.","rationale":"The reader's weakest_assumption matches the load-bearing point exactly. Full text availability does not remove the need for explicit validation of library coverage and engine fidelity; the abstract-level results alone cannot confirm that the reported mbb/hand figures arise from the intended interface rather than partial or biased retrieval. No other internal inconsistency (e.g., in the reported baselines or variance numbers) appears load-bearing.","tokens_in":1809,"tokens_out":344,"duration_ms":33652,"concrete_test":"Sample 200 game states uniformly across street, pot size, and board texture; for each, have two independent poker experts list the minimal skill fragments that should apply, then run the context engine and measure recall; if average recall <95% or if any state misses a fragment that changes the legal-action set, re-run the GPT-5.5 XHigh evaluation on those states with the missing fragments added.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires that rule-based skills plus LLM yield competitive play precisely because the library supplies all relevant fragments and the engine retrieves them without omission or bias. If either fails for some states (e.g., unusual board textures, multi-way pots, or specific bet-sizing histories), the LLM receives incomplete grounding and the performance numbers cannot be attributed to the claimed mechanism. The abstract states the library is \"entirely designed by human poker experts\" and the engine is \"deterministic,\" but provides no coverage metric, retrieval-error audit, or ablation that isolates engine failures.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces PokerSkill, a training-free and solver-free framework that combines a human-expert-designed layered rule-based skill library with LLMs via a deterministic context engine for action grounding in poker; it reports that GPT-5.5 XHigh with PokerSkill achieves -57 ±21 mbb/hand against GTOWizard (with similar results for Claude variants), reducing losses by 49-61% vs. default LLM baselines and outperforming Slumbot, claiming this is the first LLM competitive performance in a complex imperfect-information game without training or solvers.","tokens_in":1947,"tokens_out":497,"duration_ms":17049,"significance":"If the central results hold after verification, the work would be significant for demonstrating a hybrid approach that leverages interpretable rule-based skills to constrain LLMs without game-specific training or equilibrium solvers, potentially generalizable to other domains; the public code release at https://github.com/lbn187/PokerSkill is a clear strength supporting reproducibility.","major_comments":[{"comment":"Abstract: the performance numbers (e.g., -57 ±21 mbb/hand, -80 ±29 mbb/hand) are reported with error bars and comparisons to GTOWizard and Slumbot, but without any mention of total hands played, variance-reduction methods, or controls for selection effects in the skill library, which directly limits verification of the 49-61% loss-reduction claim and the assertion that the combination outperforms both components alone.","section":"Abstract"},{"comment":"Abstract / context-engine description: the strongest claim—that rule-based skills plus LLM yield competitive play precisely because the library supplies all relevant fragments and the engine retrieves them without omission or bias—is load-bearing, yet the manuscript provides no coverage metric, retrieval-error audit, or ablation study isolating cases where the deterministic engine might fail (e.g., unusual board textures or multi-way pots), leaving open whether performance gains can be attributed to the claimed mechanism.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: model identifiers such as 'GPT-5.5 XHigh' and 'Claude Opus 4.6/4.7' should be defined or footnoted for clarity, as they do not correspond to standard public releases.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their insightful comments on our manuscript. We address each major comment below and indicate the revisions we will make to improve the clarity and verifiability of our results.","responses":[{"response":"We agree that the abstract would benefit from additional context on the experimental methodology. The full manuscript details the evaluation over a large number of hands with standard error for the reported metrics, and the skill library was designed by experts to cover typical poker scenarios without selection bias. To address this, we will revise the abstract to include a note on the scale of the experiments and the variance estimation approach. We will also ensure the main text explicitly discusses the controls used in the skill library design.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the performance numbers (e.g., -57 ±21 mbb/hand, -80 ±29 mbb/hand) are reported with error bars and comparisons to GTOWizard and Slumbot, but without any mention of total hands played, variance-reduction methods, or controls for selection effects in the skill library, which directly limits verification of the 49-61% loss-reduction claim and the assertion that the combination outperforms both components alone."},{"response":"The paper emphasizes that the deterministic context engine retrieves relevant skills based on explicit state analysis, and the results show the hybrid approach outperforms both rule-based and LLM-only baselines. We acknowledge that explicit coverage metrics would further support the mechanism. In the revised manuscript, we will include a dedicated analysis of the engine's retrieval coverage, including audits for various board textures and pot scenarios, to demonstrate the absence of systematic omission or bias.","revision_made":"yes","referee_comment":"[Abstract] Abstract / context-engine description: the strongest claim—that rule-based skills plus LLM yield competitive play precisely because the library supplies all relevant fragments and the engine retrieves them without omission or bias—is load-bearing, yet the manuscript provides no coverage metric, retrieval-error audit, or ablation study isolating cases where the deterministic engine might fail (e.g., unusual board textures or multi-way pots), leaving open whether performance gains can be attributed to the claimed mechanism."}],"tokens_in":1516,"tokens_out":466,"duration_ms":36066,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point to take away is that this is the first reported case of an LLM reaching numbers close to strong bots in no-limit poker by using a fixed library of expert rules as a grounding layer instead of any CFR training or online solver calls. The authors show GPT-5.5 XHigh losing 57 mbb/hand to GTOWizard, with similar results for the Claude versions, and they beat the plain-prompt baselines by 49-61 percent while also topping Slumbot. That combination result is the actual novelty: rules alone are too rigid and LLMs alone are too loose, but the structured interface narrows the LLM's choices enough to produce usable play.\n\nThe setup is straightforward and the code is public, which is useful. They emphasize that the library was written by human experts and the context engine is deterministic, so the method stays training-free and solver-free.\n\nThe soft spot is the missing verification on the mechanism itself. The performance numbers rest on the assumption that the skill fragments cover every relevant state and that the engine retrieves the right ones without omission or bias for the hands that were actually played. The abstract gives no hand count, no breakdown by board texture or bet size, and no ablation that isolates retrieval failures. Without those, the error bars could reflect selection effects or incomplete grounding rather than the claimed synergy. The comparison to GTOWizard is external, which is good, but the lack of protocol details caps how far the result can be trusted right now.\n\nThis is worth sending to a referee who can look at the full experimental log and the library coverage. People working on grounded agents or low-compute game AI would get value from the idea even if the numbers need tightening. I would not cite it yet, but it deserves peer review rather than a desk reject.","headline":"The paper's core claim is that a human-expert skill library plus deterministic retrieval lets LLMs reach competitive poker without training or solvers, but the abstract gives no hand counts, variance controls, or retrieval-coverage checks to back it up.","tokens_in":2412,"tokens_out":457,"would_cite":false,"duration_ms":16766,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Combining human poker rules with LLMs produces competitive play without training or solvers.","keywords":["poker","large language models","imperfect information games","training-free agents","solver-free methods","strategic decision making","game theory optimal play"],"falsifier":"A large-scale evaluation of the agent against a stronger equilibrium solver that checks whether the reported mbb/hand loss range holds or widens substantially on particular hand categories or game variants.","tokens_in":2712,"feed_emoji":"♠","tokens_out":726,"duration_ms":30165,"temperature":0.7,"pith_summary":"The paper establishes that large language models reach competitive poker performance in this imperfect-information game when guided by a structured set of human-expert rules rather than used alone or trained from scratch. A deterministic context engine pulls only the relevant skill fragments for the current state from a layered library, restricting the model to sensible actions. This hybrid reduces losses against the GTOWizard benchmark by 49 to 61 percent relative to direct prompting and beats the strong bot Slumbot. The authors report that neither the rules nor the LLM suffices by itself, yet the pairing yields an agent that needs neither training nor solver access. The result indicates a route to capable strategic agents in similar domains without the heavy computation required for equilibrium methods.","feed_headline":"LLMs reach competitive poker with human rules only, no training","feed_subtitle":"Context engine and skill library cut losses by half against GTO benchmarks while beating Slumbot","key_machinery":"The deterministic context engine that retrieves relevant fragments from the human-expert layered skill library to ground and constrain LLM action selection in poker states.","core_discovery":"PokerSkill is a training-free and solver-free framework that uses detailed rule-based poker skills as a structured action-grounding interface for LLMs. A deterministic context engine analyzes the current state and retrieves only the relevant fragments from a layered skill library designed by human poker experts, constraining the LLM's choice to reasonable actions. Against GTOWizard, GPT-5.5 XHigh with PokerSkill achieves -57 ± 21 mbb/hand, Claude Opus 4.6 achieves -80 ± 29 mbb/hand and Claude Opus 4.7 achieves -87 ± 64 mbb/hand, reducing losses by 49-61% compared to default-prompt baselines and outperforming Slumbot.","pith_inferences":["The method could be tested in other imperfect-information settings such as bridge or multi-agent negotiation.","An adaptive version of the skill library might further improve results if the deterministic retrieval misses edge cases.","The work implies that explicit expert structure can address LLM weaknesses in long-horizon strategic planning."],"forward_implications":["LLMs guided by expert rules can reach performance levels previously requiring millions of core-hours of counterfactual regret minimization.","Pure rule-based agents remain below equilibrium play while pure LLMs also fall short, but the combination closes much of the gap.","The resulting agent needs no game-specific training and no solver queries during play.","This is presented as the first demonstration of competitive LLM performance in a complex imperfect-information game without those resources."],"fun_headline_variants":["LLMs competitive in poker with human rules no training","Expert rules enable LLM poker without training or solvers","Human skill library grounds LLMs for competitive poker","No training needed for LLMs to compete in poker with rules"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The human-expert skill library comprehensively covers every relevant game situation and the context engine retrieves the correct fragments without omission or bias.","fun_headline_variants_meta":{"raw":{"variants":["LLMs competitive in poker with human rules no training","Expert rules enable LLM poker without training or solvers","Human skill library grounds LLMs for competitive poker","No training needed for LLMs to compete in poker with rules"]},"model":"grok-4.3","cost_usd":0.006109,"raw_usage":{"total_tokens":2964,"prompt_tokens":826,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":61087000,"prompt_tokens_details":{"text_tokens":826,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2077,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":826,"tokens_out":61,"duration_ms":18158,"temperature":1.0,"reasoning_tokens":2077,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:45:39.120174+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A large-scale evaluation of the agent against a stronger equilibrium solver that checks whether the reported mbb/hand loss range holds or widens substantially on particular hand categories or game variants.","supporting_citations":[],"review_version":1}