Pith. sign in

REVIEW 48 references

TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems

T0 review · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark parses LLM reasoning traces on Go life-and-death problems into search trees and shows that search organization, not token volume, distinguishes stronger reasoners.

arxiv 2608.13221 v1 pith:552DAJ6C submitted 2026-08-13 cs.AI

Shunwen Bai , Ziping Ma , Chaoyang Zhang , Yarong Wang , Jiale Liu , Zhen Qin , Qingpei Guo This is my paper · ORCID
classification cs.AI
keywords searchreasoningefficiencymodelsorganizationproblemsprocess-leveltsugo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TsuGO is a benchmark that tests how large language models search through possible moves when solving Go life-and-death puzzles, known as tsumego. Each puzzle has one or a few correct first moves, and the full solution tree is verified by human experts and a Go engine. Models are given the board as coordinates, a grid, an image, or a mix of these, and either four candidate moves to choose from (K=4) or no candidates at all (K=None).

The authors convert each model's written reasoning trace into a search tree: which moves were considered, in what order, how much effort was spent on each branch, and which branches were judged winning or losing. From these trees they compute Search Efficiency (SearchE), which combines three signals: effort wasted on wrong branches, whether the correct move was examined first, and how early the correct move appeared. They compare this with Token Efficiency, which is simply accuracy divided by the number of tokens used.

On 600 puzzles across easy, medium, and hard tiers, most LLMs solved only a fraction of problems. Stronger models such as Gemini-3.1-Pro did much better on easy puzzles but still collapsed on hard open-ended ones. The authors find that models with better SearchE also have higher accuracy, while models that produce long chains of thought do not necessarily search better. They contrast this with KataGo, a neural-guided Go program that solves most easy puzzles with the same budget, and with plain MCTS, which behaves like shallow undirected search. The paper argues that evaluating how models allocate reasoning effort across branches is a missing dimension in LLM benchmarking.

Extended reading notes

Core claim

Current LLMs remain far from stable tsumego solving, and Search Efficiency, a process-level score computed from wrong-branch waste, first-hit behavior, and correct-candidate rank, aligns with answer accuracy more consistently than token efficiency. As stated in §5.3: 'SearchEisastrongerprocesssignalthanTokenE.' If true, this establishes search organization as a measurable, distinct dimension of LLM reasoning beyond answer accuracy and token cost.

Load-bearing premise

The pipeline assumes that visible CoT traces faithfully externalize the model's internal search organization. The paper acknowledges in Appendix L: 'Search trees are extracted from observable CoT traces, so our process metrics describe the visible reasoning path rather than the model's latent computation.' If proprietary summaries or omitted branches misrepresent internal search, then SearchE comparisons measure text organization, not the claimed search ability. This premise enters at §4.1 and is load-bearing for every search metric in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central new quantity, SearchE, is a designed composite rather than a derived quantity. Its weights and normalization are chosen by hand, and it depends on ground-truth answers, so the benchmark's 'discovery' of SearchE-accuracy alignment carries a built-in correlation. No new physical or ontological entities are introduced.

free parameters (2)
  • SearchE weights = 0.5/0.3/0.2
    Weights on (1-SWR), SFH, and (1-SCR) in Eq. (2); manually chosen, though Table 15 shows nearby variants preserve correlations.
  • TokenE scaling constant = 1000
    Eq. (3) divides thinking tokens by 1000; an arbitrary normalization that changes TokenE magnitudes but likely not rankings.
assumptions (3)
  • domain assumption Visible CoT traces faithfully externalize the model's internal search organization
    The entire process-tree analysis treats written reasoning as a record of search. Stated in §4.1 and acknowledged as a limitation in Appendix L.
  • domain assumption Gemini-2.5-Pro extraction preserves metric-relevant tree structure with 93-98% agreement
    Human validation on 300 samples supports this, but the extractor itself is an LLM whose errors could bias SearchE; Appendix C.
  • domain assumption First-move hit rate is a valid measure of tsumego solving
    Accuracy is defined only on the first move; full-game correctness is not assessed. §4.4 and Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems." pith.science (2026). https://pith.science/paper/552DAJ6C

@misc{pith2026260813221,
  author       = {Pith},
  title        = {Pith review of: TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/552DAJ6C}},
  note         = {Machine review of arXiv:2608.13221}
}
read the original abstract

The evaluation of LLM reasoning is moving from final-answer accuracy to process-level assessment, yet existing methods still fail to capture how models plan reasoning paths and allocate reasoning resources--that is, how they organize search. Prior process-level methods focus on the coherence and redundancy of chain-of-thought (CoT), and most benchmark tasks have a single objective solvable by static capabilities such as derivation and tool use, leaving search organization unmeasured. We introduce TsuGO, a process-level reasoning benchmark for evaluating Search Efficiency in LLM reasoning through Go life-and-death problems. These problems provide closed and verifiable solution spaces with an inherent adversarial structure, making candidate generation, response checking, branch comparison, and backtracking necessary parts of reasoning rather than incidental trace patterns. By constraining the solution space, TsuGO disentangles domain knowledge from search organization, parses CoT into a structured search tree, and reports Search Efficiency together with Token Efficiency and other diagnostic metrics and visualizations. Experiments show that current LLMs remain far from stable tsumego solving: stronger models succeed by finding the correct candidate earlier and sustaining effort on productive branches, but most models still behave much closer to unguided search algorithms than to neural-guided KataGo. Longer CoT or higher Token Efficiency does not necessarily imply better search. Our results identify search organization and reasoning-resource allocation as missing dimensions in LLM reasoning evaluation.

Figures

Figures reproduced from arXiv: 2608.13221 by the authors.

Figure 1
Figure 1. TsuGO reveals search organization that is hidden from Answer- and Token-Efficiency-based evaluation. Judger measures necessary reasoning and structural redun￾dancy from dependency graphs (Li et al. 2026); ReEfBench maps traces into logical structures to analyze reasoning effi￾ciency and behavioral patterns (Fu et al. 2026); and process￾supervision or PRM-based methods evaluate intermediate steps through step-level f… view at source ↗
Figure 2
Figure 2. Overview of TsuGO: verified problems are rendered into five modalities, solved through bounded or open K-Search, parsed into process search trees, and evaluated by Search Efficiency with supporting diagnostics. verification, and revision. Branching is therefore not inher￾ently redundant; effective reasoning depends on whether ex￾ploration is organized around valuable paths. Classical game AI and recent LLM search st… view at source ↗
Figure 3
Figure 3. Overview of the TsuGO dataset construction and evaluation settings [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Aggregated process search trees. Each panel shows [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: SearchE aligns with accuracy more consistently [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Example normalized tsumego boards sampled from Easy, Medium, and Hard splits. Labels use benchmark display [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Accuracy–SearchE scatter plots under alternative SearchE weight schemes. The relationship remains monotonic across [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Representative Smargo MCTS/UCT tree for a [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: K = 4 extracted process trees for eight displayed Easy examples. Green panels denote correct final answers and red panels denote wrong final answers. to targeted training frameworks that supervise branch prior￾itization, opponent-response verification, state maintenan…
Figure 11
Figure 11. Figure 11: Qualitative Logos probe on the first benchmark problem. The verified first move is [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Failure-mode distribution from the balanced qual [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [1]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  2. [2]

    Measuring mathematical problem solving with the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , journal=. Measuring mathematical problem solving with the

  3. [3]

    NeurIPS , year=

    Chain-of-thought prompting elicits reasoning in large language models , author=. NeurIPS , year=

  4. [4]

    OpenAI Blog , year=

    Learning to reason with. OpenAI Blog , year=

    work page

  5. [5]

    arXiv preprint arXiv:2501.12948 , year=

    DeepSeek-. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  6. [6]

    CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in

    Li, Siyi and Shi, Jiajun and Ni, Shiwen and Zhang, Ge and Li, Shuaimin and Wang, Shijian and Wen, Zhoufutu and Li, Yizhi and Alinejad-Rokny, Hamid and Liu, Jiaheng and Yang, Min and Huang, Wenhao , journal=. CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in

    work page

  7. [7]

    ReEfBench: Quantifying the Reasoning Efficiency of

    Fu, Zhizhang and Gu, Yuancheng and Hu, Chenkai and Liu, Hanmeng and Zhang, Yue , journal=. ReEfBench: Quantifying the Reasoning Efficiency of

    work page

  8. [8]

    Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and others , booktitle=

    work page

Show all 48 references
  1. [9]

    International Conference on Learning Representations , year=

    Language Models are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought , author=. International Conference on Learning Representations , year=

  2. [10]

    Lin, Bill Yuchen and Le Bras, Ronan and Richardson, Kyle and Sabharwal, Ashish and Poovendran, Radha and others , journal=

  3. [11]

    Liu, Yiwei and Li, Yucheng and Li, Xiao and Cheng, Gong , journal=

  4. [12]

    Golovneva, Olga and Chen, Moya Peng and Poff, Spencer and Corredor, Martin and Zettlemoyer, Luke and Fazel-Zarandi, Maryam and Celikyilmaz, Asli , booktitle=

  5. [13]

    2023 , doi=

    Prasad, Archiki and Saha, Swarnadeep and Zhou, Xiang and Bansal, Mohit , booktitle=. 2023 , doi=

    2023

  6. [14]

    Inference-Time Computations for

    Parashar, Shubham and Olson, Blake and Khurana, Sambhav and Li, Eric and Ling, Hongyi and Caverlee, James and Ji, Shuiwang , journal=. Inference-Time Computations for

  7. [15]

    , journal=

    Gandhi, Kanishk and Chakravarthy, Ayush and Singh, Anikait and Lile, Nathan and Goodman, Noah D. , journal=. Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective

  8. [16]

    Mastering the game of

    Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others , journal=. Mastering the game of

  9. [17]

    Mastering the game of

    Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian and others , journal=. Mastering the game of

  10. [18]

    Bandit based

    Kocsis, Levente and Szepesv. Bandit based. Machine Learning: ECML 2006 , pages=. 2006 , publisher=

    2006

  11. [19]

    , journal=

    Wu, David J. , journal=. Accelerating self-play learning in

  12. [20]

    arXiv preprint , year=

    Do Not Think That Much for 2+3! Overthinking with Chain-of-Thought , author=. arXiv preprint , year=

  13. [21]

    arXiv preprint , year=

    Qwen3 Technical Report , author=. arXiv preprint , year=

  14. [22]

    arXiv preprint arXiv:2511.21631 , year=

    Qwen3-. arXiv preprint arXiv:2511.21631 , year=

    arXiv

  15. [23]

    arXiv preprint , year=

    Gemini: A family of highly capable multimodal models , author=. arXiv preprint , year=

  16. [24]

    arXiv preprint arXiv:2412.19437 , year=

    DeepSeek-. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  17. [25]

    arXiv preprint arXiv:2602.02276 , year=

    Kimi. arXiv preprint arXiv:2602.02276 , year=

    arXiv

  18. [26]

    arXiv preprint arXiv:2605.26494 , year=

    The. arXiv preprint arXiv:2605.26494 , year=

    arXiv

  19. [27]

    2026 , howpublished=

    2026

  20. [28]

    2026 , howpublished=

    Gemini 3.1 Pro: A Smarter Model for Your Most Complex Tasks , author=. 2026 , howpublished=

    2026

  21. [29]

    arXiv preprint arXiv:2302.13071 , year=

    Chess as a Testbed for Language Model State Tracking , author=. arXiv preprint arXiv:2302.13071 , year=

    arXiv

  22. [30]

    ICLR , year=

    Emergent world representations: Exploring a sequence model trained on a synthetic task , author=. ICLR , year=

  23. [31]

    Think you have solved question answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal=. Think you have solved question answering? Try

  24. [32]

    TMLR , year=

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. TMLR , year=

  25. [33]

    ICLR , year=

    Let's verify step by step , author=. ICLR , year=

  26. [34]

    2025 , doi=

    Song, Mingyang and Su, Zhaochen and Qu, Xiaoye and Zhou, Jiawei and Cheng, Yu , booktitle=. 2025 , doi=

    2025

  27. [35]

    ICML , year=

    AlphaZero-like tree-search can guide large language model decoding and training , author=. ICML , year=

  28. [36]

    Stream of search (

    Gandhi, Kanishk and Lee, Denise and Grand, Gabriel and Liu, Muxin and Cheng, Winson and Suhr, Alane and Goodman, Noah D , journal=. Stream of search (

  29. [37]

    Advances in Neural Information Processing Systems , volume=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  30. [38]

    International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations , year=

  31. [39]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Graph of Thoughts: Solving Elaborate Problems with Large Language Models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2024 , doi=

    2024

  32. [40]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Reasoning with Language Model is Planning with World Model , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  33. [41]

    arXiv preprint arXiv:2310.04406 , year=

    Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models , author=. arXiv preprint arXiv:2310.04406 , year=

    arXiv

  34. [42]

    Zheng, Chujie and Zhang, Zhenru and Zhang, Beichen and Lin, Runji and Lu, Keming and Yu, Bowen and Liu, Dayiheng and Zhou, Jingren and Lin, Junyang , booktitle=

  35. [43]

    Luo, Haotian and Shen, Li and He, Haiying and Wang, Yibo and Liu, Shiwei and Li, Wei and Tan, Naiqiang and Cao, Xiaochun and Tao, Dacheng , journal=

  36. [44]

    Li, Zhiyuan and Chang, Yi and Wu, Yuan , journal=

  37. [45]

    Shen, Chenhui and others , journal=

  38. [46]

    arXiv preprint arXiv:2503.16419 , year=

    Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models , author=. arXiv preprint arXiv:2503.16419 , year=

    arXiv

  39. [47]

    arXiv preprint arXiv:2412.15797 , year=

    Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning , author=. arXiv preprint arXiv:2412.15797 , year=

    arXiv

  40. [48]

    Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of

    Ma, Yichuan and Li, Linyang and Chen, Yongkang and Li, Peiji and Ye, Jiasheng and Guo, Qipeng and Lin, Dahua and Chen, Kai , booktitle=. Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of. 2025 , doi=

    2025

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.