Pith. sign in

REVIEW 4 major objections 5 minor 7 references

Can reasoning models comprehend mathematical problems in Chinese ancient texts? An empirical study based on data from Suanjing Shishu

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that reasoning models, which score 70-95% on modern math benchmarks, drop to 63% closed-book and 70% open-book accuracy on a new 538-problem benchmark drawn from the Ten Mathematical Classics, because they lack Classical…

desk verdict A genuinely useful new benchmark for classical Chinese math problems, but every headline accuracy rides on an unvalidated LLM judge. read the letter →

arxiv 2505.16660 v3 pith:P65H6L6W submitted 2025-05-22 cs.CL cs.AI

Chang Liu , Dongbo Wang , Liu liu , Zhixiao Zhao This is my paper
classification cs.CLcs.AI
keywords Chinesemathematicalclassicsancientmathematicsreasoningmodelsbenchmarkconstructionclassicalevaluationunitconversionlargelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether AI reasoning models can actually solve mathematics written in Classical Chinese, and its answer is: only partially. On Guji_MATH, a new benchmark of 538 problems extracted from eight of the Ten Mathematical Classics, the best model reaches 63.01% accuracy when answering on its own and 70.25% when given the original solution method from the ancient text. The same models report at least 70% to over 95% on modern math benchmarks such as Math-500, so the gap is not generic arithmetic difficulty. The paper attributes most failures to missing historical and cultural context, misunderstanding of ancient mathematical concepts, and errors in unit conversion and data processing. The point of the benchmark is to show that mathematical reasoning in language models is not language-neutral, and that Classical Chinese and its embedded historical knowledge are a real bottleneck.

What carries the argument

The central object is Guji_MATH, a benchmark of 538 problems organized as Question-Answer-Solution triples with appended premise conditions, fifteen problem-type labels, and four difficulty levels. The evaluating mechanism is a dual-mode prompt design: closed-book mode gives the model only the problem source, question, and premise; open-book mode adds the original classical solution method and asks the model to reproduce it. Final correctness is decided by a QwQ-32B judge that is given a hand-built historical unit-conversion table for length, area, capacity, weight, currency, and calendar units, and instructed to treat equivalent values as correct. The unit-conversion judge is what turns raw model outputs into the accuracy tables, and the premise-append step is what makes the extracted problems self-contained for both models and human readers.

What would settle it

Re-grade a random sample of model outputs with two independent human annotators using the paper's own unit-conversion rules and compare their correctness labels with the QwQ-32B judge's labels; a disagreement of more than a few percentage points would shift the headline 63.01% and 70.25% figures and possibly the model ranking.

Watch

Extended reading notes

Core claim

The paper's claim is that current reasoning models can partially comprehend and solve ancient Chinese mathematical problems but fall far short of their modern-benchmark performance, and that the shortfall is mainly cultural and linguistic rather than computational. On the 538 Guji_MATH problems, DeepSeek R1 scores 63.01% in closed-book mode; when the original ancient solution steps are included, Qwen-Plus scores 70.25%, the best open-book result. Both figures sit well below reported modern scores of at least 70% and often above 95% on Math-500 and similar benchmarks. Error analysis of DeepSeek R1, the most stable model, finds that missing specific historical/cultural context accounts for 23.24% of errors, data processing and unit-conversion errors for 20.51%, and misunderstanding of mathematical concepts for 18.75%, with strong co-occurrence among these categories.

Load-bearing premise

The headline accuracies assume that the QwQ-32B judge, prompted with a hand-built unit-conversion table, correctly decides when a model's answer is numerically equivalent to the canonical ancient-text answer, and the paper reports no human validation of those judgments.

Editorial extensions

If this is right

  • Accuracy on Guji_MATH separates reasoning models that look similar on modern math benchmarks, so the benchmark can serve as a stress test for cross-cultural and cross-linguistic mathematical reasoning.
  • Supplying the original classical solution method in open-book mode improves every model's accuracy, with the biggest gains on Level 4 problems, indicating that ancient algorithmic text acts as useful scaffolding for reasoning models.
  • The dominant error types point to a concrete improvement target: models should be trained or prompted with Classical Chinese comprehension and historical metrology knowledge rather than only more arithmetic practice.
  • The Question-Answer-Solution processing framework and the dual-mode evaluation are transferable to other non-Latin-script classical texts, allowing similar benchmarks for other historical scientific traditions.
  • Because performance falls steadily from Level 1 to Level 4 and from basic arithmetic toward quadratic and cubic equations, Guji_MATH can be used to diagnose which mathematical problem types a model has genuinely learned.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's linguistic-bottleneck story would be to translate the same 538 problems into modern Chinese and rerun the closed-book evaluation; the error analysis predicts a sharp accuracy jump, but the paper does not run this ablation.
  • The scoring pipeline is itself a potential confound: if a second, independently checked judge agrees with only some of the QwQ-32B equivalence decisions, the headline accuracies and the model ranking could shift even when the solvers' raw outputs are unchanged.
  • The same benchmark could be used to evaluate a model specialized in Classical Chinese rather than a general-purpose reasoning model, which would separate language competence from mathematical reasoning ability.
  • Extending the dataset from Han-to-Tang mathematics to Song-Yuan-Ming topics such as Tianyuan Shu and Duoji Shu would reveal whether the cultural-context bottleneck grows or shrinks as the mathematics becomes more algorithmic, a question the paper explicitly leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Guji_MATH, a benchmark of 538 mathematical problems extracted from eight classical Chinese mathematical texts (Suanjing Shishu), organized as question-answer-solution triples with added premise conditions, difficulty levels, and problem-type labels. Six reasoning models (DeepSeek R1, Qwen-Plus-2025-04-28, QwQ-32B, Skywork-OR1-32B-Preview, DeepSeek-R1-Distill-Qwen-32B, and Light-R1-32B) are evaluated in closed-book and open-book modes, with accuracy determined by an LLM judge (QwQ-32B) that applies a manually constructed historical unit-conversion table. The central empirical claim is that even the strongest tested models reach only 63.01% accuracy in closed-book mode and 70.25% in open-book mode, far below their reported performance on modern mathematical benchmarks, with the dominant error causes attributed to missing historical/cultural context, misunderstanding of mathematical concepts, and data-processing/conversion errors.

Significance. If the headline accuracies withstand validation, Guji_MATH is a valuable and novel resource: it is the first benchmark, to my knowledge, targeting mathematical reasoning over classical Chinese texts, and the dual closed-book/open-book design is a useful way to separate comprehension from the ability to follow an ancient solution procedure. The paper also ships the dataset, code, and experimental results openly, which is a concrete strength. The error taxonomy, while preliminary, offers a reasonable framework for diagnosing failures in low-resource, culturally embedded reasoning tasks. The main caveat is that the numerical results are only as strong as the unvalidated LLM-based answer-equivalence judge, so the benchmark's contribution is currently more credible as a dataset artifact than as an established set of accuracy numbers.

major comments (4)
  1. [Section 2.3, Tables 5-6] Every accuracy figure in Tables 5 and 6 is produced by a single LLM judge, QwQ-32B, applying a hand-built unit-conversion table, with no human validation, agreement statistic, or judge-error analysis reported. The judge prompt instructs that answers equal after simple conversion or added units should be treated as correct, but Section 3.4's own example of a numerically correct value expressed as an inverted fraction (81/56 instead of 56/81) illustrates how easily scoring depends on the judge's interpretation of format and historical notation. Because the headline claim (63.01% closed-book, 70.25% open-book) and the model ranking rest entirely on these binary judgments, I ask for a human-scored validation sample of, say, 100-200 items with per-model agreement rates and corrected accuracies, or a sensitivity analysis showing that the reported conclusions are robust to plausible judge misclassification rates.
  2. [Sections 2.1.2 and 3.3] The dataset-size accounting is internally inconsistent. Section 2.1.2 states that 27 problems lack explicit solution steps and reports 511 problems with solution steps, while Section 3.3 says the open-book mode used a dataset 'adjusted to Level 1: 25, Level 2: 266, Level 3: 178, Level 4: 42,' which sums to 511, yet two sentences later claims 'a reduction of 23 data points.' Since every open-book accuracy in Table 6 depends on the exact denominator, the correct reduction (23 or 27) must be stated and used consistently throughout the text and tables.
  3. [Section 2.1.3, Tables 2-3, Figures 5-6] The difficulty levels and problem-type labels are generated by a closed-source Qwen reasoning model with no reported human agreement or validation, and the error-cause analysis in Section 3.4 also relies on LLM-generated categories and annotations with only a vague mention of human annotators. The type-level accuracy analyses in Figures 5-6 and the difficulty-stratified results in Tables 5-6 therefore inherit any systematic biases in the auto-generated labels. Please report inter-annotator agreement on a sample (e.g., Cohen's kappa between the model and human annotators) or provide a manual audit of the labels; without this, conclusions such as 'models achieve their highest accuracy in linear equations/systems' are not firmly grounded.
  4. [Sections 3.2 and 4] The claim that 'two trillion-parameter models (DeepSeek R1 and Qwen-Plus)' outperform smaller models is not factually supported. DeepSeek R1 has 671B total parameters, not two trillion, and Qwen-Plus-2025-04-28's parameter count is not disclosed anywhere in the manuscript. The discussion's 'parameter scale effects' conclusion therefore conflates model scale with other differences such as training data, post-training procedures, API access, and inference infrastructure. Please either rephrase the scale claim to what the data actually show (e.g., API-based models outperformed locally deployed 32B models) or provide verifiable parameter counts for all models.
minor comments (5)
  1. [Section 3.1, Table 4] Table 4 lists repetition_penalty = 1, which is the neutral/default value, but the text says 'a higher repetition penalty factor was applied'; please reconcile the value in the table with the prose, and also confirm whether '32678' is a typo for '32768'.
  2. [Section 3.4, Table 7] Table 7's 'Annotation Count' sums to 512, although only 199 problems were answered incorrectly by DeepSeek R1 in closed-book mode; because multi-label classification is allowed, please state explicitly in the caption or text that the counts are label occurrences, not numbers of problems.
  3. [Sections 3.2 and 3.4] The comparison to Math-500 is based on 'reported performance' rather than a controlled re-evaluation under the same prompts, decoding settings, and answer-extraction procedure; given the use of temperature 0.6 and the appended </think> token, the claimed gap to modern benchmarks should be stated as an approximate comparison or supported by running the same models on Math-500 under identical conditions.
  4. [Throughout] There are numerous typos and terminological inconsistencies, such as 'Suanshu Shijin' for 'Suanjing Shishu' in Section 2.1.3, 'Data is applicable' in the Data availability statement, and inconsistent formatting of reference entries; a thorough language and consistency edit is needed.
  5. [Section 2.1.2 and Table 1] Table 1 reports zero extracted problems for Zhou Bi Suan Jing and Shushu Jiyi, and the text explains that these texts contain method introductions rather than fixed-answer problems; if so, the paper should consistently refer to the benchmark as covering eight texts rather than the 'Ten Mathematical Classics' whenever describing the actual experimental data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline accuracies are independent measurements against externally sourced ground-truth answers; the LLM-judge and in-house punctuation tool are validity caveats, not circular derivations.

full rationale

The paper's central claims are empirical measurements rather than derivations from fitted quantities. Ground-truth answers and solution steps were extracted from primary sources (Ctext, Qian Baocong's 1963 edition) and then manually verified, so the benchmark is not defined in terms of the models being evaluated. Model outputs are scored by an LLM judge using a hand-built unit-conversion table, but the judge's decisions are not statistically forced by any fitted parameter; the conversion table is an external metrological standard taken from Needham's work and the texts themselves. The use of the authors' own Xunzi model for punctuation is a preprocessing step followed by manual review, so it does not carry the argument. Difficulty and problem-type labels were annotated independently of model performance. The comparison to Math-500 uses reported numbers but does not reduce the Guji_MATH measurement to those numbers. No uniqueness theorem, ansatz smuggling, or renaming of a known result appears. The unvalidated QwQ-32B judge and the reliance on LLM annotation are validity and reproducibility risks, but they are not circularity: no equation or construction in the paper equates the evaluation outcome to its own input by definition.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces a benchmark dataset and evaluation protocol but no new theoretical entities. The central empirical claim rests on domain assumptions about text fidelity, premise completeness, LLM-judge reliability, and taxonomy exhaustiveness. The only free parameters are inference hyperparameters, which are not fitted to maximize accuracy.

free parameters (4)
  • temperature = 0.6
    Sampling temperature chosen from DeepSeek R1 and Qwen guidelines, affects generation but is not fitted to the result.
  • top_p = 0.95
    Nucleus sampling threshold from guidelines, not fitted to data.
  • top_k = 20
    Sampling candidate count from guidelines, not fitted to data.
  • repetition_penalty = 1 (Table 4; text mentions a higher factor)
    Repetition penalty setting; the text and table disagree about the value used, a minor inconsistency.
assumptions (4)
  • domain assumption The digitized texts from Ctext.org are faithful transcriptions of Suanjing Shishu, and the Xunzi model punctuation is accurate enough for problem extraction.
    Section 2.1.1: texts downloaded from Ctext, six texts OCR'd from Siku Quanshu and punctuated by Xunzi model without reported manual correction of punctuation errors.
  • domain assumption The Question-Answer-Solution triplet structure plus manually appended premises captures every problem's full conditions.
    Section 2.1.2: authors add hidden and shared premises manually; completeness is unverified, and 27 problems lack explicit solution steps.
  • domain assumption The LLM judge QwQ-32B, guided by the provided unit conversion table, correctly determines answer equivalence.
    Section 2.3: no human validation of judge decisions is reported; all accuracy metrics depend on this assumption.
  • domain assumption Needham's classification of 15 mathematical methods is exhaustive for the problems in Suanjing Shishu.
    Section 2.1.3.1: used to generate problem-type labels; the taxonomy may not cover all solution methods present in the texts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can reasoning models comprehend mathematical problems in Chinese ancient texts? An empirical study based on data from Suanjing Shishu." pith.science (2026). https://pith.science/paper/P65H6L6W

@misc{pith2026250516660,
  author       = {Pith},
  title        = {Pith review of: Can reasoning models comprehend mathematical problems in Chinese ancient texts? An empirical study based on data from Suanjing Shishu},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P65H6L6W}},
  note         = {Machine review of arXiv:2505.16660}
}
read the original abstract

This study addresses the challenges in intelligent processing of Chinese ancient mathematical classics by constructing Guji_MATH, a benchmark for evaluating classical texts based on Suanjing Shishu. It systematically assesses the mathematical problem-solving capabilities of mainstream reasoning models under the unique linguistic constraints of classical Chinese. Through machine-assisted annotation and manual verification, 538 mathematical problems were extracted from 8 canonical texts, forming a structured dataset centered on the "Question-Answer-Solution" framework, supplemented by problem types and difficulty levels. Dual evaluation modes--closed-book (autonomous problem-solving) and open-book (reproducing classical solution methods)--were designed to evaluate the performance of six reasoning models on ancient Chinese mathematical problems. Results indicate that reasoning models can partially comprehend and solve these problems, yet their overall performance remains inferior to benchmarks on modern mathematical tasks. Enhancing models' classical Chinese comprehension and cultural knowledge should be prioritized for optimization. This study provides methodological support for mining mathematical knowledge from ancient texts and disseminating traditional culture, while offering new perspectives for evaluating cross-linguistic and cross-cultural capabilities of reasoning models.

Figures

Figures reproduced from arXiv: 2505.16660 by the authors.

Figure 3
Figure 3. Utilizing Default Premise Conditions in Mathematical Problems from Classical Texts [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Example of Evaluation Data 2.2 Evaluation Methods and Model Selection 2.2.1 Evaluation Methods For evaluation, we modified prompts and adopted two distinct strategies to assess the performance of open-source and closed-source reasoning models: (1) Closed-Book Mode: Only the problem source, question description, and premise conditions from the JSON tuple were included in the prompt, requiring the model to directly ou… view at source ↗
Figure 5
Figure 5. Bar Chart of Mathematical Problem-Solving Performance by Reasoning Models in Closed-Book Mode The above chart illustrates the performance of reasoning models across different mathematical problem types. Excluding problems with extremely low sample sizes (e.g., linear programming, higher-degree equations, and arithmetic/geometric sequences), all models demonstrated strongest performance in basic arithmetic operations… view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Bar Chart of Mathematical Problem-Solving Performance by Reasoning Models in Open-Book Mode [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Co-occurrence Network Map of Error Labels As shown in [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 2 canonical work pages

  1. [1]

    Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024). Large Language Models for Mathematical Reasoning: Progresses and Challenges (arXiv:2402.00157). arXiv. https://doi.org/10.48550/arXiv.2402.00157 Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., ...

    arXiv doi:10.48550/arxiv.2402.00157 2024

  2. [3]

    Cambridge University Press

    Mathematics and the sciences of the heavens and the earth. Cambridge University Press. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training languag...

    arXiv doi:10.48550/arxiv.2412.15115 2022

  3. [9]

    Summary of Model Prompts Used in the Study Task Type Prompt Model Prompt Example Text Punctuation Xunzi- Qwen1.5- 7B_chat 请为以下文本添加标点符号：{text} Text Structuring Qwen2.5- 14B-Instruct 接下来将给你一段中国古代数学典籍文本， 里面可能涉及到 多组数学问题，及其对应的答案和解析，请你使用将每组问 题、答案和解析都抽取出来.其中问题往往以‘问曰’开头， 答案通常以‘答曰’开头，解析通常以‘术曰’开头，请你 按照 json 格式输出多组抽取结果，json 的属性是：问题、答 案和解析。如果某项不存在则输出为空 \n 给定的文本 为:{te...

    work page 2025

  4. [30]

    Y., Guu, K., Yu, A

    https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa- Abstract.html Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2022). Finetuned Language Models Are Zero -Shot Learners (arXiv:2109.01652). arXiv. https://doi.org/10.48550/arXiv.2109.01652 Wei, J., Wang, X., Schuurmans, D., Bos...

    arXiv doi:10.48550/arxiv.2109.01652 2022

  5. [1901]

    https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a- Abstract.html OpenAI. (2023). ChatGPT (May 2,

    work page 2023

  6. [2023]

    新趋势、 新议题、新期待：中国数学史研究的理论与实践

    Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., & Qiu, X. (2024). Evaluating the Performance of Large Language Models on GAOKAO Benchmark (arXiv:2305.12474). arXiv. https://doi.org/10.48550/arXiv.2305.12474 Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024). Retrieval-Augmented Generation for AI -Gener...

    arXiv doi:10.48550/arxiv.2305.12474 2024

  7. [2025]

    https://chatgpt.com/ Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., & Che, W

    [Large language model]. https://chatgpt.com/ Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., & Che, W. (2025). Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models (arXiv:2503.09567). arXiv. https://doi.org/10.48550/arXiv.2503.09567 Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M....

    arXiv doi:10.48550/arxiv.2503.09567 2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.