A Report on the llms evaluating the high school questions

Chen Wei; Zhu Jiawei

REVIEW 4 major objections 5 minor 1 references

A Report on the llms evaluating the high school questions

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper ranks eight LLMs on Gaokao math and crowns Qwen2.5-7B-Instruct, even as multiple-choice accuracy sits at 10-20 percent.

desk verdict The paper's own tables sink its ranking: weighted scores contradict the claimed order, accuracy claims switch leaders, and the grading pipeline is unvalidated. read the letter →

arxiv 2505.00057 v1 pith:SRDGFFJL submitted 2025-04-30 cs.CL

Zhu Jiawei , Chen Wei This is my paper · ORCID

classification cs.CL

keywords largelanguagemodelsGaokaomathematicsLLMevaluationautomaticgradingchain-of-thoughtpromptingeducationaltechnologyresponsetimebenchmarkingcreativity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a quantitative benchmark for large language models in education by testing eight LLM APIs on Chinese college-entrance (Gaokao) mathematics questions from 2019 to 2023. The authors categorize questions by type and difficulty, collect single- and multi-solution answers from each model, and score them on accuracy, response time, logical reasoning, prompt guidance, and creativity. Their main conclusion is that model performance depends heavily on question format: multiple-choice accuracy is strikingly low (0.10-0.20 for every model), while comprehensive-question accuracy reaches 0.70-0.95; in the overall weighted ranking they name Qwen2.5-7B-Instruct first and GLM-4-Flash second, though their own Table 15 gives GLM-4-Flash a higher weighted score. If the evaluation pipeline is sound, the results would give educators and developers a usable basis for choosing models and for steering them with prompts.

What carries the argument

The carrying instrument is the paper's four-way evaluation pipeline. For multiple-choice questions it uses direct string matching after answer normalization; for fill-in-the-blank it uses fuzzy semantic matching built on word-vector embeddings; for comprehensive questions it uses an AI-assisted neural grader plus random expert re-scoring of one tenth of the answers; and creativity is measured by decomposing multi-solution outputs with regular expressions and greedy scanning, then scoring correctness, richness, complexity, logic, and time. These scores feed weighted tables, especially Table 15, that produce the final model ranking.

What would settle it

Take a random sample of the model outputs previously graded by the pipeline, regrade them with independent human experts using the standard answers, and measure agreement; if agreement on multiple-choice or comprehensive questions is low or systematically tilted in one direction, the reported accuracy differences and rankings will not survive.

Watch

Extended reading notes

Core claim

The report's central discovery, stated on its own terms, is a comparative performance profile of eight LLMs on a curated set of Gaokao math problems. It reports that all models answer only 10-20% of multiple-choice questions correctly under direct string matching, whereas on comprehensive questions they score between 0.70 and 0.95, with ERNIE-Speed-128K at 0.95 and Qwen2.5-7B-Instruct at 0.85; multi-solution accuracy is highest for Qwen2.5-7B-Instruct (0.64) and ERNIE-Speed-128K (0.63). It also reports that prompt framing as a mathematician, exam student, or teacher improves accuracy by up to 22% (GLM-4-Flash under the student prompt). The intended headline conclusion is that Qwen2.5-7B-Instruct performs best across all metrics and weights; the table accompanying that conclusion assigns GLM-4-Flash the higher weighted score (0.983 vs 0.812), so the ranking as printed is internally inconsistent.

Load-bearing premise

The entire ranking assumes the automatic grading pipeline labels model outputs correctly, even though the paper never checks that pipeline against an independent human-scored gold set.

Editorial extensions

If this is right

If the accuracy figures hold, LLMs in their current form are unreliable for standardized multiple-choice math assessment, scoring at or below chance, while being much more useful on open-ended problems.
Prompt guidance is a practical lever: framing the task with an expert or student persona raised accuracy for most models by 2-22 percentage points, so deployment should tune prompts, not just models.
Speed separates models by an order of magnitude (0.32 s vs 4.50 s on multiple-choice; 688 s vs 7400 s for multi-solution batches), which matters for real-time tutoring applications.
The weighted score is intended as a composite education-suitability score, giving later work a starting point even though the printed ranking is internally inconsistent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 10-20% multiple-choice accuracy is likely an artifact of strict string matching plus answer-format mismatch; if the grader were validated against human labels, the choice accuracy would probably rise, but the paper does not test this.
Because Table 15 gives GLM-4-Flash (0.983) a higher weighted score than the prose winner Qwen2.5-7B-Instruct (0.812), the printed ranking cannot be reproduced from the paper's own table; any user should recompute the weights before trusting the winner.
The described pipeline could be turned into a proper benchmark by human double-scoring a random sample of outputs; until then the accuracy numbers and rankings are conditional on the unvalidated automatic grader.
Neighboring educational uses, such as personalized tutoring, automatic homework grading, and exam generation, inherit all of these caveats, so the report is best read as a pilot rather than a foundation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read on 2505.00057.

The paper tries to do something useful: compare eight LLM APIs on Gaokao mathematics questions across accuracy, latency, logic, guidance, and creativity, with two prompt conditions. That design is reasonable for a practical evaluation. But the execution doesn't hold up. The central deliverable — the ranking in Section 5.5.3 — is contradicted by the paper's own Table 15. Qwen2.5-7B-Instruct is declared first with weighted score 0.812, while GLM-4-Flash scores 0.983 and is placed second. Yi-34B scores 0.607 and is ranked eighth while Meta-Llama-3.1-8B-Instruct scores 0.483 and is ranked seventh. No monotone rule maps those numbers to that order. Table 17 also states Spark-lite ranks first in accuracy, but Tables 12 and 13 show ERNIE-Speed-128K with the highest comprehensive-question accuracy (0.95) and Qwen with higher multiple-choice and fill-in-the-blank accuracy than Spark-lite. There's no aggregation that makes Spark the accuracy leader.

The soft spots are load-bearing, not cosmetic. Section 4.1.1's grader is unvalidated: exact string matching for multiple-choice, an unspecified fuzzy method for fill-in-the-blank, and an AI-assisted score with only a 10% expert sample for comprehensive questions. No independent gold set is used. The reported multiple-choice accuracies (0.10–0.20) are below random for four-choice items, while comprehensive accuracy is 0.70–0.95; that pattern strongly suggests the grader is broken, not that models are worse at multiple choice.

The presentation also undermines confidence. Table 9's prompt definitions are swapped: the single-solution prompt asks for multiple solutions, and the multi-solution prompt asks for only the answer. The weighted creativity score uses arbitrary weights, thresholds like the 80% multi-solution correctness rule and the adaptive fuzzy threshold are unsupported, and no error bars, statistical tests, or data/code are provided. The reference list is thin and doesn't cite existing Gaokao benchmarks (e.g., GaokaoBench), so the paper situates itself poorly in the literature.

Credit where it's due: the question selection across exam years and difficulty bins is thoughtful, and the idea of logging multi-solution richness and complexity is a reasonable direction for evaluating creativity. But those seeds aren't enough to rescue an evaluation whose headline results are internally inconsistent.

Who is this for? Maybe a practitioner who wants a scrap of comparative latency data across these APIs — but even that is uncertain given no release of the prompts or data. This paper should not be sent to an external referee; it needs major methodological work before it merits review.

My advice: desk-reject, and tell the authors to fix the table inconsistencies, validate the grader, and release data if they resubmit.

Referee Report

4 major / 5 minor

Summary. This manuscript reports an empirical evaluation of eight large language models (GLM-4-Flash, ERNIE-Speed-128K, Spark-lite, hunyuan-lite, Qwen2.5-7B-Instruct, gemma-2-9b-it, Meta-Llama-3.1-8B-Instruct, and Yi-34B) on a set of Chinese Gaokao mathematics questions from 2019 to 2023. The authors define metrics for accuracy, response time, logical reasoning, guidance, and creativity, then present per-model tables and figures, and conclude with a comprehensive ranking in which Qwen2.5-7B-Instruct is ranked first. The central deliverable is this ranking and the claim that the data provide a solid quantitative foundation for assessing LLMs in education.

Significance. If the evaluation pipeline and ranking were internally consistent, this study could provide a useful practical benchmark for educational deployment of LLMs, because it uses real exam questions and compares eight readily available models across multiple dimensions. The paper also has the strength of separating single-solution and multi-solution responses, and of reporting response-time statistics. However, the significance is undermined by multiple internal contradictions in the reported numbers, and by an unvalidated automatic grading pipeline. As presented, the results do not support the paper's central claims, so the contribution is currently not usable by the community.

major comments (4)

[Section 5.5.3, Table 15] The weighted scores in Table 15 are not consistent with the stated ranking. Qwen2.5-7B-Instruct is given a weighted score of 0.8124917 but is ranked first, while GLM-4-Flash has a score of 0.983135 and is ranked second. Similarly, Yi-34B has a score of 0.6072355 and is ranked eighth, whereas Meta-Llama-3.1-8B-Instruct has a lower score of 0.483141 and is ranked seventh. A higher weighted score should correspond to a better rank, but this ordering is reversed in two places. Because the ranking is the main conclusion of the paper, this contradiction is load-bearing.
[Section 5.6, Tables 16 and 17 versus Section 5.2.1, Table 12] Table 17 states that Spark-lite and Qwen2.5-7B-Instruct rank first in accuracy, but Table 12 shows ERNIE-Speed-128K with the highest comprehensive-question accuracy (0.95) and Spark-lite with the lowest multiple-choice accuracy (0.10). No aggregation rule is provided that would make both Qwen and Spark-lite the accuracy leaders, and the stated accuracy ranking is therefore not reproducible from the paper's own data. This inconsistency directly affects the per-metric conclusions in Section 5.6.
[Section 4.1.1 and Tables 12 and 17] The automatic grading pipeline is not validated against an independent human-labeled gold set, and the resulting numbers are implausible in a way that suggests grader failure. Multiple-choice accuracies of 0.10 to 0.20 for all models are at or below random guessing for four-option questions, while the same models show comprehensive-question accuracies of 0.70 to 0.95. The paper does not explain how a model could score below chance on multiple-choice questions while scoring very highly on open-ended comprehensive questions. Since the accuracy metric is central to the ranking, the lack of grader validation is a load-bearing weakness.
[Section 4.5 and Figure 24] The creativity composite weighted score used for the final ranking is not fully specified. The paper lists five dimensions (correctness, solution diversity, solution richness, time taken, and solution complexity) and shows a bar chart labeled 'Factor percentage', but it never gives the exact formula or the weight vector that maps the raw dimensions to the weighted scores in Table 15. Without this information, the top-level ranking in Table 15 is not reproducible, and the claim that one model 'performs the best when considering all metrics and weights' cannot be independently verified.

minor comments (5)

[Table 3] The definitions for 'Advanced Questions' and 'Difficult Questions' are duplicated verbatim, and both contain the same error about difficulty factors below 0.3, which makes the table confusing.
[Tables 10 and 11] Both tables are labeled 'Data synthesis tables', and one of them contains a column headed with the Chinese word for 'difficult' rather than an English description; the duplicated numbering should be corrected.
[Figures 12-16] Several figure captions contain typos, such as 'Average orrectness' instead of 'Average correctness', and the axis labels in Figures 14 and 15 are similarly misspelled.
[Section 4.3] The formula for the comprehensive score includes the term '(Guidance Attempts /1 × 0.6)', where the division by 1 is meaningless and likely a typographical error; the intended formula should be clarified.
[Section 5.1.3] The model name 'ERNIE-Speed-128K' is misspelled as 'LRNE-Speed-128K' in the text of Section 5.1.3, which is distracting and should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is a purely empirical measurement report with no fitted-input/prediction loop and no load-bearing self-citation chain.

full rationale

The paper is an evaluation study: it measures model outputs on a fixed question set and reports accuracies, times, and composite scores. There is no derivation chain in which a predicted quantity reduces to a fitted parameter, because nothing is fitted and nothing is predicted. The weighted creativity and comprehensive scores are author-chosen aggregations of measured metrics, so the weighted score is defined by those metrics rather than being an independent claim smuggled back in as evidence. The only self-referencing elements, such as citing the authors' own CCoT-related or chain-of-thought references, are not load-bearing: the logical-reasoning metric is defined operationally by guidance attempts and success rates, not by a uniqueness theorem or by a prior result asserted by the same authors. The internal contradictions between Table 15's weighted scores and the claimed ranking, and between Table 17's accuracy claims and Table 12, are correctness/consistency problems, not circularity: they do not show that an output is equivalent to an input by construction. Likewise, the questionable validity of the automatic grader and the 10% expert review is a measurement-integrity concern, not a circular derivation. Since the report is self-contained as an empirical benchmark and contains no step where the conclusion is identical to its premises, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central ranking rests on unpublished composite weights, arbitrary thresholds, and unvalidated automatic grading assumptions. The paper invents no physical or formal entities, but it introduces ad hoc scoring constructs whose coefficients are not fully specified and whose validity is untested.

free parameters (4)

Creativity composite score weights
Section 5.5.3 reports a comprehensive weighted score but does not list the weights; the final rankings depend on these unpublished coefficients.
Multi-solution correctness threshold = 80%
Section 4.1.2: a multi-solution response is judged correct if at least 80% of its solutions are correct; this threshold is chosen without justification and changes the accuracy measure.
Fuzzy matching adaptive threshold
Section 4.1.1: an adaptive threshold setting mechanism dynamically adjusts similarity thresholds by difficulty, but no formula or calibration is given, leaving the grading rule unspecified.
Difficulty factor bins = 0.0-0.3, 0.3-0.5, 0.5-0.7, 0.7-1.0
Section 2.2: question difficulty is binned by hand-assigned difficulty factors from unspecified national assessment data; Table 3's text for intermediate and advanced rows is identical and internally inconsistent.

assumptions (4)

domain assumption The eight API models' outputs are deterministic or adequately controlled by the chosen prompts and temperature settings.
Section 3.2 and 4.5 state temperature is adjusted for multi-solution responses but never specify the value or number of runs, so the accuracy numbers implicitly assume one sample per question is representative.
domain assumption The hand-assigned standard answers in LaTeX and the string or fuzzy matching against them reflect true mathematical correctness.
Section 2.3 stores an 'An' field as standard answer; Section 4.1.1 grades by optimization and fuzzy matching, but there is no human validation of the matching for free-form answers.
domain assumption The expert re-evaluation of 10% of comprehensive answers is an unbiased ground truth that validates the AI-assisted grader.
Section 4.1.1 states a random 1/10 sample is re-evaluated by domain experts, yet no expert credentials, grading rubric, or inter-annotator agreement is reported.
domain assumption The hand-selected Gaokao questions represent the full distribution of high school mathematics difficulty and question types.
Section 2.2 selects a fixed pattern of question numbers per paper, excludes diagrams, and evenly distributes difficulty, but no statistical justification or comparison to the full Gaokao population is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Report on the llms evaluating the high school questions." pith.science (2026). https://pith.science/paper/SRDGFFJL

@misc{pith2026250500057,
  author       = {Pith},
  title        = {Pith review of: A Report on the llms evaluating the high school questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRDGFFJL}},
  note         = {Machine review of arXiv:2505.00057}
}

read the original abstract

This report aims to evaluate the performance of large language models (LLMs) in solving high school science questions and to explore their potential applications in the educational field. With the rapid development of LLMs in the field of natural language processing, their application in education has attracted widespread attention. This study selected mathematics exam questions from the college entrance examinations (2019-2023) as evaluation data and utilized at least eight LLM APIs to provide answers. A comprehensive assessment was conducted based on metrics such as accuracy, response time, logical reasoning, and creativity. Through an in-depth analysis of the evaluation results, this report reveals the strengths and weaknesses of LLMs in handling high school science questions and discusses their implications for educational practice. The findings indicate that although LLMs perform excellently in certain aspects, there is still room for improvement in logical reasoning and creative problem-solving. This report provides an empirical foundation for further research and application of LLMs in the educational field and offers suggestions for improvement.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 linked inside Pith

[1]

1 Nayab, S., Rossolini, G., Buttazzo, G., Manes, N., Giacomelli, F., & Fabrizio, G. (2024). Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost. arXiv:2407.19825. 2 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR. 3 Liu, Y ., & Zhang, Y ...

arXiv 2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

1 Nayab, S., Rossolini, G., Buttazzo, G., Manes, N., Giacomelli, F., & Fabrizio, G. (2024). Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost. arXiv:2407.19825. 2 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR. 3 Liu, Y ., & Zhang, Y ...

arXiv 2024