Pith. sign in

REVIEW 4 major objections 5 minor 20 references

From Struggle (06-2024) to Mastery (02-2025) LLMs Conquer Advanced Algorithm Exams and Pave the Way for Editorial Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The newest large language models score at the level of top students on a challenging university algorithms exam, with o3-mini reaching the 98th percentile and the top 3% of the class.

desk verdict Useful dataset and a plausible trend, but the unspecified graph-input protocol and inconsistent numbers undercut the top-5% claim. read the letter →

arxiv 2506.04965 v1 pith:4SKBEEKA submitted 2025-06-05 cs.CL cs.AI

Adrian Marius Dumitran , Theodor-Pierre Moroianu , Vasile Paul Alexe This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsalgorithmsexamsgrapheducationalassessmentmultilingualevaluationLLMconsistencyeditorialgenerationautomatedgrading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests fourteen state-of-the-art large language models on a real university-level advanced algorithms exam, given in Romanian and in English translation, and scores each model's single-session attempt by the same instructor who graded the human students. Its central claim is that the newest models—released in the months before the test—have crossed a threshold: o3-mini scores 86.60 points, placing it in the top 3% of the student cohort, and Gemini 2.0 Flash scores above 70, placing it in the top 15%, while models older than about six months mostly land in the bottom half. The paper reads this as evidence that frontier models can now serve as useful partners in advanced algorithm education, and it reports on a human-in-the-loop grading-scheme generator and an editorial-generation platform built on that idea. It also documents remaining weaknesses, particularly on problems that require reading a graph from a drawing.

What carries the argument

The load-bearing apparatus is the one-shot exam interaction: each model receives the entire 11-problem exam in a single prompt through a vendor user interface with default settings, and the course instructor grades the response with the same rubric used for students. Percentile placement within the real student grade distribution, grouped by model release date, is what converts raw scores into the 'top 3%' and 'top 15%' claims. Consistency is quantified by the standard deviation of final grades over five independent runs, and collaboration is tested by feeding each model either its own previous answers (RunAvg) or the answers of three other models (RunAvgAll). The release-date split (models released within the last four months vs. older) is the paper's main explanatory axis for score differences.

What would settle it

Inspect the recorded sessions or rerun the exam with controlled inputs, giving o3-mini and Gemini 2.0 Flash the same 11 problems with the graph sent as an image, as adjacency lists, and omitted entirely; if per-problem scores on the visual tasks do not differ across conditions, the paper's visual-analysis explanation is wrong, and if the figure was never sent in the original runs, the student-percentile comparison is not measuring the same task.

Watch

Extended reading notes

Core claim

The paper's claim is that LLM capability on hard algorithmic exams has improved so sharply with recent releases that the best models are no longer below-average test takers but top-quartile or top-decile performers: in the authors' evaluation, o3-mini averages 86.60 points (98th student percentile), o3-mini-high 84.00 (97th), and Gemini 2.0 Flash 72.30 (88th), while GPT-4 Legacy, Qwen2.5Max and others score below 40 and rank in the bottom half. The same exam given in Romanian and English shows that recent models have near-balanced multilingual performance, whereas older models gain as much as 27.5 points when the exam is in English. The authors also claim that the strongest models are consistent across repeated runs (standard deviation around 2.4–2.7 points) and that feeding a model the solutions of other models can substantially lift weaker solvers. On graph questions that require visual reasoning, however, most models still fail, which the authors attribute to limited multimodal interpretation.

Load-bearing premise

The whole top-percentile conclusion assumes every model received the same exam content, including the graph figure that Problems 1–6 rely on, but the paper never says whether the figure was actually shown to them.

Editorial extensions

If this is right

  • If the central claim holds, advanced algorithm courses using similar exams should expect top LLMs to score at or above the level of most enrolled students in a one-shot setting.
  • The low standard deviation of the best models (SD about 2.4 to 2.7 points) means their performance is reproducible enough to be used in grading support, not a lucky outlier.
  • The multilingual result implies recent models can serve non-English-speaking classrooms on technical content without the strong English bias of older systems.
  • The RunAvgAll result suggests that exposing a weaker model to stronger models' solutions can raise its score by 12 to 15 points, a cheap way to improve tutoring tools.
  • Visual graph-reading tasks remain a boundary: until models can reliably interpret a graph drawing, fully automatic grading of such exams is not on the table.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never states whether the graph figure was sent to the models, so the visual-task conclusions are only as strong as that implicit assumption; if the drawing was omitted for text-only models, the student-percentile comparison would not be apples-to-apples.
  • A direct testable extension is to re-run a subset of the models with the graph supplied as an image versus as an adjacency list; if scores on Problems 1–3 converge, the visual-analysis story would need revision.
  • If the trend in the release-date plot continues, within a year even mid-tier open-weight models may pass this exam at a passing level, which would change how take-home algorithm assessments are designed.
  • The RunAvgAll gains suggest a practical ensemble strategy for educational tools: route a student's attempt to multiple models and present the best joint answer, which the paper did not itself test as a student-facing feature.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates a range of LLMs on a university-level advanced-algorithms exam originally written in Romanian, using one-shot interactions through vendor UIs and a separately translated English version. It reports that newer models, particularly o3-mini and Gemini 2.0 Flash, score at or above the top percentiles of the student grade distribution; it also presents consistency measurements across repeated runs, a small collaboration experiment, a multilingual comparison, and a prototype platform for generating grading schemes and editorial feedback. The main empirical claim is that recent models have crossed a threshold where they perform on par with top students on this exam, while graph-based and visually presented tasks remain weak.

Significance. If the headline result is methodologically sound, the paper would provide a useful, timely benchmark for LLM progress on complex algorithmic reasoning, and the released exam dataset in Romanian is a valuable low-resource contribution. The instructor-scored evaluation and the one-shot exam protocol are appropriate choices for an out-of-the-box usage scenario, and the consistency and collaboration experiments, although small, are creative extensions. However, the central 'top 5%' and 'top 3%' claims rest on an assumption—that all models received the same exam input, including the graph figure—that is not documented. Because several evaluated models are accessed through text-only interfaces and the paper's own future-work section calls for multimodal capabilities, the current procedure likely did not provide the graph image to all models. This makes the student-vs-LLM comparison and the visual-analysis interpretation of per-problem failures unverifiable without additional details or a rerun.

major comments (4)
  1. [Section 3.1, Section 3.6, Appendix A] The exact input protocol for graph-based problems is never specified. Problems 1–6 and Problem 8 require a graph drawing or network, and the paper does not state whether each model received the figure from Figure 6 as an image, an adjacency list, a textual description, or no graph at all. This matters because the comparison in Table 4 and the percentile claims in Table 3 assume students and LLMs solved the same exam. If text-only models such as DeepSeekR1, Llama3.3-70B, and Mixtral-8x22B did not receive an image, their low scores on Problems 1 and 3 may reflect an under-specified task rather than a failure of 'visual analysis' as claimed in Section 3.6. The paper's own Section 6 lists multimodal interpretation as future work, which further suggests the current pipeline was not multimodal. Please specify, per model and per problem, exactly how Figure 6 and the Problem 8 network were supplied, and if the image was omitted for any model, rerun the affected conditions with a consistent textual encoding (and, where applicable, the actual image) so that the equivalence with the student exam is supported.
  2. [Section 3.5, Table 3, Figure 2] The percentile comparison lacks necessary statistical grounding. The student cohort size is not reported, nor are the distribution moments or confidence intervals for the percentiles. The table treats each LLM grade as a fixed point, although many models were apparently run once, so the difference between the 97th and 98th percentile for o3-mini is not meaningful without an interval. The additional statement that o3-mini is 'likely top 1% nationwide' is an extrapolation beyond the data presented and should be removed or supported with the relevant national statistics.
  3. [Sections 3.1, 3.3, 3.4, Tables 1 and 2] Most of the headline scores in Table 3 come from a single run, and the consistency and collaboration experiments use very small samples: three models with five runs each in Section 3.3, and one run per cell in Section 3.4. This is sufficient to show directional evidence, but not to support the strength of the conclusions drawn, such as a strong correlation between consistency and performance or the claim that collaboration reliably improves weaker models. Please report the number of runs used for each number in Table 3, add repeated runs for at least the models whose percentiles are being highlighted, and present ranges or standard deviations for the collaborative results.
  4. [Section 3.7, Table 5] The grouping of models into Group A and Group B by a cutoff of 'after 09-2024' is arbitrary, and the conclusion that Group A shows a 'statistically unimportant difference' is not backed by any statistical test. With six models in Group B and twelve in Group A, plus heterogeneous models within each group, the aggregate deltas should be interpreted descriptively rather than as a demonstrated trend in multilingual ability. Please either add an appropriate test (e.g., a permutation test on the deltas) or soften the language to avoid implying a conclusion not supported by the sample.
minor comments (5)
  1. [Section 3.1] The paper says models are selected 'from the ChatBot Arena LLM Leaderboard' but the selection criteria beyond 'a few from each family' are not given; a sentence on inclusion criteria (e.g., popularity, accessibility, recency) would help reproducibility.
  2. [Section 3.2, Figure 1] The recency cutoff of 'four months' is used to define the two groups in Figure 1, while the introduction refers to 'older than six months'; please align these definitions and state why the particular cutoff was chosen.
  3. [Table 3] The model name '4o' appears without the 'GPT-' prefix, unlike the naming convention used elsewhere for GPT-4 Legacy; please make the naming uniform.
  4. [Section 3.3, Table 1] The standard deviation alone is a reasonable consistency measure, but reporting the full set of five run scores for each model would make the 'black swan' run for Sonnet 3.5 more transparent and would let readers assess skew as well as spread.
  5. [Section 4.2] The platform section describes prompt engineering that adds 'on average 2000 additional characters' to requests, but no example of the final prompt is included; adding the template would materially improve the reproducibility of the editorial-generation part.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark with non-load-bearing self-citations, not a derivation that reduces to its own inputs.

full rationale

This paper is an empirical evaluation study rather than a derivation chain. The central claims—that newer LLMs such as o3-mini and Gemini 2.0 Flash score in the top percentiles of the student distribution—are direct measurements: LLM responses were scored by the course instructor and compared with the student grade distribution. No parameter is fitted to the data and then renamed as a prediction, and no target quantity is defined in terms of another quantity it is supposed to predict. The self-citations [3], [4], and [5] point respectively to the authors' prior competitive-programming study, the released exam dataset, and the grading-scheme repository; none of these is used to justify the central claim or to rule out alternatives. The exam statement is reproduced in Appendix A, and the per-model and per-problem scores are reported in Tables 3–5, making the evaluation externally checkable. The paper's main weakness—the unspecified way the graph figure for Problems 1–6 was presented to text-only models—is an experimental-validity concern about whether LLMs and students solved the same task, not a circularity: it does not make the reported scores equivalent to the input by construction. There are no equations that reduce to themselves, no fitted inputs renamed as predictions, and no load-bearing uniqueness theorems imported from the authors' prior work. The empirical results stand independently of the self-citations, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted model parameters or new scientific entities. Its claims depend on measurement assumptions: consistent grading, no test-set leakage, and equivalence between vendor UI conditions. The publication or release cutoff choices in the figures are arbitrary but not central to the main result.

free parameters (2)
  • Recency cutoff for Fig. 1 = four months
    Models are split into 'within last four months' versus 'more than four months' by release date; the cutoff is chosen by hand and shapes the main trend plot.
  • Group A/B cutoff for Table 5 = 09-2024
    Models are split at September 2024 to compute multilingual deltas; this arbitrary threshold is not justified by data.
assumptions (3)
  • domain assumption Instructor grading of LLM outputs is consistent and unbiased relative to student grading.
    All LLM responses were scored by the course instructor (Section 3.1), but no rubric, double-grading, or inter-rater check is reported; percentile comparisons depend on grading equivalence.
  • domain assumption LLMs did not encounter the exam during pretraining.
    The GitHub dataset was released in 2024; if recent models memorized or saw the exam, their high scores would not reflect reasoning. No contamination check is reported.
  • domain assumption User-interface runs with default settings are comparable across models and versions.
    Models were accessed via ChatGPT, AI Studio, and Together AI with unspecified internal parameters; vendor UIs update models over time, threatening exact reproducibility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Struggle (06-2024) to Mastery (02-2025) LLMs Conquer Advanced Algorithm Exams and Pave the Way for Editorial Generation." pith.science (2026). https://pith.science/paper/4SKBEEKA

@misc{pith2026250604965,
  author       = {Pith},
  title        = {Pith review of: From Struggle (06-2024) to Mastery (02-2025) LLMs Conquer Advanced Algorithm Exams and Pave the Way for Editorial Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SKBEEKA}},
  note         = {Machine review of arXiv:2506.04965}
}
read the original abstract

This paper presents a comprehensive evaluation of the performance of state-of-the-art Large Language Models (LLMs) on challenging university-level algorithms exams. By testing multiple models on both a Romanian exam and its high-quality English translation, we analyze LLMs' problem-solving capabilities, consistency, and multilingual performance. Our empirical study reveals that the most recent models not only achieve scores comparable to top-performing students but also demonstrate robust reasoning skills on complex, multi-step algorithmic challenges, even though difficulties remain with graph-based tasks. Building on these findings, we explore the potential of LLMs to support educational environments through the generation of high-quality editorial content, offering instructors a powerful tool to enhance student feedback. The insights and best practices discussed herein pave the way for further integration of generative AI in advanced algorithm education.

Figures

Figures reproduced from arXiv: 2506.04965 by the authors.

Figure 1
Figure 1. LLM Performance on Advanced Algorithms Exam vs. Release Date. Models are grouped by release date: more than four months ago (red line, crosses) and within the last four months (blue line, circles). It is also important to mention that neither DeepSeekR1 nor the new Qwen 2.5 Pro perform well; both fail the exam and rank in the bottom half. Addi￾tionally, high server load hindered DeepSeekR1’s usability via both the U… view at source ↗
Figure 2
Figure 2. Distribution of student grades with LLM scores highlighted. From the figure, we observe that o3-mini and Gemini2.0 Flash perform well, achieving scores that place them among the top percentiles of student performance. In contrast, models such as Mixtral-8x22B and GPT-4 Legacy rank lower. Top Half Bottom Half Model Grade Percentile Model Grade Percentile o3-mini 86.60 98 4o 49.50 40 o3-mini-high 84.00 97 DeepSeekR1 4… view at source ↗
Figure 3
Figure 3. Examples of visual tasks from Problems 1 and 3 that challenge LLMs. – o3-mini offers a valid solution: • Red: {3, 8} • Blue: {1, 4, 6, 7} Problem 3 poses a similar challenge by asking for the minimal number of edges that must be removed from a given graph to allow for an Eulerian path. In this case, some of the newer thinking models—such as DeepSeek and Gemini2.0 Flash Experimental Thinking attempt backtracking but … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Design of our application. The application aims at being a tool tailored to professors and students work speeding up their editorial / grading scheme creation, respectively their learning experience, and is based on the observation that modern LLMs are sufficiently goo…
Figure 5
Figure 5. Figure 5: Framework for generating solutions to a problem. nisms. Exposing an unprotected application querying LLMs to the internet can incur unexpected costs due to an abusive usage of our API: attackers can use techniques such as [12] for bypassing our prompts and run arbitrar…
Figure 6
Figure 6. Figure 6: Graph representa￾tions for Exercises 1–6. For the graph given below, solve tasks 1–6 and justify your answers; the neighbors of a vertex are considered in lexicographical order. 1. (5p) Specify a connected bipartite induced sub￾graph with the maximum number of nodes an…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

  1. [1]

    In: Proceedings of the Seventh AAAI/ACM Conference on AI, Ethics, and Society (AIES ’24)

    Anton, A., et al.: Examining the behavior of LLM architectures within the frame- work of standardized national exams in brazil. In: Proceedings of the Seventh AAAI/ACM Conference on AI, Ethics, and Society (AIES ’24). ACM/AAAI (2024). https://doi.org/10.48550/arXiv.2408.05035, arXiv:2408.05035 [cs.CL]

    arXiv doi:10.48550/arxiv.2408.05035 2024

  2. [2]

    Also available athttps://lmarena.ai/?leaderboard

    Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., Stoica, I.: Chatbot arena: An open platform for evaluating LLMs by human preference (2024), arXiv:2403.04132 [cs.CL]. Also available athttps://lmarena.ai/?leaderboard

    arXiv 2024

  3. [3]

    Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysis

    Dumitran, A.M., Badea, A.C., Muscalu, S.G.: Evaluating the performance of large language models in competitive programming: A multi-year, multi-grade analysis (2024). https://doi.org/10.48550/arXiv.2409.09054, arXiv:2409.09054 [cs.AI]

    work page Pith review arXiv doi:10.48550/arxiv.2409.09054 2024

  4. [4]

    https://github.com/marius135/ExameneAF2025 (2024), gitHub repos- itory

    Dumitran, A.M., Moroianu, T.P., Alexe, V.P.: Advanced algorithms exam dataset (06-2024). https://github.com/marius135/ExameneAF2025 (2024), gitHub repos- itory. Last accessed 15 Feb 2025

    work page 2024

  5. [5]

    https://github.com/marius135/ExameneAF2025 (2025), accessed: 2025- 04-22

    Dumitran, A.M., Moroianu, T.P., Alexe, V.P.: Comprehensive grading scheme dataset. https://github.com/marius135/ExameneAF2025 (2025), accessed: 2025- 04-22. Title Suppressed Due to Excessive Length 15

    work page 2025

  6. [6]

    https://doi.org/10.48550/ arXiv.2502.06807, arXiv:2502.06807 [cs.LG]

    El-Kishky, A., Wei, A., Saraiva, A., Minaev, B., Selsam, D., Dohan, D., Song, F., Lightman, H., Clavera, I., Pachocki, J., Tworek, J., Kuhn, L., Kaiser, L., Chen, M., Schwarzer, M., Rohaninejad, M., McAleese, N., et al.: Competitive programming with large reasoning models (2025). https://doi.org/10.48550/ arXiv.2502.06807, arXiv:2502.06807 [cs.LG]

    arXiv doi:10.48550/arxiv.2502.06807 2025

  7. [7]

    Automatic Feedback Generation for Short Answer Questions using Answer Diagnostic Graphs

    Furuhashi, M., Funayama, H., Iwase, Y., Matsubayashi, Y., Isobe, Y., Naga- hama, T., Sugawara, S., Inui, K.: Automatic feedback generation for short answer questions using answer diagnostic graphs. In: Proceedings of the 16th Interna- tional Conference on Education and New Learning Technologies (EDULEARN25) (2025). https://doi.org/10.48550/arXiv.2501.1577...

    work page Pith review arXiv doi:10.48550/arxiv.2501.15777 2025

  8. [8]

    https://aistudio.google.com/ (2024), last accessed 15 Feb 2025

    Google: Google AI studio. https://aistudio.google.com/ (2024), last accessed 15 Feb 2025

    work page 2024

Show all 20 references
  1. [9]

    Learning and Individual Differences 103, 102274 (2023)

    Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., ...

    2023

  2. [10]

    Computers and Education: Artificial Intelligence 6, 100213 (2024).https://doi.org/10.1016/j.caeai.2024.100213

    Lee, G.G., Latif, E., Wu, X., Liu, N., Zhai, X.: Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence 6, 100213 (2024).https://doi.org/10.1016/j.caeai.2024.100213

    2024

  3. [11]

    In: Findings of the Association for Compu- tational Linguistics: EMNLP 2024

    Lee, S., Cai, Y., Meng, D., Wang, Z., Wu, Y.: Unleashing large language models’ proficiency in zero-shot essay scoring. In: Findings of the Association for Compu- tational Linguistics: EMNLP 2024. pp. 181–198. Association for Computational Linguistics (2024)

    2024

  4. [12]

    Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X.: Prompt injection attack against LLM-integrated applications (2023), arXiv:2306.05499 [cs.CR]

    2023 arXiv

  5. [13]

    ETS Research Report Series 2017(1), 1–26 (2017)

    Madnani, N., Loukina, A., Burstein, J.: Automated essay scoring with e-rater V.2. ETS Research Report Series 2017(1), 1–26 (2017). https://doi.org/10.1002/ ets2.12130

    2017

  6. [14]

    https://llm.moroianu

    Moroianu, T.P.: LLM editorial generation framework. https://llm.moroianu. work/generate-editorials (2025), last accessed Feb 2025

    2025

  7. [15]

    https://chatgpt.com/ (2023), last accessed 15 Feb 2025

    OpenAI: Chatgpt. https://chatgpt.com/ (2023), last accessed 15 Feb 2025

    2023

  8. [16]

    Phi Delta Kappan 47(5), 238–243 (1966)

    Page, E.B.: The imminence of grading essays by computer. Phi Delta Kappan 47(5), 238–243 (1966)

    1966

  9. [17]

    Together AI: The AI acceleration cloud.https://www.together.ai/ (2024), last accessed 15 Feb 2025

    2024

  10. [18]

    In: Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020)

    Wang, W., Chen, Z., Yan, G.: Transformer-based automated essay scoring with discourse-aware attention. In: Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020). pp. 6276–6287. International Com- mittee on Computational Linguistics, Barcel...

    2020

  11. [19]

    White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt (2023), arXiv:2302.11382 [cs.SE]

    2023 arXiv

  12. [20]

    https://doi.org/10

    Xie, W., Niu, J., Xue, C.J., Guan, N.: Grade like a human: Rethinking au- tomated assessment with large language models (2024). https://doi.org/10. 48550/arXiv.2405.19694, arXiv:2405.19694 [cs.HC]

    2024 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.