Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

The paper claims that automatically generated coding tasks, with test suites built without ground-truth solutions, expose a large and measurable reliance of LLMs on memorization rather than reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

UniCode's LLM-generated coding benchmark drops top-model pass@1 to 70.3% and indicates current LLMs rely on memorized seed logic instead of generalizing to new algorithmic problems.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection UniCode's no-reference test-suite pipeline is a real contribution, but the paper's central diagnostic claim about memorization and scalability reasoning outruns the evidence. the 4 major comments →

arxiv 2510.17868 v3 pith:P4YHE2OV submitted 2025-10-16 cs.SE

UniCode: Augmenting Evaluation for Code Reasoning

Xinyue Zheng , Haowei Lin , Shaofei Cai , Yaodong Yang , Zilong Zheng , Yitao Liang This is my paper
classification cs.SE
keywords generative evaluationcode generationLLM evaluationcompetitive programmingtest case synthesisdata contaminationbenchmark generationmemorization vs reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a fully generative coding benchmark can unmask weaknesses in large language models that static, human-written benchmarks miss. To do this it builds UniCode, a framework that turns existing competition problems into novel ones by extending, fusing, and cross-breeding them, and attaches test suites whose expected outputs are produced without any canonical solution—using brute-force checks on small inputs, majority voting among filtered LLM-generated solvers on large inputs, and LLM adjudication for ties. On a curated set of 492 problems and 19 models, pass@1 scores collapse by over 30% compared with trivial paraphrases of the same seeds, and the best model solves only 70.3% of problems. The paper argues this collapse reflects genuine conceptual and scalability reasoning failures rather than syntax errors, and that models revert to memorized seed logic. A reader should care because if the pipeline's labels are sound, dynamic benchmark generation becomes a practical way to keep evaluation ahead of contamination.

Core claim

UniCode's central claim is that generative evaluation can replace static, contamination-prone benchmarks. The framework mutates and recombines seed problems via three strategies, then builds test suites without a canonical solution: brute-force solvers vet small inputs, filtered LLM solvers vote on large inputs, and an LLM adjudicates ties. On the resulting 492 tasks, pass@1 drops by over 30% compared with trivial paraphrases of the seeds, with the strongest model at 70.3%; the decline sharpens on hard problems and traces to conceptual and scalability failures, not syntax. The paper reads this as models defaulting to memorized seed logic.

What carries the argument

The stress-driven test-suite synthesis pipeline is the load-bearing mechanism. It generates trusted outputs for small inputs with a brute-force solver, uses those outputs to filter a pool of LLM-generated optimized solutions, runs the surviving solvers on large inputs, and lets a strong LLM break ties. This pipeline produces ground truth for novel problems without a reference solution; the three mutation strategies (single-problem extension, same-type fusion, cross-type fusion) supply the problem novelty that makes the tests a true reasoning probe.

Load-bearing premise

The load-bearing premise is that LLM-generated optimized solvers that agree after small-scale brute-force filtering also produce correct outputs on large inputs; if they share a systematic blind spot, the test labels—and every score and ranking in the paper—could be wrong.

What would settle it

Hand-write or otherwise independently obtain exact outputs for a sample of the 492 problems' large-scale test inputs (the ones beyond brute force), compare them to the pipeline's majority-vote/adjudicated labels, and count disagreements. Even a 5% disagreement rate would meaningfully undo the reported pass rates; near-perfect agreement would confirm the pipeline.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A trustworthy generative pipeline removes the human bottleneck: benchmarks can be refreshed on demand, staying ahead of data contamination.
  • The large performance gap on generated problems over paraphrased ones indicates that current coding models partly succeed by retrieval; novel composite problems are the diagnostic that exposes this.
  • Adversarial test cases are systematically harder than random ones, an average 11 percentage-point drop, so the framework can measure robustness to edge cases, not just average correctness.
  • Pass@k results rise from 66.1% to 83.5% at k=10 yet plateau, showing the benchmark remains hard even with repeated sampling; gains concentrate in solution-diverse problem types.
  • The pipeline deliberately discards any problem every baseline solves, so the benchmark is engineered to sit at the frontier of current capability rather than to estimate absolute accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the large-scale ground-truth labels are wrong in a way shared by the LLM solver pool, the reported rankings and collapse percentages could be an artifact of the labeling pipeline; running independent exact solvers on a sample of large inputs would test this directly.
  • Because generation, filtering, and adjudication all lean on a single closed model family, the benchmark may be inadvertently biased toward that family's solution conventions; generating with heterogeneous models would make the benchmark's conclusions more robust.
  • The framework's difficulty filter (drop problems every baseline solves) means UniCode measures headroom relative to the current model generation; as models improve, the benchmark's absolute pass rates should rise while its discriminative power may fade, necessitating re-generation.
  • Per-tag pass@k patterns suggest that gains from sampling concentrate in tasks with multiple valid solution paths (graph, tree, math), so UniCode could double as a tool for studying solution-space diversity, not just correctness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces UniCode, a generative framework for creating novel competitive-programming problems by extending, fusing same-type, or fusing cross-type seed problems, together with a stress-driven pipeline for automatically generating test suites without a canonical reference solution. The pipeline uses brute-force grounding on small-scale inputs to filter LLM-generated optimized solvers, then uses strict majority voting over the filtered solvers for large-scale inputs, with LLM adjudication of ties. The authors curate 492 problems, report human-validated solvability of 98%, and evaluate 19 LLMs, finding that the best model, o4-mini-high, achieves 70.3% pass@1, that adversarial inputs hurt more than random inputs, and that performance drops by over 30% on UniCode problems relative to narrative paraphrases of seed problems. The framework is intended as a contamination-resistant, scalable alternative to static coding benchmarks.

Significance. If the test-suite labels are reliable at the claimed level, UniCode is a promising step toward generative, contamination-resistant code evaluation. The paper's strengths include a transparent three-stage test-generation design, an external validation on Test-Eval (94.5% correctness, 86% coverage), a blinded human study, planned public release of problems and provenance metadata, and a strong correlation with LiveCodeBench. However, the central quantitative claims—the 70.3% top-model pass rate, the >30% collapse, and the inference that models revert to memorized seed logic—all rest on the large-scale ground-truth labels produced by the LLM-consensus pipeline, which is not independently audited on UniCode's own problems. The paper's interpretive claim about 'conceptual modeling and scalability reasoning' is also not supported by any error analysis. These gaps are addressable and do not undermine the framework's basic idea.

major comments (4)
  1. [Section 2.2.2 (Stages 2–3), Table 2] The large-scale ground-truth labels for the 492 UniCode problems are produced by strict majority vote among LLM-generated optimized solvers that were pre-filtered only on small-scale brute-force inputs, with unresolved disagreements adjudicated by o4-mini. The Stage 1 filter does not verify correctness at large scale; if the solver pool shares a systematic blind spot, the large-scale labels are wrong. Section 3.3 validates the pipeline on Test-Eval, which has known reference solutions, but never on UniCode's own large-scale labels. Appendix B's total-error bound (Eq. 10) only applies once α and q_e are empirically measured, and no such audit is reported. Since Table 2 pass rates, the reported 31.2% collapse, and the 'revert to memorized seed logic' conclusion all depend on these labels, an independent audit is required: sample problems, obtain reference solutions from diverse non-LLM sou
  2. [Abstract; Section 4.2] The abstract attributes the observed performance collapse to 'deficiencies in conceptual modeling and scalability reasoning rather than syntactic errors,' but the paper reports no error analysis that distinguishes these categories. Pass@1, RandPass, and AdvPass (Table 2) measure functional correctness and input-type sensitivity; they do not reveal whether a failed submission is conceptually wrong, algorithmically unscalable, or syntactically buggy. This attribution is load-bearing for the paper's interpretation and must either be supported by a per-failure categorization or explicitly downgraded to a hypothesis.
  3. [Section 4.3, Figure 3] The seed-vs-shadow-vs-UniCode experiment is small (apparently 20 seed problems and 48 generated problems, with the text also mentioning 50; please resolve) and the conclusion that models 'revert to memorized seed logic' is overreach. The drop on UniCode problems is equally consistent with the generated problems being more algorithmically demanding (e.g., cross-type fusion produces genuinely harder tasks). A controlled comparison that holds algorithmic complexity fixed while varying narrative—exactly what the shadow set does—is needed to support the memory-reversion claim; the current design does not provide it.
  4. [Sections 2.1, 3.1, and 6] The same model family (o4-mini) is used to generate the problems (Section 3.1), to generate and adjudicate ground-truth outputs (Section 2.2.2), and is the top-scoring evaluated model (Table 2). The paper's Section 6 acknowledges the generation bias, but the evaluation section does not quantify how this affects rankings or the 'contamination-resistant' claim. Since o4-mini's high score may partly reflect distributional alignment with the generator's style, the headline 'only 70.3%' should be accompanied by a sensitivity analysis using problems generated by a different model, or by an explicit statement that the leaderboard is relative to the o4-mini-generated distribution.
minor comments (4)
  1. [Abstract vs. Introduction] The abstract says 'o4-mini achieves a pass rate of only 70.3%' while the introduction says 'o4-mini achieves a pass@1 rate of only 66.1%.' Clarify that 70.3% refers to o4-mini-high and 66.1% to o4-mini-medium, and use consistent naming throughout.
  2. [Section 4.3] The text refers to '50 problems in LiveCodeBench v1,' then says 'Seed questions: the original 20 problems,' and later 'Each dataset contains 48 problems.' These numbers are inconsistent; please provide the exact construction and dataset sizes.
  3. [Table 2] The relationship between Overall Pass@1, RandPass, and AdvPass is not defined. A problem is solved only if all 50 test cases pass, so RandPass/AdvPass appear to be per-case pass rates on subsets, but this should be stated explicitly, and the gap Δ(A−R) should be defined accordingly.
  4. [Appendix B / Section 3.3] The statement in Section 3.3 that the mathematical proof in Appendix B demonstrates the benchmark's accuracy is 'trustworthy' is too strong. Equation (10) requires empirical estimates of α and q_e; without an audit these quantities are unknown. Either add the audit or soften the wording to say the bound applies once those parameters are measured.

Circularity Check

2 steps flagged

UniCode's headline difficulty and top-model claims are entangled with its own curation filter and with o4-mini serving as both ground-truth adjudicator and evaluated model.

specific steps
  1. fitted input called prediction [Section 3.1 (Benchmark Curation); Section 4.3 (Evaluating Generalization in LLMs)]
    "To ensure the problems are challenging, we filter out any that can be solved perfectly by all our baseline models. Concretely, we run a panel of baselines as in Table 2 if every model achieves a 100% pass rate on the test suite, the problem is removed. ... All LLMs exhibited a significant performance drop on UniCode questions."

    The reported 31.2% performance collapse is measured on the filtered UniCode problem set, while the ShadowQS control is not filtered in the same way. Because the inclusion rule removes every problem that all baseline models solve perfectly, the retained benchmark is guaranteed to contain at least one model failure, so part of the measured 'collapse' is an artifact of selection rather than an independent measure of novel-reasoning failure.

  2. other [Section 2.2.2, Stage 3; Section 3.1; Table 2]
    "In this case, we identify the top two most frequent outputs, o1 and o2, and present them to a powerful LLM (e.g., o4-mini). The LLM is tasked with analyzing the problem statement, the input, and the conflicting candidate solutions to determine which output is correct. If the LLM provides a decisive judgment, its chosen output is accepted as the ground truth."

    o4-mini-medium generates the problems (Section 3.1), and o4-mini also adjudicates unresolved test-output disagreements, thereby helping define the answer key. The same model family is then evaluated and reported as the top performer (70.3% pass@1). Thus the headline ranking partly measures how well o4-mini agrees with labels that o4-mini itself helped produce. The Test-Eval validation (Section 3.3) provides indirect evidence for the pipeline but does not independently audit the 492 large-scale labels.

full rationale

The derivation chain has two real but partial circularities. First, the benchmark's difficulty is curated by deleting every problem that all baseline models solve perfectly, so the later claim that models 'collapse' on UniCode is partly constructed by the selection rule; the shadow-question control is not filtered, so it is not a clean controlled comparison for isolating novelty-driven failure. Second, o4-mini acts as problem generator, as adjudicator of unresolved ground-truth disagreements, and as the evaluated top model, so the reported rankings are not fully independent of the answer key. The paper partially mitigates both concerns by validating the pipeline on Test-Eval (94.5% correctness), by blind human solvability review, and by explicitly acknowledging the single-generator limitation in the conclusion. Still, the specific 492-problem labels are never independently audited, and the central 31.2% collapse claim is entangled with the curation filter and with o4-mini's role in defining ground truth. This is moderate, partial circularity rather than a fully self-referential derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

Central claims depend on hand-set thresholds (k=3, 20/20/10 test composition, strict majority rule), unspecified solver-pool sizes, and a difficulty filter that removes easy problems by construction. These are not fitted to external data and no sensitivity analysis is provided. The framework introduces no new physical or formal entities; the 'three strategies' are methodological categories rather than entities requiring independent evidence.

free parameters (5)
  • safety factor k = 3
    Used in TL = k * min(T(o)) and ML = k * Mem(o*) in Section 3.1; hand-set and directly controls which optimized solutions pass time/memory limits, affecting every pass rate.
  • test suite composition = 20 random / 20 adversarial / 10 LLM-synthesized
    Fixed composition in Section 2.2.1; hand-chosen balance affects coverage and difficulty of all 492 problem suites.
  • strict majority threshold = > N/2
    Consensus rule in Section 2.2.2 Stage 2; determines when LLM adjudication is required and which outputs become ground truth.
  • number of candidate solvers M and trusted pool size N = not specified
    The pipeline requires M generated optimized solutions and N filtered solvers, but the paper never states M or observed N; consensus reliability depends on these unspecified quantities.
  • problem difficulty filter = remove if all baseline models pass 100%
    Section 3.1 removes any problem solved perfectly by all baseline models; benchmark difficulty and the magnitude of the reported collapse are therefore partly determined by construction.
axioms (6)
  • domain assumption LLM-generated brute-force solvers are correct on small-scale inputs (with multi-candidate consensus).
    Section 2.2.2 Stage 1 and footnote 1; if the brute-force solvers are wrong, the stress filter admits bad optimized solutions into the trusted pool.
  • domain assumption Majority vote among filtered LLM-generated optimized solutions gives correct outputs on large-scale inputs.
    Section 2.2.2 Stage 2; all solvers may share systematic biases because they are generated by the same type of LLM.
  • domain assumption o4-mini adjudication correctly resolves output disagreements.
    Section 2.2.2 Stage 3; the same model family is both adjudicator and top evaluated model, creating a potential same-source bias.
  • domain assumption Validation results on Test-Eval (80 problems with known solutions) transfer to UniCode's 492 novel problems.
    Section 3.3; Table 1 measures pipeline quality on Test-Eval, not on UniCode's own generated problems.
  • domain assumption The contamination model in Appendix B applies to UniCode with small alpha.
    Appendix B bounds bias by alpha, but no alpha for UniCode is measured; the numerical illustration assumes alpha = 0.06, not an estimate.
  • domain assumption Human solvability rating of 50 uniformly sampled problems generalizes to the 492-problem benchmark.
    Section 3.2; the 98% solvability and 92.3% agreement come from a 50-problem sample only.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of UniCode: Augmenting Evaluation for Code Reasoning." pith.science (2026). https://pith.science/paper/P4YHE2OV

@misc{pith2026251017868,
  author       = {Pith},
  title        = {Pith review of: UniCode: Augmenting Evaluation for Code Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4YHE2OV}},
  note         = {Machine review of arXiv:2510.17868}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Current coding benchmarks often inflate Large Language Model (LLM) capabilities due to static paradigms and data contamination, enabling models to exploit statistical shortcuts rather than genuine reasoning. To address this, we introduce UniCode, a generative evaluation framework that systematically probes LLM limits via: (1) multi-dimensional augmentation transforming seed problems into complex variations to disrupt fixed algorithmic patterns; (2) a highly reliable, automated test generation pipeline for scalable evaluation; and (3) fine-grained metrics for rich error signals. Experiments reveal a 31.2% performance collapse in state-of-the-art models on UniCode, primarily driven by deficiencies in conceptual modeling and scalability reasoning rather than syntactic errors. Furthermore, we uncover a seed-problem regression where models revert to memorized seed logic rather than following new specifications, signaling a reliance on shortcuts over reasoning. This work validates UniCode as a robust framework to expose model fragility and foster reasoning-oriented code intelligence.

Figures

Figures reproduced from arXiv: 2510.17868 by Haowei Lin, Shaofei Cai, Xinyue Zheng, Yaodong Yang, Yitao Liang, Zilong Zheng.

Figure 1
Figure 1. Figure 1: An illustration of our three problem generation strategies using seed problems. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stress-driven pipeline for ground-truth generation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison across question sets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparative performance across problem types for closed-source (left) and open-source (right) mod [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Unicode leaderboard. Pass@1 score vs. average cost per problem across various models. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-tag improvement from pass@1 to pass@3. Tags are grouped by improvement magnitude to illustrate which problem classes benefit most from candidate diversification. 0.2 0.3 0.4 0.5 0.6 0.7 Ours Pass@1 0.3 0.4 0.5 0.6 0.7 0.8 Livecodebench Pass@1 Pearson r=0.986, p=6.5e-06, n=8 0.2 0.3 0.4 0.5 0.6 0.7 Ours Pass@1 5 10 15 20 25 Livecodebenchpro Rank (1 = best) Pearson r=-0.916, p=5.2e-04, n=9 0.3 0.4 0.5 0.… view at source ↗
Figure 7
Figure 7. Figure 7: The alignment between UniCode and established benchmarks (LiveCodeBench and LiveCodeBench [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of tags, sub-tags, and skills in UniCode dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Human rating website. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Combinatorial Synthesis: Scaling Code RLVR via Atomic Decomposition and Recombination

    cs.CL 2026-05 unverdicted novelty 6.0

    ADR generates novel verifiable code tasks via atomic decomposition and recombination, outperforming heuristic baselines in originality, difficulty, and downstream RLVR gains across coding domains.

Reference graph

Works this paper leans on

21 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [10]

    Parse the c o n s t r a i n t s on the input from the problem description , such as the range of input data , specific input constraints , etc

  2. [11]

    The function should 26 Preprint

    Write a function g e n e r a t e _ t e s t _ i n p u t using the CYaRon library to generate a single a d v e r s a r i a l test input designed to ch al le nge boundary c o n d i t i o n s or worst - case c o m p l e x i t y . The function should 26 Preprint. i n t e r n a l l y r an do mi ze which a d v e r s a r i a l strategy to use , without a cce pt i...

  3. [12]

    e q u a l _ w e i g h t s

    Write a function v a l i d a t e _ t e s t _ i n p u t to verify whether the g ene ra te d test input sa ti sfi es the r e q u i r e m e n t s s pec if ie d in the problem d e s c r i p t i o n . This includes checking the input data type and c o n s t r a i n t s parsed in step 1. The function should take i n p u t _ s t r i n g as input and return a boo...

  4. [19]

    Analyze its problem design a p p r o a c h e s

  5. [20]

    Create a new va ri at ion question based on the original one 6 7Output format ( strictly follow ) : 8## Part 1: Original Problems and Solution Analysis 9Step1 : [ Describe the steps of re as on in g ] 10Step2 : xxx 11... 12 13## Part 2: New Problem D e s c r i p t i o n : 14N e w _ p r o b l e m : [ Describe the new problem clearly in natural language .] ...

  6. [31]

    Explore how to combine these concepts

  7. [42]

    Design a new c h a l l e n g i n g problem that i n t e g r a t e s them 5 6Output format ( strictly follow ) : 7## Part 1: Original Problems and Solution Analysis 8Step1 : [ Describe the steps of re as on in g ] 9Step2 : xxx 10... 11 12## Part 2: New Problem D e s c r i p t i o n : 13N e w _ p r o b l e m : [ Describe the new problem clearly in natural l...

  8. [52]

    The function should validate that the p a r a m e t e r s fall within the s pec if ie d c o n s t r a i n t s

    Write a function g e n e r a t e _ t e s t _ i n p u t using the CYaRon library to randomly generate test inputs based on a s pe ci fi ed problem size . The function should validate that the p a r a m e t e r s fall within the s pec if ie d c o n s t r a i n t s . If any p ar ame te r is out of range , the function should return None . If the p a r a m e ...

  9. [63]

    This includes checking the input data type and c o n s t r a i n t s parsed in step 1 , such as range and other c o n d i t i o n s

    Write a function v a l i d a t e _ t e s t _ i n p u t to verify whether the g ene ra te d test input sa ti sfi es the r e q u i r e m e n t s s pec if ie d in the problem d e s c r i p t i o n . This includes checking the input data type and c o n s t r a i n t s parsed in step 1 , such as range and other c o n d i t i o n s . The function should take i ...

  10. [301]

    28 Preprint

    select the 1 -3 most relevant skills from the given list , ranked by r ele va nc e . 28 Preprint

  11. [311]

    select the 1 -3 most relevant skills from the given list , ranked by r ele va nc e

  12. [322]

    The new problem must be rigorous and clearly stated , and include explicit input / output s p e c i f i c a t i o n s or c o n s t r a i n t s

  13. [323]

    Please design qu es ti on s that have one correct answer ; avoid'output one possible combination'that could have multiple valid answers

  14. [333]

    Listing 5: Prompts for Single-problem Extension 29

    Provide two example test cases to d e m o n s t r a t e the new problem . Listing 5: Prompts for Single-problem Extension 29

  15. [334]

    Listing 4: Prompts for Cross-type Fusion 1You are an expert c o m p e t i t i v e p r o g r a m m e r

    Provide two example test cases to d e m o n s t r a t e the new problem . Listing 4: Prompts for Cross-type Fusion 1You are an expert c o m p e t i t i v e p r o g r a m m e r . 2I'll provide you with one p r o g r a m m i n g problem , its solution , and the key concepts they test . 3You need to :

  16. [1999]

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models.arXiv preprint arXiv:2404.15522, 2024

    1 Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models.arXiv preprint arXiv:2404.15522, 2024. 1, 9 Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and...

    Pith/arXiv arXiv 2024

  17. [2011]

    The modern theory of biological evolution: an expanded syn- thesis.Naturwissenschaften, 91(6):255–276, 2004

    1 Ulrich Kutschera and Karl J Niklas. The modern theory of biological evolution: an expanded syn- thesis.Naturwissenschaften, 91(6):255–276, 2004. 1 Jiaqi Li, Xinyi Dong, Yang Liu, Zhizhuo Yang, Quansen Wang, Xiaobo Wang, Song-Chun Zhu, Zixia Jia, and Zilong Zheng. ReflectEvo: Improving meta introspection of small LLMs by learn- ing self-reflection. In Wa...

    Pith/arXiv arXiv 2004

  18. [2021]

    Key-point-driven data synthesis with its enhancement on mathematical reasoning

    1, 9 Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24176–24184, 2025. 9 Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, ...

    Pith/arXiv arXiv 2025

  19. [2023]

    Solvability,

    1, 9 Yaoming Zhu, Junxin Wang, Yiyang Li, Lin Qiu, ZongYu Wang, Jun Xu, Xuezhi Cao, Yuhuai Wei, Mingshi Wang, Xunliang Cai, et al. Oibench: Benchmarking strong reasoning models with olympiad in informatics.arXiv preprint arXiv:2506.10481, 2025. 1, 9 13 Preprint. A SUPPLEMENTARYEXPERIMENTALRESULTS A.1 PERFORMANCE AND COST This section analyzes the trade-of...

    Pith/arXiv arXiv 2025

  20. [2024]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    1, 9 Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 1, 9 Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecod...

    Pith/arXiv arXiv 2024

  21. [2025]

    5 Pierre Chambon, Baptiste Roziere, Benoit Sagot, and Gabriel Synnaeve

    URLhttps://arxiv.org/abs/2412.00535. 5 Pierre Chambon, Baptiste Roziere, Benoit Sagot, and Gabriel Synnaeve. Bigo (bench)–can llms generate code with controlled time and space complexity?, 2025.URL https://arxiv. org/abs/2503.15242. 9 Brian Charlesworth, Andrea J Betancourt, VB Kaiser, and Isabel Gordo. Genetic recombination and molecular evolution. InCol...

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.