{"id":"35bdc164-9f80-46b8-8082-15b60c51cf65","arxiv_id":"2304.06364","paper_version":2,"verdict":"ACCEPT","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AGIEval shows GPT-4 exceeding average human scores on SAT Math at 95% and Chinese college entrance English at 92.5%, while revealing weaker results on complex reasoning tasks.","lead":"The paper introduces AGIEval, a benchmark that tests foundation models on real standardized human exams such as SAT, LSAT, math competitions, and lawyer qualification tests. A smart generalist should read it to understand how current AI models compare to human performance on tasks that require integrated knowledge, reasoning, and decision-making.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Data leakage from widely available exam questions may explain GPT-4's high scores rather than genuine generalization.","rationale":"The reader's weakest assumption (exams as unbiased proxies) is adjacent but broader; the concrete load-bearing risk is contamination, which directly threatens whether the reported accuracies support the surpassing claim. This is a standard technical concern for any benchmark built from public human exams and is not addressed in the abstract or reader's summary.","tokens_in":1784,"tokens_out":346,"duration_ms":34400,"concrete_test":"Sample 30 questions each from the SAT-Math and Gaokao-English subsets; run exact-string and normalized-edit-distance search against Common Crawl snapshots before GPT-4's training cutoff and against known exam-answer sites; if >15% of items match with high similarity, recompute accuracies after replacing those items with freshly authored isomorphic questions and re-prompt the model.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (GPT-4 at 95% SAT Math and 92.5% Gaokao English, surpassing human averages) requires that model outputs reflect reasoning on novel items. AGIEval sources questions from public standardized exams (SAT, LSAT, math contests, Gaokao) whose full texts and solutions have circulated online for years. Foundation models trained on web-scale corpora can therefore encounter exact or near-exact matches. The paper reports capability breakdowns (understanding/knowledge/reasoning/calculation) but provides no membership checks, decontamination steps, or comparison against paraphrased variants. If a non-trivial fraction of items were seen in training, the headline performance numbers no longer demonstrate the claimed cognitive advance over humans.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces AGIEval, a benchmark assembled from publicly available human standardized exams (SAT, LSAT, math competitions, Gaokao, lawyer qualification tests). It evaluates GPT-4, ChatGPT, and Text-Davinci-003, reporting that GPT-4 exceeds average human performance on several tests (95% on SAT Math, 92.5% on Gaokao English) while showing weaker results on complex reasoning and domain-knowledge tasks. Capability breakdowns (understanding/knowledge/reasoning/calculation) and full data/code/output release are provided.","tokens_in":1910,"tokens_out":434,"duration_ms":26660,"significance":"If the headline numbers survive decontamination checks, the work supplies a more ecologically valid signal of foundation-model progress than synthetic benchmarks and supplies concrete capability diagnostics plus reproducible artifacts. The public release of all model outputs strengthens the contribution.","major_comments":[{"comment":"Abstract and evaluation section: the 95% SAT-Math and 92.5% Gaokao-English figures are presented without the number of items per test, sampling protocol, exact prompt templates, or any statistical testing; these omissions leave the central claim that GPT-4 surpasses humans only moderately supported.","section":"Abstract"},{"comment":"Evaluation methodology: no membership-inference, decontamination, or paraphrased-variant experiments are reported for the publicly circulated exam questions, even though the central claim (surpassing humans via reasoning) requires that performance not be explained by training-data overlap.","section":"Evaluation"}],"minor_comments":[{"comment":"Figure captions and axis labels could more explicitly state the human baseline source and sample size for each exam.","section":"Figures"},{"comment":"A short table summarizing prompt templates per task type would improve reproducibility.","section":"Appendix"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the only load-bearing issue; once addressed the paper would be a straightforward accept for a venue focused on evaluation benchmarks."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our AGIEval benchmark paper. The comments highlight valuable opportunities to strengthen the presentation of results and the evaluation methodology. We have revised the manuscript to incorporate additional details and experiments where feasible, and we respond to each major comment below.","responses":[{"response":"We agree that these supporting details are necessary to substantiate the central claims. In the revised manuscript, we have expanded the evaluation section to report the exact number of items per test (SAT Math: 58 questions; Gaokao English: 40 questions), clarified that evaluations used the full publicly available test sets with no subsampling, included the precise prompt templates in a new appendix, and added statistical testing via binomial proportion tests to confirm that GPT-4's accuracies significantly exceed the reported human averages. These changes provide stronger empirical grounding for the headline figures.","revision_made":"yes","referee_comment":"[Abstract] Abstract and evaluation section: the 95% SAT-Math and 92.5% Gaokao-English figures are presented without the number of items per test, sampling protocol, exact prompt templates, or any statistical testing; these omissions leave the central claim that GPT-4 surpasses humans only moderately supported."},{"response":"We acknowledge the importance of ruling out data contamination to support interpretations of reasoning ability. While full membership-inference or decontamination experiments are not feasible without access to the proprietary training data of the evaluated models, we have added paraphrased-variant experiments on subsets of the SAT and Gaokao questions in the revision; these maintain high performance, indicating robustness beyond exact memorization. We have also expanded the limitations and discussion sections to address contamination risks explicitly, noting the public nature of the exams and known training cutoffs, and we release all model outputs to support community-led analyses.","revision_made":"partial","referee_comment":"[Evaluation] Evaluation methodology: no membership-inference, decontamination, or paraphrased-variant experiments are reported for the publicly circulated exam questions, even though the central claim (surpassing humans via reasoning) requires that performance not be explained by training-data overlap."}],"tokens_in":1395,"tokens_out":460,"duration_ms":29163,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"AGIEval takes existing public exams like the SAT, LSAT, Gaokao, and math contests and turns them into a single evaluation set for foundation models. That is the main new piece: a human-centric yardstick instead of another synthetic task suite. The paper runs GPT-4, ChatGPT, and Text-Davinci-003 across these tests, reports breakdowns by capability (understanding, knowledge, reasoning, calculation), and releases the full questions, code, and every model output on GitHub. That release is the part that actually helps the field; anyone can rerun or extend the numbers without starting from scratch. The headline results show GPT-4 clearing average human scores on several exams, which lines up with the abstract's claim. The capability analysis is straightforward and points to where the models still lag on complex reasoning or narrow domain knowledge. The main soft spot is exactly the one the stress-test note flags. These exams and their solutions have circulated online for years, so web-scale training data almost certainly contains exact or near-exact matches. The paper gives no membership checks, no decontamination steps, and no comparison to paraphrased versions. Without those, the 95% SAT Math and 92.5% Gaokao English numbers cannot be read as clean evidence of reasoning gains. Prompting details and significance testing are also light in the summary. This paper is for groups that build or evaluate large models and want a ready-made set of real-world tasks. A reader who needs a practical benchmark to track progress will get immediate use from the released artifacts. The work is coherent on its own terms and shows honest engagement with the evaluation problem, even if the central performance claims require extra verification. I would send it to peer review; the benchmark itself is worth referee attention and the leakage issue is fixable with targeted experiments.","headline":"AGIEval assembles a useful benchmark from real standardized exams and releases the data, but GPT-4's reported scores need checks for training-data overlap before the generalization claims hold.","tokens_in":2405,"tokens_out":448,"would_cite":true,"duration_ms":25674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"Cost.FunctionalEquation","rs_theorem":null,"paper_passage":"GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam."},{"relation":"unclear","rs_module":"Foundation.HierarchyEmergence","rs_theorem":"hierarchy_emergence_forces_phi","paper_passage":"We introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams"}],"headline":"AGIEval LLM benchmark on human exams shows no overlap with RS cost-based derivation of constants from distinction","alignment":"orthogonal","rationale":"The paper introduces a benchmark using standardized exams (SAT, LSAT, Gaokao) to evaluate foundation models like GPT-4, claiming high accuracy (e.g., 95% on SAT Math). This is an empirical AI evaluation task focused on model performance on public test data. RS framework derives J-cost, φ, 8-tick periodicity, D=3, and constants c/ℏ/G from a single distinction via machine-checked Lean theorems (e.g., reality_from_one_distinction, washburn_uniqueness_aczel, hierarchy_emergence_forces_phi). No shared machinery: paper has no cost functional, no golden-ratio identities, no ledger forcing, no d'Alembert/RCL structure. Domain mismatch (AI benchmark vs. zero-parameter physics derivation) makes it orthogonal; skeptic note on data leakage is irrelevant to RS.","tokens_in":279790,"confidence":"high","tokens_out":381,"duration_ms":12828,"cache_read_input_tokens":260160,"cache_creation_input_tokens":0},"lean_confirmation":{"model":"grok-4.3","status":"out_of_scope","citations":[],"rationale":"This is an empirical AI evaluation paper, not one resting on a provable mathematical premise. The benchmark and results are data-driven, falling under out_of_scope for Lean confirmation.","tokens_in":279525,"confidence":"moderate","tokens_out":144,"duration_ms":28491,"inferential_bridge":"The paper presents empirical results from evaluating models on standardized tests; no mathematical or structural theorem is load-bearing. Lean cannot prove empirical performance claims.","load_bearing_premise":"The paper's central claim is that GPT-4 achieves specific accuracy rates on human exams that surpass average human performance, relying on the empirical validity of the AGIEval benchmark as a proxy for human cognitive capabilities.","cache_read_input_tokens":64,"cache_creation_input_tokens":0},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AGIEval benchmark shows GPT-4 surpassing average humans on SAT math at 95 percent and LSAT.","keywords":["AGIEval","foundation models","benchmark","GPT-4","standardized exams","SAT","LSAT","reasoning"],"falsifier":"A controlled comparison in which models achieve high AGIEval scores yet fail on equivalent non-exam problems that test the same underlying skills in open-ended or novel settings.","tokens_in":2689,"feed_emoji":"📊","tokens_out":653,"duration_ms":38365,"temperature":0.7,"pith_summary":"This paper introduces AGIEval, a benchmark that draws questions directly from standardized human exams such as college entrance tests, law school admissions, math competitions, and lawyer qualification exams to measure foundation model abilities on human-level tasks. Evaluations of GPT-4, ChatGPT, and Text-Davinci-003 find that GPT-4 exceeds average human scores on SAT, LSAT, and math competitions, reaching 95 percent accuracy on SAT math and 92.5 percent on the English section of the Chinese national college entrance exam. The models show weaker results on problems that require complex reasoning chains or narrow domain knowledge. The authors break performance into categories of understanding, knowledge, reasoning, and calculation to expose specific strengths and gaps. The approach replaces artificial datasets with tasks tied to real human cognition and decision-making.","feed_headline":"GPT-4 beats human average on SAT math at 95%","feed_subtitle":"Benchmark drawn from real exams finds foundation models outperforming people on tests but lagging on complex reasoning.","key_machinery":"The AGIEval benchmark, assembled from standardized human exams to test foundation models on understanding, knowledge, reasoning, and calculation in human-relevant contexts.","core_discovery":"AGIEval evaluates foundation models on collections of real standardized exams including SAT, LSAT, math competitions, and lawyer qualification tests. GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining 95 percent accuracy on the SAT Math test and 92.5 percent accuracy on the English test of the Chinese national college entrance exam, while remaining less proficient on tasks that demand complex reasoning or specific domain knowledge.","pith_inferences":["Sustained high scores could support deployment of models as automated tutors or graders for these exact exams.","Gaps in complex reasoning may require architectural additions rather than further scaling alone.","Extending the benchmark with harder or culturally varied exam variants could track whether gains generalize."],"forward_implications":["Foundation models can now solve many exam-style questions at or above average human levels across multiple subjects.","Performance gaps appear most clearly in complex reasoning and domain-specific knowledge, guiding targeted improvements.","Capability breakdowns by category supply concrete directions for strengthening general abilities.","Human-exam benchmarks connect model results more directly to real-world cognitive demands than synthetic tests do."],"fun_headline_variants":["GPT-4 scores 95% on SAT math in AGIEval benchmark","AGIEval benchmark evaluates models on SAT LSAT math competitions","GPT-4 scores 95% SAT math and 92.5% on Chinese exam via AGIEval"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Standardized human exams serve as valid and unbiased proxies for general cognitive capabilities without favoring current model training methods or test formats.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 scores 95% on SAT math in AGIEval benchmark","AGIEval benchmark evaluates models on SAT LSAT math competitions","GPT-4 scores 95% SAT math and 92.5% on Chinese exam via AGIEval"]},"model":"grok-4.3","cost_usd":0.008735,"raw_usage":{"total_tokens":3970,"prompt_tokens":737,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":87349500,"prompt_tokens_details":{"text_tokens":737,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3166,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":737,"tokens_out":67,"duration_ms":18351,"temperature":1.0,"reasoning_tokens":3166,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-16T09:59:54.385025+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled comparison in which models achieve high AGIEval scores yet fail on equivalent non-exam problems that test the same underlying skills in open-ended or novel settings.","supporting_citations":[],"review_version":1}