{"id":"3e0a6db2-21df-4a5c-9dce-efb7964250a1","arxiv_id":"2606.07069","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"mmPISA-bench evaluates two LLMs on 25 reasoning questions across 43 languages and finds effective multilingual reasoning at human levels with machine translations performing equivalently to human ones.","lead":"The paper introduces mmPISA-bench, a benchmark of 25 PISA reasoning questions translated into 43 languages with both official and machine translations. It tests two LLMs and reports they achieve human-comparable accuracy across languages with no accuracy drop from machine translation.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The 25 PISA items may test training-data familiarity or language-specific phrasing rather than pure cross-lingual reasoning.","rationale":"The reader's weakest assumption is exactly the load-bearing point; the full manuscript does not appear to add the controls needed to secure it, so the abstract-derived verdict remains appropriate pending that check.","tokens_in":1673,"tokens_out":271,"duration_ms":12379,"concrete_test":"Take the 5 highest-accuracy items per language; generate 3 meaning-preserving paraphrases each in English and in 3 other languages; re-evaluate both LLMs on the paraphrased set using the same prompt template; if mean accuracy drops by >15 percentage points relative to the original items, the original results cannot be attributed to reasoning alone.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim requires that accuracy on these items reflects transferable reasoning preserved by both human and machine translations. With only 25 questions drawn from a public OECD item pool, no reported decontamination check, and no paraphrased controls, it remains possible that high accuracy (and the lack of degradation under MT) partly reflects memorization of surface forms or culturally loaded content rather than the intended reasoning steps. This directly undermines the inference that LLMs “reason effectively across all evaluated languages.”","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces mmPISA-bench, consisting of 25 multiple-choice PISA questions provided in official human translations to 43 languages plus machine-translated versions (2,150 items total). It evaluates two proprietary LLMs on accuracy across languages, reasoning effort, and translation type, claiming that modern LLMs reason effectively across all languages at levels comparable to human test-takers, that machine translation does not degrade accuracy relative to human translations, and that token usage (hence cost) varies with accuracy across languages.","tokens_in":1759,"tokens_out":595,"duration_ms":14316,"significance":"If the central empirical claims hold after addressing benchmark validity, the work supplies a compact, publicly derivable multilingual reasoning test set that could serve as a practical alternative to larger existing benchmarks. The observation that high-quality MT preserves performance would support broader use of synthetic data for multilingual evaluation. The cost-accuracy trade-off analysis adds a practical dimension for deployment considerations.","major_comments":[{"comment":"Abstract and §3 (Benchmark Construction): The headline claim that LLMs 'reason effectively across all evaluated languages' and achieve human-comparable accuracy rests on the assumption that the 25 selected PISA items measure transferable reasoning rather than training-data familiarity or language-specific phrasing. No decontamination check against the public OECD item pool is reported, nor are paraphrased controls or item-level error analysis provided; with only 25 questions this directly undermines the inference that accuracy reflects preserved reasoning steps under both human and machine translation.","section":"Abstract and §3"},{"comment":"§4 (Experimental Setup) and Results: The abstract asserts accuracy 'comparable to human test-takers' and no degradation under machine translation, yet supplies neither exact per-language accuracies, statistical significance tests, confidence intervals, nor prompting templates and decoding parameters. These omissions make it impossible to verify the quantitative support for the cross-lingual and translation-type claims.","section":"§4 and Results"},{"comment":"§5 (Analysis): The finding that machine-translated questions preserve accuracy is load-bearing for the recommendation that synthetic data can substitute for official translations. Without details on the MT system, translation-quality metrics, or controls that isolate reasoning from surface-form overlap, the result cannot be generalized beyond the specific 25 items.","section":"§5"}],"minor_comments":[{"comment":"The abstract refers to 'two mainstream proprietary LLMs' without naming the models or release versions; this information should appear in §4 for reproducibility.","section":"Abstract"},{"comment":"Token-usage and cost figures are mentioned but lack per-language tables or statistical comparison; adding these would strengthen the final analysis paragraph.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We agree that additional details on benchmark construction, experimental setup, and analysis are needed to strengthen the claims. We will revise the manuscript to incorporate the suggested improvements and address each point below.","responses":[{"response":"We acknowledge the need to demonstrate that the items primarily test reasoning. We will add a decontamination check in the revised §3 by systematically comparing the 25 selected items against the publicly released OECD PISA item pool and reporting any overlaps found. We will also include an item-level accuracy breakdown across languages to show consistency of performance rather than item-specific effects. While generating paraphrased variants is not feasible within the constraints of using official translations, the observed stability across 43 languages and between human/MT versions provides supporting evidence against language-specific phrasing. These additions will be made to support the claims despite the compact item set.","revision_made":"yes","referee_comment":"[Abstract and §3] Abstract and §3 (Benchmark Construction): The headline claim that LLMs 'reason effectively across all evaluated languages' and achieve human-comparable accuracy rests on the assumption that the 25 selected PISA items measure transferable reasoning rather than training-data familiarity or language-specific phrasing. No decontamination check against the public OECD item pool is reported, nor are paraphrased controls or item-level error analysis provided; with only 25 questions this directly undermines the inference that accuracy reflects preserved reasoning steps under both human and machine translation."},{"response":"We agree that full quantitative details are required for verification. The revised manuscript will include tables reporting exact per-language accuracies, 95% confidence intervals, and results of statistical significance tests (e.g., paired tests comparing human vs. machine translation accuracies). We will also document the complete prompting templates used and all decoding parameters (temperature, top-p, max new tokens, etc.). These elements will be added to §4 and the results section.","revision_made":"yes","referee_comment":"[§4 and Results] §4 (Experimental Setup) and Results: The abstract asserts accuracy 'comparable to human test-takers' and no degradation under machine translation, yet supplies neither exact per-language accuracies, statistical significance tests, confidence intervals, nor prompting templates and decoding parameters. These omissions make it impossible to verify the quantitative support for the cross-lingual and translation-type claims."},{"response":"We will expand §5 to specify the exact machine translation system and version used, report translation quality metrics (e.g., BLEU or COMET scores against human references), and add an analysis of surface-form overlap by quantifying lexical differences and examining accuracy on subsets with high vs. low overlap. This will better contextualize the generalization to synthetic data while remaining grounded in the 25-item set.","revision_made":"yes","referee_comment":"[§5] §5 (Analysis): The finding that machine-translated questions preserve accuracy is load-bearing for the recommendation that synthetic data can substitute for official translations. Without details on the MT system, translation-quality metrics, or controls that isolate reasoning from surface-form overlap, the result cannot be generalized beyond the specific 25 items."}],"tokens_in":1456,"tokens_out":676,"duration_ms":25160,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The new thing is mmPISA-bench: 25 multiple-choice PISA questions rendered in official translations across 43 languages plus machine-translated versions, for a total of 2150 items. They run two proprietary LLMs on them and report that accuracy stays roughly stable across languages, matches human levels, and does not drop with machine translation. They also note higher cost and lower accuracy in some languages.\n\nThat dataset construction is the main concrete addition. Having a compact set with both human and synthetic translations is practical for anyone who wants to spot-check many languages without hunting down official versions. The observation that MT holds up is worth knowing for scaling evaluations.\n\nThe problems are straightforward. Twenty-five questions is a tiny sample for claims about reasoning across dozens of languages. The abstract gives no prompting details, no statistical tests, no error breakdown, and no check against training-data overlap even though PISA items are public. The core inference—that high accuracy shows transferable reasoning preserved by translation—depends on the questions not being solvable by surface familiarity or language-specific cues, but nothing in the reported work tests that. Without decontamination or paraphrased controls, the results could partly reflect memorization rather than the intended skill.\n\nThis is useful as a quick internal check for teams already working on multilingual evals, but it does not yet support the headline claims at the level needed for wider citation or adoption. I would not bring it to a reading group and would not cite it. A serious editor should desk-reject rather than send to review; the scope and missing controls make it too preliminary.","headline":"A small new multilingual benchmark from PISA items, but the claims about cross-lingual reasoning rest on untested assumptions and very limited data.","tokens_in":2230,"tokens_out":393,"would_cite":false,"duration_ms":12016,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Modern LLMs reason effectively across all 43 evaluated languages on a compact PISA-derived benchmark and match human test-taker accuracy with some variations.","keywords":["multilingual reasoning benchmark","LLM evaluation","PISA questions","cross-lingual performance","machine translation","reasoning accuracy","inference cost"],"falsifier":"A finding that accuracy on the 25 questions falls substantially below human levels in most languages or that the questions require language-specific cultural knowledge would falsify the central claim.","tokens_in":2570,"feed_emoji":"🌍","tokens_out":626,"duration_ms":13242,"temperature":0.7,"pith_summary":"The paper creates mmPISA-bench from 25 PISA multiple-choice questions that require reasoning. Each question appears in official translations across 43 languages plus machine-translated versions. Two proprietary LLMs are tested on all versions for accuracy, token use, and cost. Results indicate the models handle reasoning tasks at human-comparable levels in every language tested. Machine translations perform no worse than official ones, though some languages show both higher cost and lower accuracy.","feed_headline":"LLMs match human reasoning accuracy across 43 languages on PISA questions","feed_subtitle":"A 25-question benchmark shows consistent results with machine translations performing as well as official ones.","key_machinery":"mmPISA-bench, the set of 25 multiple-choice reasoning questions drawn from PISA and supplied in 43 languages with both human and machine translations.","core_discovery":"The paper establishes that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. Machine-translated questions do not degrade accuracy relative to official human translations, suggesting high-quality machine translation is often adequate for large-scale multilingual reasoning evaluations.","pith_inferences":["Global deployment of LLMs for reasoning tasks becomes more feasible if the pattern holds beyond the 43 languages tested.","Developers may need to optimize token usage separately for lower-performing languages to control costs.","Extending the same question set to additional languages or question formats could test whether the observed consistency generalizes.","The finding that machine translations work well reduces reliance on scarce human translation resources for future benchmarks."],"forward_implications":["LLMs can handle reasoning tasks in many languages at levels comparable to humans without major accuracy loss.","High-quality machine translation suffices for building large multilingual reasoning benchmarks when official translations are unavailable.","Inference cost and accuracy vary together by language, with some languages proving both more expensive and less accurate.","The benchmark provides a compact way to compare cross-lingual reasoning without relying on language-specific knowledge."],"fun_headline_variants":["LLMs match human accuracy on PISA across 43 languages","Machine-translated PISA questions match official LLM performance","mmPISA-bench shows LLM reasoning consistency in 43 languages","LLMs reason consistently across 43 languages on PISA benchmark"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 25 selected PISA questions test general reasoning ability rather than language-specific knowledge, memorization, or test familiarity, and translations preserve these reasoning requirements.","fun_headline_variants_meta":{"raw":{"variants":["LLMs match human accuracy on PISA across 43 languages","Machine-translated PISA questions match official LLM performance","mmPISA-bench shows LLM reasoning consistency in 43 languages","LLMs reason consistently across 43 languages on PISA benchmark"]},"model":"grok-4.3","cost_usd":0.004572,"raw_usage":{"total_tokens":2248,"prompt_tokens":623,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":45724500,"prompt_tokens_details":{"text_tokens":623,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1557,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":623,"tokens_out":68,"duration_ms":14913,"temperature":1.0,"reasoning_tokens":1557,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T22:05:13.592465+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A finding that accuracy on the 25 questions falls substantially below human levels in most languages or that the questions require language-specific cultural knowledge would falsify the central claim.","supporting_citations":[],"review_version":1}