{"id":"25db03e2-48b3-4664-b9d9-227bdac8300c","arxiv_id":"2412.00559","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Polish-English medical exam benchmark shows GPT-4o answering at or above average human level, with smaller and medical-specific models lagging and persistent cross-lingual gaps.","lead":"This paper introduces a benchmark of over 22,000 Polish medical exam questions, including professionally translated English versions, and tests how well AI models answer them. It finds GPT-4o performs at or above average human level in Polish, while smaller and medical-specific models lag, and models generally do better in English.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline performance numbers may reflect memorization of the public CEM/NIL question bank; the paper's own Limitations section concedes this risk without a mitigation, so the benchmark's validity as a capability measure is unproven.","rationale":"The Reader's verdict identified the same load-bearing assumption: the benchmark measures capability only if the public exam questions were not memorized during model training. I agree that this is the single most important threat to the paper's central claim. The authors themselves acknowledge the risk in the Limitations section, but they do not test for it, quantify it, or provide a decontaminated subset. The temporal holdout test on 2025 exams would settle the question cleanly, because those questions were not available at the model cutoffs and the non-public-bank portions are especially informative. I do not see a stronger competing concern. The dataset-size discrepancy between the abstract ('over 24,000') and Table 1 (22,604 usable questions) is real but cosmetic for the main claim. The lack of repeated runs and statistical uncertainty is also a genuine limitation, but it would affect confidence intervals rather than overturn the main result. The uncontaminated-resource value of the benchmark is real, and the paper's methodology is largely transparent, so I would not reject the paper. I would keep the Reader's CONDITIONAL verdict: the benchmark is a useful contribution, but the headline performance numbers should not be taken at face value until contamination is either ruled out by a temporal holdout or explicitly bounded and the claims are re-reported on a decontaminated subset.","tokens_in":19622,"tokens_out":4739,"duration_ms":50079,"concrete_test":"Run a temporal holdout evaluation: score gpt-4o-2024-08-06 and Meta-Llama-3.1-70B-Instruct on the 2025 Spring and Autumn LEK, LDEK, and PES sessions, which were published after the models' knowledge cutoffs, using the exact prompts, answer parsing, and pass thresholds from the paper. For LEK and LDEK, restrict the comparison to the roughly 30% of items that are not drawn from the public pre-2024 question bank, so those items could not have been memorized from the benchmark source. If held-out accuracy stays within 2-3 points of the corresponding Table 2 entries, contamination is not the dominant explanation; if it drops by more than about 5-10 points, or if GPT-4o no longer passes the same proportion of exams, the public-bank scores are inflated and the paper should be revised to report decontaminated results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the benchmark measures medical question-answering capability and that GPT-4o reaches near-human performance. The load-bearing premise is that models have not memorized the exam questions. That premise is insecure: every evaluated question comes from the publicly accessible CEM and NIL websites and predates the training cutoffs of models such as GPT-4o-2024-08-06 and Llama-3.1-70B-Instruct. These are internet-scale models, and the exact multiple-choice question stems and answer keys are exactly the kind of content that is plausibly included in web-crawl training data. Because a single correct answer is supplied, verbatim or near-verbatim memorization would inflate accuracy without requiring medical reasoning. The paper's Limitation section explicitly acknowledges this possibility: 'There is a potential risk for these exams being included in the training datasets of evaluated LLMs.' It then compares the risk to MMLU and public leaderboards, but that analogy does not mitigate the problem; it concedes that the benchmark shares the contamination flaw it is meant to avoid. The statement that 'the training dataset is not provided' also does not help, because the questions themselves were already public before model training, so withholding a local copy does not prevent contamination. If contamination is present, the headline numbers in Table 2, the passed-exam counts in Table 3, and the near-human comparisons in Table 7 are not interpretable as measures of capability. The cross-lingual comparisons in Tables 4 and 5 are less directly threatened if the contamination is comparable across languages, but the main performance claims remain unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new benchmark for Polish medical question answering, built from public LEK, LDEK, and PES licensing and specialization exams, including a Polish-English parallel subset with professionally produced English translations. The authors evaluate sixteen LLMs, reporting per-exam accuracy and pass rates (Tables 2 and 3), cross-lingual comparisons on matched Polish/English question subsets (Tables 4 and 5), and comparisons with human exam performance (Tables 6 and 7). The headline claims are that GPT-4o achieves near-human performance on these exams, that most models perform better in English than in Polish, and that general-purpose multilingual models outperform medical-specific models.","tokens_in":19828,"tokens_out":5611,"duration_ms":54989,"significance":"If the benchmark measures what it claims, it is a useful resource for non-English medical QA evaluation and cross-lingual knowledge-transfer studies. The dataset draws on an authoritative examination source, includes professionally translated parallel questions rather than machine-translated ones, covers a long time span (2008-2024), and is publicly released. The appendices document data acquisition and preprocessing in unusual detail, and the inclusion of human results is a valuable feature. The main significance hinges on whether the public, pre-cutoff exam questions can support capability claims rather than memorization-based scores; this is the central validity question the paper does not resolve.","major_comments":[{"comment":"The contamination risk is acknowledged but not mitigated, and it is load-bearing for the paper's central claim. All exam questions are publicly available on the CEM and NIL websites and predate the training cutoffs of several evaluated models, including gpt-4o-2024-08-06 and Llama-3.1-70B-Instruct. The Limitations paragraph states that 'there is a potential risk for these exams being included in the training datasets of evaluated LLMs' and then compares the situation to MMLU, but that analogy concedes rather than resolves the problem. The statement that 'the training dataset is not provided' is also not a mitigation, because the questions themselves were already public. As a result, the near-human GPT-4o results in Table 7, the pass counts in Table 3, and the accuracy ranking in Table 2 are not interpretable as measures of medical reasoning ability unless the authors provide a concrete contamination analysis. I would ask for at least one of the following: a question-level overlap analysis with known training corpora, an evaluation restricted to questions published after the model training cutoffs, an answer-only memorization probe, or a clear reframing of the results as an upper-bound benchmark in which memorization is an explicitly uncontrolled factor.","section":"Limitations (final section); Tables 2, 3, and 7"},{"comment":"All accuracy results are single-run point estimates with no error bars, confidence intervals, or significance tests. For exams with roughly 200 questions, a difference of two to three percentage points can easily lie within binomial sampling noise, yet the paper makes comparative claims such as 'GPT-4o is the best performing model overall' and 'the performance gap between languages narrows' without any uncertainty quantification. The authors should report repeated runs with the decoding parameters (temperature, top-p, number of runs), or at minimum binomial confidence intervals for each reported accuracy. Without this, the ranking of closely clustered models in Tables 2 and 4 and the cross-lingual gap analysis cannot be evaluated.","section":"Section 4, Tables 2-5; Section 5, Tables 4-5"},{"comment":"The human comparison may not be apples-to-apples. The dataset description states that questions containing images and invalidated questions were removed, and Table 1 reports 'invalidated questions' for every sub-dataset. However, Table 7 compares raw model scores (e.g., 184 out of 200 for GPT-4o) with published human averages and standard deviations from the CEM webpage. It is not stated whether the human statistics were recomputed on the same filtered question subset or whether the model scores were rescaled to account for excluded questions. In addition, the text says the comparison covers '977 LEK and 984 LDEK questions' from four sessions of each exam, but each LEK and LDEK session has only 200 questions, so four sessions yield at most 800 questions; this number needs clarification. The authors should specify the identical question set used for both humans and models, and recompute human statistics on that set if necessary.","section":"Section 6, Table 7"}],"minor_comments":[{"comment":"The abstract states that the dataset 'comprises over 24,000 exam questions,' but Table 1 sums to 22,604 valid questions plus 436 invalidated questions, i.e., 23,040 total. Section 1 says 'over 22,000 questions.' The numbers should be reconciled, and the abstract's figure should be corrected.","section":"Abstract; Table 1"},{"comment":"The text says 'For the PES dataset, we collected a total of 180,712 questions,' but Table 1 reports 8,532 valid PES questions. If the 180,712 figure is the raw scraped count before filtering and selection of the most recent exam per specialty, this should be stated explicitly and connected to the analysis set.","section":"Section 3, Table 1"},{"comment":"In Table 10, the last column header 'Incorrect PL, Incorrect EN' appears to be a typo for 'Incorrect PL, Correct EN' based on the category definitions in the same appendix. Table 11 uses the correct 'Correct EN, Incorrect PL' label for the corresponding column.","section":"Appendix F, Tables 10 and 11"},{"comment":"The model naming is inconsistent: the text lists 'GPT-4-o' while Table 2 uses 'gpt-4o-2024-08-06' and 'gpt-4o-mini-2024-07-18.' Standardizing model names across the text and tables would improve reproducibility.","section":"Section 4, model list and Table 2"},{"comment":"The sentence 'To prevent models from being trained on the benchmark data, the training dataset is not provided' is confusing because the benchmark itself is released on Hugging Face. Clarify what is withheld, and note that public availability of the original exam questions makes this statement ineffective as a contamination safeguard.","section":"Limitations section"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a valuable resource and a generally careful data-construction process, but the central capability claims are currently unprotected against the acknowledged contamination risk. This is not a fatal flaw in the benchmark's utility, but it is a load-bearing validity issue that should be addressed with concrete analysis rather than an analogy to other public benchmarks. The absence of uncertainty quantification is also a fixable but important weakness. If the authors add a contamination probe or reframe the claims accordingly, and tighten the human-comparison methodology, the paper could be a solid contribution to multilingual medical QA evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper's real contribution is the benchmark, not the accuracy numbers. The LDEK dataset and the professionally translated PL-EN parallel subset fill a genuine gap. Second, the headline result – GPT-4o near human performance on Polish medical exams – is not secure because every question is public and predates the models' training cutoffs. The authors acknowledge this in Limitations but wave at MMLU, which concedes the problem rather than fixing it.\n\nWhat's genuinely good: scraping and processing are described in unusual detail, invalidated questions are handled explicitly, and the dataset is released. The cross-lingual question-level analysis (Appendix F) is the most interesting part; the fact that wrong answers are usually the same in both languages is real evidence for shared knowledge, and the idea of using cross-language disagreement as a low-confidence filter is worth testing. Comparing models to human score distributions is also better than just pass/fail.\n\nSoft spots, in proportion. The contamination risk is load-bearing, not a footnote. A single correct letter is published with each question, so verbatim memorization inflates accuracy without requiring medical reasoning. The paper's Limitations section says the risk exists and then compares to MMLU; that does not mitigate it, and the statement that the training dataset is not provided is beside the point since the questions themselves are public. The cross-lingual comparisons are somewhat safer if contamination is symmetric across Polish and English, but the near-human claims in Tables 2 and 7 are capability claims built on sand until there is some test – time-split evaluation, paraphrased questions, or held-out not-public items. Second, every result is a single-run point estimate. No error bars, no significance tests; differences of a few percentage points between models are noise. Third, the abstract says 'over 24,000 questions' while Table 1 reports 22,604 valid; small but sloppy. The PES dataset has 180k raw questions but only the most recent exam per specialty is analyzed – explained, but it means the paper's analysis is on a smaller slice than the dataset headline suggests.\n\nWho is this for: people working on multilingual medical QA, or on non-English benchmarks generally. It is not a paper that settles a scientific question; it is a resource paper with model evaluation attached. The resource deserves a serious referee; the evaluation section needs work before headline numbers are taken at face value. I'd send it to review with the clear instruction that contamination mitigation and uncertainty estimates are required.","headline":"Useful Polish medical benchmark and professional PL-EN parallel data, but the near-human GPT-4o results rest on an acknowledged and unmitigated contamination risk.","tokens_in":20455,"tokens_out":3573,"would_cite":true,"duration_ms":32078,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark built from Polish medical licensing exams shows GPT-4o performing at roughly the level of an average medical student, while most other models answer English versions of the same questions better than Polish ones.","keywords":["Polish medical exams","LEK","LDEK","PES","medical question answering","cross-lingual evaluation","large language models","benchmark"],"falsifier":"Have the Medical Examination Center or an independent panel write new LEK, LDEK, and PES-style questions that have never been published, give the same models the same prompt, and compare accuracy on the private set with accuracy on the public benchmark; if GPT-4o's near-human scores drop substantially on the private set while human scores do not, the public benchmark's headline numbers overestimate model capability due to memorization.","tokens_in":19407,"feed_emoji":"🩺","tokens_out":8577,"duration_ms":180894,"temperature":0.7,"pith_summary":"The paper assembles a benchmark from Polish medical licensing and specialization exams (LEK, LDEK, and PES) that were published by the Medical Examination Center and the Chief Medical Chamber, producing over 22,000 usable multiple-choice questions plus a parallel subset in which the English versions are professional translations made by the examination center itself. It uses this benchmark to ask how well large language models answer Polish medical questions, how their scores compare with those of human examinees, and how much medical knowledge transfers from English to Polish. On the benchmark, GPT-4o scores near the average human on LEK and LDEK, passes 68 of 72 PES exams, and outperforms the median human on a majority of PES specializations, while almost every other tested model scores higher on English than on Polish versions of identical questions. The authors conclude that general-purpose models outperform medical-specific models, that the Polish-English gap narrows as model quality improves, and that performance remains too uneven across medical specialties for unsupervised clinical use.","feed_headline":"New Polish medical exam benchmark: GPT-4o near human level","feed_subtitle":"22,000+ Polish exam questions show most models answer English versions better than identical Polish ones.","key_machinery":"The load-bearing object is the parallel Polish-English benchmark. Its engine is the examination center's professionally translated English versions of LEK and LDEK questions, which make the Polish and English forms semantically equivalent and thereby let a score difference between languages be read as a language-transfer effect rather than translation noise. The three exam types give the benchmark a difficulty gradient: LEK and LDEK are final licensing exams with a high share of questions from a public bank, while PES is a harder specialization exam whose questions are not public. The same questions are paired with anonymized human score distributions, which is what allows model accuracy to be converted into human-percentile comparisons and per-specialty pass/fail judgments.","core_discovery":"The central claim is that a structured benchmark built from publicly available LEK, LDEK, and PES exam questions can support valid measurement of Polish medical question answering and of cross-lingual medical knowledge transfer, because the English portion is a human-expert translation of the Polish portion produced by the examination center. On that benchmark, the paper reports GPT-4o answering 89.4% of LEK questions correctly, 75.35% of PES questions correctly, and scoring within one standard deviation of the average human on LEK and LDEK, while Meta-Llama-3.1-70B-Instruct is the best open model. It reports that most models score higher on English versions of the same questions, that the gap narrows as overall performance improves (about 13 percentage points for Llama-3.1-8B on LEK versus less than 2 points for Llama-3.1-70B), and that GPT-4o scores slightly higher in Polish than in English. The paper also claims that medical-specific models fine-tuned on English data do not beat general-purpose models on these Polish exams, which it attributes to the language mismatch of the fine-tuning data.","pith_inferences":["A private held-out set of newly written, never-published exam questions would separate genuine medical reasoning from memorization; if GPT-4o's near-human scores drop sharply on that set, the public benchmark's headline numbers should be read as upper bounds.","The benchmark's Polish-English parallel structure could be extended by having the exam center translate a PES subset; this would test whether the cross-lingual gap grows with specialization difficulty, which the current data cannot answer.","The question-level agreement pattern suggests a practical clinician-facing filter: asking a model the same question in both languages and flagging answers that disagree could highlight low-confidence responses for human review, a use the paper only hypothesizes."],"forward_implications":["If the benchmark measures capability as intended, GPT-4o is the only tested model that consistently performs at the level of an average medical student, and the only one that passes nearly every PES specialization.","Most models answering identical questions better in English means that English-centric training leaves a residual Polish medical knowledge deficit; for smaller models, evaluating in English would overstate their Polish clinical competence.","The narrowing of the Polish-English gap with model scale implies that larger models transfer medical knowledge across languages more effectively, so cross-lingual capability should be reported as a function of model size.","General-purpose models beating medical-specific models implies that English-only medical fine-tuning does not translate into an advantage on Polish exams; specialized models need per-language evidence before deployment.","Per-specialty differences, with dental specialties hardest and laboratory diagnostics easiest, imply that LLM deployment in Polish medicine should be specialty-aware rather than uniform."],"supporting_citations":[{"why":"It supplies the earlier 297-exam GPT-3.5/4 evaluation across 57 PES specialties that this benchmark extends with LEK, LDEK, and additional models.","marker":"Pokrywka et al. (2024)"},{"why":"It provides the prior GPT-3.5 and GPT-4 evaluation on the Polish LEK that set the baseline for this paper's LEK results.","marker":"Rosoł et al. (2023)"},{"why":"It contributes the earlier multi-model LEK evaluation whose shared-weakness analysis the paper extends to more models and exam types.","marker":"Bean et al. (2024)"},{"why":"It established the cross-lingual healthcare benchmark with machine-translated questions that this paper contrasts with expert-translated questions.","marker":"Jin et al. (2024)"},{"why":"It introduced the MedQA medical-exam dataset in English and Chinese that the paper positions its Polish benchmark as a counterpart to.","marker":"Jin et al. (2021)"},{"why":"It provides the MMLU public-dataset precedent the paper cites when cautioning that its results face training-data contamination.","marker":"Hendrycks et al. (2020)"},{"why":"It documents the Llama 3.1 model family, a key open general-purpose comparator in the evaluation.","marker":"Dubey et al. (2024)"},{"why":"It documents the Qwen2.5 model family that produces two of the top open-model results.","marker":"Team (2024)"},{"why":"It is the technical report cited for the GPT-4 model family that includes the evaluated GPT-4o and GPT-4o-mini.","marker":"Achiam et al. (2023)"}],"fun_headline_variants":["Polish medical benchmark: GPT-4o near human, most models lag in Polish","Cross-lingual gap found in 24K Polish medical exam questions","English beats Polish for most AI on new medical exam benchmark","GPT-4o near humans on Polish exams, other AI lag in Polish","Polish medical exam data show AI still weaker in native language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement stands or falls on the assumption that a model's score reflects its medical knowledge rather than its memory of questions it already saw during training, because every question was publicly available before several evaluated models' training cutoffs and the paper does not filter for overlap.","fun_headline_variants_meta":{"raw":{"variants":["Polish medical benchmark: GPT-4o near human, most models lag in Polish","Cross-lingual gap found in 24K Polish medical exam questions","English beats Polish for most AI on new medical exam benchmark","GPT-4o near humans on Polish exams, other AI lag in Polish","Polish medical exam data show AI still weaker in native language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2398,"prompt_tokens":981,"completion_tokens":1417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":597,"tokens_out":1417,"duration_ms":10057,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:13:09.774865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have the Medical Examination Center or an independent panel write new LEK, LDEK, and PES-style questions that have never been published, give the same models the same prompt, and compare accuracy on the private set with accuracy on the public benchmark; if GPT-4o's near-human scores drop substantially on the private set while human scores do not, the public benchmark's headline numbers overestimate model capability due to memorization.","supporting_citations":[],"review_version":1}