{"id":"b39e2187-d959-4710-900d-ee8ee9fe9179","arxiv_id":"2502.02289","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Evalita-LLM is a native-Italian, multi-prompt benchmark for LLMs built from ten Evalita datasets, with development-phase scores for six mid-size instruction-tuned models.","lead":"The authors built Evalita-LLM, a benchmark for testing Italian-language AI models using original Italian tasks, including both multiple-choice and open-ended generation. It is meant to give researchers a fairer way to compare Italian LLMs by testing each task with several different prompts instead of one.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The iterative prompt/task selection uses the same six dev LLMs that later demonstrate prompt robustness, so reported multi-prompt metrics may be inflated by selection; external validity beyond 7B-9B instruct models is untested.","rationale":"The reader's weakest assumption is the representativeness of the dev LLM cluster and the resulting selection bias. I agree with that, but the sharper formulation is internal: the same data (six dev LLMs) is used both to select prompts/tasks and to measure prompt robustness. Because uncooperative prompts are removed, the reported CPS/saturation on dev LLMs is optimistic by construction. This does not make the benchmark useless, but it means the paper cannot claim to have demonstrated robustness or objectivity from its own experiments; it is a conditional contribution pending held-out validation. The anonymization of dev LLMs compounds the issue, since the reader cannot assess how narrow the cluster really is. These points reinforce, rather than overturn, the reader's CONDITIONAL verdict. I would not reject: the benchmark artifacts are public, tasks are native Italian, and the multi-prompt infrastructure is a genuine improvement over single-prompt practice.","tokens_in":29098,"tokens_out":6922,"duration_ms":67267,"concrete_test":"Hold out 2 of the 6 multiple-choice prompt templates per task before the selection loop and freeze them; after the benchmark is finalized, compute MinP, AvgP, CPS, and prompt-variance on the same six dev LLMs for held-out versus selected prompts. If held-out prompts show materially lower AvgP/CPS or higher spread, the selection step has inflated the reported robustness, and the paper's multi-prompt \"objective evaluation\" claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 describes an iterative loop in which candidate prompts that are \"not well understood by any of the assessed LLMs\" are discarded and candidate tasks \"too complex for the dev LLMs\" are excluded, yet Section 6 reports the resulting MinP/MaxP/AvgP/CPS saturation scores on the very same six dev LLMs. This is a selection-on-the-evaluation-set problem: robust-looking prompt sets are partly a consequence of having removed low-performing prompts, not a property of the tasks. The \"minimum criteria\" promised in Section 2 are never quantified in Section 6, so the selection rule is unfalsifiable. Additionally, the six dev LLMs are anonymized and deliberately narrow (open-source, 7B-9B, instruction-tuned, Italian-pretrained), so the benchmark's validity for stronger, weaker, base, or non-Italian-centric models is assumed rather than shown. The central claim of a \"methodologically well-founded\" benchmark therefore rests on the unverified assumption that selection on this particular model cluster does not distort task difficulty and prompt sensitivity for the broader population.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Evalita-LLM, a benchmark for evaluating large language models on Italian, built from ten existing native-Italian datasets. Six tasks are cast as multiple-choice and four as generative (generate-until), each with several Italian prompt templates. The authors describe an iterative development methodology in which candidate tasks and prompts are validated on six anonymized 7B-9B instruction-tuned LLMs, report development-phase results using MaxP/AvgP/CPS metrics adapted from Mizrahi et al., and release the datasets on Hugging Face. The central claim is that the benchmark is methodologically well-founded for Italian LLM evaluation.","tokens_in":29275,"tokens_out":6698,"duration_ms":59517,"significance":"If the methodological claims held, the contribution would be a useful public resource: native-language evaluation tasks, multiple prompts per task, inclusion of generative tasks, and detailed per-task documentation with baselines. The use of lm-evaluation-harness and public Hugging Face datasets are strengths for reproducibility, and the explicit comparison of acc and acc_norm for multiple-choice scoring is a careful detail. However, the benchmark's validity hinges on the task and prompt selection procedure, and that procedure is currently underspecified and partly circular; the significance is therefore conditional on the issues below being resolved.","major_comments":[{"comment":"The iterative selection procedure uses the same six anonymized dev LLMs on which all reported dev-phase statistics are computed. Section 2 states that prompts 'not well understood by any of the assessed LLMs' are discarded and tasks 'too complex for the dev LLMs' are excluded, while Section 6.5 uses MaxP/AvgP/CPS values from those same models to conclude that the final prompt set captures different characteristics. This is a selection-on-the-evaluation-set problem: the reported multi-prompt robustness is partly a consequence of having removed low-performing prompts and tasks, not a property of the tasks themselves. The 'minimum criteria' promised in Section 2 are never quantified in Section 6, making the selection rule unfalsifiable. Please specify the criteria, report the number and identity of rejected tasks and prompts, and either validate the final benchmark on a held-out set of models or explicitly qualify the scope of the validity claims.","section":"Section 2, Section 6.1, Section 6.5"},{"comment":"Several generative tasks that were selected under the stated 'not too complex' criterion show zero or near-zero zero-shot performance for some dev LLMs: Lexical Substitution (Table A7) gives 0.00 for LLM-1 and LLM-3 on both prompts, Summarization (Table A8) gives 0.00 for LLM-1 on both prompts and an unexplained N/A for LLM-1 few-shot, and Relation Extraction (Table A10) gives 0.00 for LLM-1 on both prompts. If these tasks satisfied the minimum criteria, the criteria must be very close to zero; if they did not, the selection rule was not applied consistently. This matters because Section 6.5 uses the same tables to claim that the generative prompts capture useful signal. Please reconcile the selection rule with these results or provide per-task minimum thresholds.","section":"Appendix A (Tables A7, A8, A10), Section 6.4"},{"comment":"For the four generative tasks the reported experiments use only two of the four generative templates (p7/p9 for Lexical Substitution and Summarization, p8/p10 for NER and Relation Extraction), even though Section 4.3 presents four templates. The paper does not say whether the other two templates were tried and rejected, and if so on what evidence. The MaxP/AvgP/CPS numbers for generative tasks are therefore computed over two prompts only, which weakens the 'all tasks evaluated against multiple prompts' claim for exactly the tasks where prompt sensitivity is emphasized. Please state the intended final prompt set per task and justify the reduction.","section":"Section 4.3, Tables A7-A10"}],"minor_comments":[{"comment":"The caption and text call the Textual Entailment results 'zero-shot F1', but Section 3.2 states that the TE evaluation metric is accuracy; the column header and narrative should be aligned.","section":"Table 14, Section 6.2"},{"comment":"There are typos ('yje', 'featured implemented') and the discussion of byte-length normalization implicitly assumes English byte counts; for Italian, accented characters are multi-byte in UTF-8, so the normalization's effect on accented answers should be acknowledged.","section":"Section 6.3"},{"comment":"The text uses 'affermative' for 'affirmative' and 'for instances' for 'for instance', and says LLM-4 scores 63.10 with p6 in Sentiment Analysis while Table 15 reports 63.08; please harmonize.","section":"Section 4.2, Section 4.3, Section 6.2"},{"comment":"The dataset splits label the trial file 'multichoice v1 test.jsonl' and the test file 'multichoice v1 dev.jsonl', which is confusing for users; rename the files or add a note explaining the convention.","section":"Section 3.5"},{"comment":"The appendix mixes 'LLM 1' and 'LLM-1' naming, and Table A8 contains an unexplained N/A for LLM-1 few-shot Summarization; add a footnote explaining the missing value.","section":"Appendix A"},{"comment":"Table 1 has spacing issues ('F AQ', 'T ask') and Reference [18] has a typo in the title ('NotSo' instead of 'Not So').","section":"Table 1, References"}],"recommendation":"major_revision","confidential_remarks":"The selection-on-evaluation-set concern is real and should be addressed editorially. I am not asking the authors to un-anonymize the dev models, but the paper should either name them or provide enough selection statistics (number of tasks and prompts tried and rejected, thresholds) to make the iterative process auditable. The near-zero generative scores and the two-prompt generative setup are the most concrete places where the claimed methodology and the reported results diverge. The resource itself is likely useful after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Evalita-LLM is a genuine new artifact: a public Italian-language benchmark with ten native tasks, six multiple-choice and four generative, a fixed set of prompt templates, and a combined performance score. The paper documents data conversions, prompts, metrics, and dev-phase results in unusual detail, and the datasets are available on Hugging Face. For Italian NLP this is immediately useful, and the generative tasks address a real gap in translated-benchmark evaluation. The multi-prompt scoring follows Mizrahi et al. correctly. This is the kind of benchmark paper that should become a reference for Italian LLM evaluation.\n\nThe soft spots are real but not fatal. The stress-test note is on point: Section 2 describes an iterative loop where prompts that are 'not well understood' and tasks 'too complex' for the six dev LLMs are discarded, and Section 6 then reports prompt-robustness statistics on the same six models. This selection-on-evaluation-set means the reported MinP/MaxP/AvgP/CPS numbers are partly a consequence of the filtering, not an independent property of the tasks. The promised 'minimum criteria' are never quantified, so the selection rule is unfalsifiable. The dev LLMs are anonymized and deliberately narrow (open-source, 7B-9B, instruction-tuned, Italian-pretrained), so external validity for stronger, weaker, base, or non-Italian-centric models is untested. If this is to be a reference benchmark, the dev models need to be disclosed and the selection criteria stated, or the validation should be repeated on a held-out set of models not used in development.\n\nThe near-zero zero-shot scores on some generative tasks (e.g., Lexical Substitution is 0.00 for two models) are still a fact worth noting; the few-shot results are more informative. The abstract's claim that multi-prompt evaluation allows 'objective' evaluation is stronger than the evidence supports, since there are no human baselines or external validation. That is a framing issue rather than a technical one.\n\nWho is this for? Anyone evaluating Italian LLMs or building multilingual benchmarks. It deserves a serious referee; the methodological concerns are addressable and do not undermine the value of the resource. I would accept it for review and ask the authors to confront the selection issue head-on.","headline":"A useful, reproducible Italian benchmark whose robustness numbers are partly self-selected; worth reviewing and citing.","tokens_in":29851,"tokens_out":3373,"would_cite":true,"duration_ms":29832,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark evaluates Italian LLMs on native tasks with multiple prompts.","keywords":["Evalita-LLM","Italian LLM benchmark","large language models","multi-prompt evaluation","generative tasks","prompt validation","native language evaluation"],"falsifier":"Run the benchmark on a set of deliberately diverse models, including a smaller non-instruction model, a much larger instruction-tuned model, and a model trained primarily on English; if task scores or prompt variability shift dramatically relative to the dev-phase results, the task selection and prompt choices would be shown to depend on the dev model population.","tokens_in":28839,"feed_emoji":"🇮🇹","tokens_out":1320,"duration_ms":13883,"temperature":0.7,"pith_summary":"Evalita-LLM is a benchmark for evaluating large language models on Italian, built from existing native-Italian datasets rather than translated English benchmarks. The paper claims that this design avoids translation artifacts and cultural bias, and that scoring every task under several prompt templates gives a fairer picture of model ability than single-prompt evaluation. The authors also propose an iterative development method in which candidate tasks and prompts are validated on a small set of anonymized instruction-tuned models. If the benchmark is sound, it gives the Italian LLM community a public, reproducible evaluation suite that measures both peak performance and prompt robustness.","feed_headline":"Italian LLMs get a native-language benchmark with multi-prompt scoring","feed_subtitle":"Ten tasks, six prompt variants, and a combined score that penalizes models sensitive to how questions are phrased.","key_machinery":"The central machinery is the iterative validation methodology: candidate Italian datasets are adapted to a common evaluation harness, multiple prompt templates are designed for each task following explicit guidelines (Italian language, no role preambles, minimal verbosity, named input type), and both tasks and prompts are tested on a set of dev LLMs. The task is kept only if it is neither too easy nor too hard for those models, and a prompt is kept only if it is understood and produces differentiated results. The scoring framework adds a Combined Performance Score ($CPS$) that multiplies a model's best prompt performance ($MaxP$) by a saturation score ($1 - (MaxP - AvgP)$), so a model ranks well only if it is both strong and insensitive to prompt variation.","core_discovery":"The paper's central claim is that a methodologically well-founded benchmark for Italian LLMs can be built by combining three properties: all tasks are native Italian, the suite includes generative tasks alongside multiple-choice ones, and every task is evaluated under multiple prompt templates. The authors select ten tasks from Evalita and other Italian resources, adapt them to the lm-evaluation-harness format, define six prompt templates for multiple-choice tasks and four for generative tasks, and score them with metrics for maximum performance, average performance, and a combined performance score that penalizes prompt sensitivity. They report development-phase results from six anonymized 7B-9B instruction-tuned models, showing that tasks are challenging yet solvable and that different prompts capture different model characteristics. The paper argues that this constitutes a reference benchmark for Italian LLM evaluation that avoids the quality, content, and stylistic problems of translated benchmarks.","pith_inferences":["The anonymized dev models are all roughly 7B-9B and instruction-tuned, so the selected tasks may be skewed toward what mid-size instruction-tuned models can do; larger or non-instruction models might behave differently, and the benchmark may under-represent very hard or very easy tasks.","The multi-prompt design could be reused for other low-resource languages by following the same adaptation pipeline, provided equivalent native datasets exist.","A testable extension would be to compare Evalita-LLM scores against the same tasks rendered via translated prompts, to quantify how much of the performance gap is due to language and culture rather than task difficulty.","The CPS metric could be adopted as a general standard for any benchmark wanting to penalize prompt sensitivity, not just Italian-language ones."],"forward_implications":["Italian LLM providers can use Evalita-LLM to compare models on native language tasks without worrying that translations or cultural content skewed the results.","A model's reported score becomes more trustworthy because it must perform well across several prompts, not just one hand-picked template.","The generative tasks (lexical substitution, NER, relation extraction, summarization) push evaluation toward natural interaction, revealing which models can follow output-format instructions.","The development-phase scores give early, anonymized evidence about how current open Italian LLMs behave, including the difficulty of conditioning output format in generative tasks.","The benchmark's public release on Hugging Face makes the suite usable by others for reproducible evaluation."],"supporting_citations":[{"why":"Supplies the lm-evaluation-harness library that implements task definition, evaluation, and few-shot prompting for all benchmark tasks.","marker":"[5]"},{"why":"Provides the multi-prompt evaluation framework, including the maximum, average, and combined performance score notation used for all tasks.","marker":"[19]"},{"why":"Documents the problems with translated English benchmarks that motivate the decision to use only native Italian datasets.","marker":"[2]"},{"why":"The MMLU benchmark is cited as an example of cultural bias in translated benchmarks, supporting the need for native tasks.","marker":"[3]"},{"why":"Defines the Word in Context task that is adapted into the benchmark's WiC multiple-choice task.","marker":"[6]"},{"why":"Defines the SENTIPOLC sentiment analysis task, adapted as the benchmark's Sentiment Analysis task.","marker":"[8]"},{"why":"Describes the Italian medical specialty test dataset used as the basis for the Admission Tests task.","marker":"[10]"}],"fun_headline_variants":["Italian LLMs face native benchmark with multi-prompt scoring","New Italian LLM benchmark: native tasks, generative tests, prompt variety","Native Italian benchmark for LLMs penalizes prompt sensitivity","Italian LLM benchmark uses native tasks and multiple prompt templates","Ten native Italian tasks, six prompts: new LLM benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the six anonymized dev LLMs, a small set of mid-size instruction-tuned open models, are representative enough of the LLMs that will later be evaluated that tasks and prompts validated on them will work for the broader population of Italian LLMs.","fun_headline_variants_meta":{"raw":{"variants":["Italian LLMs face native benchmark with multi-prompt scoring","New Italian LLM benchmark: native tasks, generative tests, prompt variety","Native Italian benchmark for LLMs penalizes prompt sensitivity","Italian LLM benchmark uses native tasks and multiple prompt templates","Ten native Italian tasks, six prompts: new LLM benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2933,"prompt_tokens":858,"completion_tokens":2075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":1991}},"tokens_in":474,"tokens_out":2075,"duration_ms":13251,"temperature":1.0,"reasoning_tokens":1991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:39:30.745189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the benchmark on a set of deliberately diverse models, including a smaller non-instruction model, a much larger instruction-tuned model, and a model trained primarily on English; if task scores or prompt variability shift dramatically relative to the dev-phase results, the task selection and prompt choices would be shown to depend on the dev model population.","supporting_citations":[{"cited_title":"In: Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024) (2024)","cited_arxiv_id":null,"evidence_quote":"Documents the problems with translated English benchmarks that motivate the decision to use only native Italian datasets."},{"cited_title":"Proceedings of the International Conference on Learning Representations (ICLR) (2021)","cited_arxiv_id":null,"evidence_quote":"The MMLU benchmark is cited as an example of cultural bias in translated benchmarks, supporting the need for native tasks."},{"cited_title":"EV ALITA (2023)","cited_arxiv_id":null,"evidence_quote":"Defines the Word in Context task that is adapted into the benchmark's WiC multiple-choice task."},{"cited_title":": Overview of the evalita 2016 sentiment polarity classification task","cited_arxiv_id":null,"evidence_quote":"Defines the SENTIPOLC sentiment analysis task, adapted as the benchmark's Sentiment Analysis task."},{"cited_title":"In: CLiC-it (2023)","cited_arxiv_id":null,"evidence_quote":"Describes the Italian medical specialty test dataset used as the basis for the Admission Tests task."}],"review_version":1}