REVIEW 7 cited by
Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multiple-choice questions (MCQ) are frequently used to assess large language models (LLMs). Typically, an LLM is given a question and selects the answer deemed most probable after adjustments for factors like length. Unfortunately, LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to inherent biases of priori unbalanced probabilities, influencing the prediction of answers based on these IDs. Previous research has introduced methods to reduce this ''selection bias'' by simply permutating options on a few test samples and applying to new ones. Another problem of MCQ is the lottery ticket choice by ''random guessing''. The LLM does not learn particular knowledge, but the option is guessed correctly. This situation is especially serious for those small-scale LLMs. To address them, a more thorough approach involves shifting from MCQ to open-style questions, which can fundamentally eliminate selection bias and random guessing issues. However, transitioning causes its own set of challenges in (1) identifying suitable open-style questions and (2) validating the correctness of LLM open-style responses against human-annotated ground-truths. This work aims to tackle these significant difficulties, and establish a new LLM evaluation benchmark through entirely open-style questions. Consequently, we introduce the Open-LLM-Leaderboard to track various LLMs' performance and reflect true capability of them, such as GPT-4o/4/3.5, Claude 3, Gemini, etc. Our code and dataset are available at https://github.com/VILA-Lab/Open-LLM-Leaderboard.
Forward citations
Cited by 7 Pith papers
-
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
HumorRank ranks nine LLMs on textual humor using GTVH-grounded pairwise tournaments and Adaptive Swiss aggregation on the SemEval-2026 MWAHAHA dataset, finding that comedic mechanism mastery matters more than scale.
-
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Map-reduce scaffolding degrades measured safety mainly by stripping multiple-choice options (40–89% of the loss is format conversion); scaffold architecture explains only 0.4% of variance and composite safety scores h...
-
Human-Centric Evaluation for Foundation Models
In free-form research collaborations rated by humans, Grok 3 scores highest, followed by DeepSeek R1 and Gemini 2.5; OpenAI o3 mini trails.
-
Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans
LLMs align more closely with human ratings on affective and cognitive word norms than on sensory-perceptual norms, suggesting a gap tied to embodied experience.
-
KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
A reusable per-topic knowledge graph, built once from Wikipedia, lets an LLM generate multi-hop multiple-choice questions whose difficulty is set by path depth, with human-audited quality and model rankings that track MMLU.
-
SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
A new benchmark of 15 small language models across 23 datasets and 11 metrics shows clear accuracy-versus-energy trade-offs, with no single model dominating.
-
DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
A small model prompted with evidence and knowledge graphs generated by GPT-4o scores much higher on QA benchmarks, but the result is not true distillation and may be contaminated by teacher answer leakage.
Discussion (0). Sign in to comment.