Reliable, Reproducible, and Really Fast Leaderboards with Evalica

Dmitry Ustalov

Reliable, Reproducible, and Really Fast Leaderboards with Evalica

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.11314 v1 pith:XW6RXAGD submitted 2024-12-15 cs.CL

Dmitry Ustalov This is my paper · ORCID

classification cs.CL

keywords evalicainterfacelanguageleaderboardsreliablereproducibleadvancementcommand-line

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

The rapid advancement of natural language processing (NLP) technologies, such as instruction-tuned large language models (LLMs), urges the development of modern evaluation protocols with human and machine feedback. We introduce Evalica, an open-source toolkit that facilitates the creation of reliable and reproducible model leaderboards. This paper presents its design, evaluates its performance, and demonstrates its usability through its Web interface, command-line interface, and Python API.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

JointRank: Rank Large Set with Single Pass
cs.IR 2025-06 conditional novelty 6.0 of 10

JointRank partitions candidates into overlapping blocks, ranks each block in parallel with an LLM, and reconstructs a global ranking by aggregating the resulting pairwise comparisons.

Pith tools