LLM judges over-credit wrong answers without a reference, and adding one can flip their decisions by up to 85%.

Summary. The manuscript studies whether LLM-as-judge systems can reliably score open-ended responses in no-reference settings. It proposes a two-stage pipeline: (a) calibration experiments that probe the judge model’s own knowledge of the evaluation task, and (b) sensitivity experiments that measure how presence and positioning of a reference answer in the prompt change the judge’s correct/incorrect decisions. Across experiments in three languages, the authors report that judges systematically over-credit incorrect answers when no reference is provided, that inserting reference information flips decisions by as much as 85% in some settings, and that these reference-driven flips generally align with a subset of human annotations. They conclude that LLM judges should be calibrated on reference-aware samples before being used in reference-free setups, and present the pipeline as a reusable blueprint.

Significance. If the reported over-crediting pattern and large decision flips hold under transparent methods and adequate sample sizes, the work would be practically important for the growing literature and industry practice of LLM-as-judge evaluation. The explicit recommendation to calibrate before reference-free use, and the framing of a reusable two-stage methodology, would give practitioners a concrete reliability check rather than an informal warning. The multi-language scope and the human-alignment check are strengths relative to purely English, model-only studies. Because only the abstract is available for this review, significance remains conditional on the uninspectable experimental design.