Pioneer Agent automates the full lifecycle of adapting and continually improving small language models via diagnosis-driven data synthesis and regression-constrained retraining, delivering gains of 1.6-83.8 points on benchmarks and large lifts in production-style tasks.
Llm-as-a-judge: Reassessing the performance of llms in extractive qa
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
Extractive QA tasks are commonly evaluated using Exact Match (EM) and F1-score, but these metrics often fail to reflect true model performance. Recent studies have proposed using large language models (LLMs) as judges (LLM-as-a-judge), yet they often lack comprehensive evaluation across datasets and overlook key factors such as sensitivity to answer types, prompt variations, and self-preference bias. In this work, we conduct a systematic study of LLM-as-a-judge across four extractive QA datasets and various prompt variations, assessing multiple LLM families in both answering and judging roles. Our results show that LLM-as-a-judge judgments correlate much more strongly with human evaluations than EM (0.22) and F1 (0.40), achieving correlations up to 0.85 with open-source models. Further analysis reveals that LLM-as-a-judge performs particularly well on number-related answers but faces challenges with more complex types, such as job titles. Contrary to findings in other NLP tasks, we observe no self-preference bias, even when the same model serves as both QA model and judge. Finally, we find that prompt phrasing has minimal impact, and zero-shot, context-free judging often yields the best evaluation performance.
years
2026 2verdicts
UNVERDICTED 2representative citing papers
SA-RAG uses tables as a compact intermediate representation for RAG, introduces a quality-aware table metadata generation framework, and reports outperformance over baselines on two noisy real-world datasets.
citing papers explorer
-
Pioneer Agent: Continual Improvement of Small Language Models in Production
Pioneer Agent automates the full lifecycle of adapting and continually improving small language models via diagnosis-driven data synthesis and regression-constrained retraining, delivering gains of 1.6-83.8 points on benchmarks and large lifts in production-style tasks.
-
Structure-Aware RAG: Structured Retrieval Augmented Generation from Noisy Data for Conversational Agents
SA-RAG uses tables as a compact intermediate representation for RAG, introduces a quality-aware table metadata generation framework, and reports outperformance over baselines on two noisy real-world datasets.