Ouvia is a user-centered evaluation framework for speech translation usability in real-world scenarios, showing limited usability rates and the superiority of QA-based metrics.
Title resolution pending
6 Pith papers cite this work, alongside 8 external citations. Polarity classification is still indexing.
6
Pith papers citing it
8
external citations · OpenAlex
fields
cs.CL 6years
2026 6representative citing papers
Dialectal robustness and generation are dissociated in LLMs: benchmarks are driven by pretraining and SFT while alignment reshapes generation invisibly to benchmarks, and the method maximizing dialectal reward is least preferred by human evaluators.
COMPASS is a new reproducible benchmarking framework for S2ST that deploys 46 metrics on 1248 configurations, shows single-metric rankings mislead, reduces to 10 metrics per direction, and finds domain-specific metrics better match human judgments than standalone MOS predictors.
A multi-reference audit framework for LLM translations of the Pali Canon uses embedding drift from a human reference centroid to triage candidates for LLM-judge adjudication, showing drift correlates with major error rates and model-specific differences in the high-drift tail.
Compact 0.8B-7B models for bidirectional Japanese-English translation outperform large multilingual models on real-world domain benchmarks.
citing papers explorer
-
Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios
Ouvia is a user-centered evaluation framework for speech translation usability in real-world scenarios, showing limited usability rates and the superiority of QA-based metrics.
-
DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
Dialectal robustness and generation are dissociated in LLMs: benchmarks are driven by pretraining and SFT while alignment reshapes generation invisibly to benchmarks, and the method maximizing dialectal reward is least preferred by human evaluators.
-
Benchmarking Speech-to-Speech Translation Models
COMPASS is a new reproducible benchmarking framework for S2ST that deploys 46 metrics on 1248 configurations, shows single-metric rankings mislead, reduces to 10 metrics per direction, and finds domain-specific metrics better match human judgments than standalone MOS predictors.
-
From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication
A multi-reference audit framework for LLM translations of the Pali Canon uses embedding drift from a human reference centroid to triage candidates for LLM-judge adjudication, showing drift correlates with major error rates and model-specific differences in the high-drift tail.
-
CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation
Compact 0.8B-7B models for bidirectional Japanese-English translation outperform large multilingual models on real-world domain benchmarks.
- Why Do Safety Guardrails Degrade Across Languages?