Qualitative study of 19 practitioners reveals ten LLM product evaluation practices and introduces the results-actionability gap as a key barrier to turning findings into improvements.
Zamfirescu-Pereira, Björn Hartmann, Aditya G
6 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
6
Pith papers citing it
1
external citations · external index
years
2026 6representative citing papers
Even with temperature pinned at 0 and greedy decoding, 1-2 of 7 borderline safety items flip pass/fail verdicts across runs on real LLM providers, and some models deprecate temperature control entirely.
The authors introduce a three-part ontology-based verification system for AI agents that generates regulatory and adversarial test scenarios and issues machine-verifiable trust certificates, with pilot results indicating improved coverage over baselines in four industries.
Gemini 2.5 Flash with a Combined Budget debiasing strategy achieves 71.0% judge agreement at ~$0.001/evaluation, outperforming frontier models at 15x lower cost.
CausaDisco integrates Aristotle's Four Causes into LLM prompts to produce more engaging, exploratory, and multifaceted self-learning dialogues, as evidenced by controlled user studies.
Agentic-J is a multi-agent AI assistant that converts natural language descriptions of biological image analysis tasks into executable, reproducible scripts for ImageJ/Fiji with specialised sub-agents for plugin management, code generation, debugging and reporting.
citing papers explorer
-
Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild
Qualitative study of 19 practitioners reveals ten LLM product evaluation practices and introduces the results-actionability gap as a key barrier to turning findings into improvements.
-
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Even with temperature pinned at 0 and greedy decoding, 1-2 of 7 borderline safety items flip pass/fail verdicts across runs on real LLM providers, and some models deprecate temperature control entirely.
-
Toward Pre-Deployment Assurance for Enterprise AI Agents: Ontology-Grounded Simulation and Trust Certification
The authors introduce a three-part ontology-based verification system for AI agents that generates regulatory and adversarial test scenarios and issues machine-verifiable trust certificates, with pilot results indicating improved coverage over baselines in four industries.
-
Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Gemini 2.5 Flash with a Combined Budget debiasing strategy achieves 71.0% judge agreement at ~$0.001/evaluation, outperforming frontier models at 15x lower cost.
-
Enhanced Self-Learning with Epistemologically-Informed LLM Dialogue
CausaDisco integrates Aristotle's Four Causes into LLM prompts to produce more engaging, exploratory, and multifaceted self-learning dialogues, as evidenced by controlled user studies.
-
Agentic-J: An AI Agent for Biological Microscopy Image Analysis
Agentic-J is a multi-agent AI assistant that converts natural language descriptions of biological image analysis tasks into executable, reproducible scripts for ImageJ/Fiji with specialised sub-agents for plugin management, code generation, debugging and reporting.