A prior audit-repair episode in context lowers LLM verifier false alarms by up to 11.5 percentage points via a criterion shift, and at the tested operating point most removed false alarms were fabrications.
arXiv preprint arXiv:2402.13764 , year=
9 Pith papers cite this work. Polarity classification is still indexing.
9
Pith papers citing it
abstract
Critique ability, i.e., the capability of Large Language Models (LLMs) to identify and rectify flaws in responses, is crucial for their applications in self-improvement and scalable oversight. While numerous studies have been proposed to evaluate critique ability of LLMs, their comprehensiveness and reliability are still limited. To overcome this problem, we introduce CriticEval, a novel benchmark designed to comprehensively and reliably evaluate critique ability of LLMs. Specifically, to ensure the comprehensiveness, CriticEval evaluates critique ability from four dimensions across nine diverse task scenarios. It evaluates both scalar-valued and textual critiques, targeting responses of varying quality. To ensure the reliability, a large number of critiques are annotated to serve as references, enabling GPT-4 to evaluate textual critiques reliably. Extensive evaluations of open-source and closed-source LLMs first validate the reliability of evaluation in CriticEval. Then, experimental results demonstrate the promising potential of open-source LLMs, the effectiveness of critique datasets and several intriguing relationships between the critique ability and some critical factors, including task types, response qualities and critique dimensions.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
JETTS, a new benchmark, shows LLM-as-judges are competitive in response reranking, worse than process reward models in beam search, and ineffective as critique providers for refinement.
Introduces (ε,q,t,A)-behavioral indistinguishability and shows via Qwen/Llama experiments that LoRA distillation boosts semantic similarity but leaves detectable behavioral differences under adversarial evaluation.
DDC couples confidence-weighted Bayesian early stopping with trend-aware pruning of reasoning paths, cutting inference tokens by over 10x on some benchmarks while roughly maintaining accuracy.
A multi-agent AI system generates novel biomedical hypotheses that show promising experimental validation in drug repurposing for leukemia, new targets for liver fibrosis, and a bacterial gene transfer mechanism.
CRITICTOOL, a benchmark of 2,740 tool-calling error scenarios built from BFCL and T-Eval with GPT-4o-based error injection, finds most LLMs rarely recover from tool-use errors, with GPT-4o best at 69.01 overall and tool-finetuned ToolLLaMA2 at 0.58.
A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.
The authors create a 1,000-query multi-modal RAG benchmark with GPT-4o-based metrics, show multi-stage generation beats single-stage, and report fine-tuned 7B-8B models beating GPT-4o only in the single-stage comparison.
LLMs can improve their own long-context reasoning by fine-tuning on outputs selected by Minimum Bayes Risk consensus, without external annotations.
citing papers explorer
-
Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
A prior audit-repair episode in context lowers LLM verifier false alarms by up to 11.5 percentage points via a criterion shift, and at the tested operating point most removed false alarms were fabrications.
-
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
JETTS, a new benchmark, shows LLM-as-judges are competitive in response reranking, worse than process reward models in beam search, and ineffective as critique providers for refinement.
-
Bounded Behavioral Indistinguishability for Black-Box LLM Distillation
Introduces (ε,q,t,A)-behavioral indistinguishability and shows via Qwen/Llama experiments that LoRA distillation boosts semantic similarity but leaves detectable behavioral differences under adversarial evaluation.
-
Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling
DDC couples confidence-weighted Bayesian early stopping with trend-aware pruning of reasoning paths, cutting inference tokens by over 10x on some benchmarks while roughly maintaining accuracy.
-
Towards an AI co-scientist
A multi-agent AI system generates novel biomedical hypotheses that show promising experimental validation in drug repurposing for leukemia, new targets for liver fibrosis, and a bacterial gene transfer mechanism.
-
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
CRITICTOOL, a benchmark of 2,740 tool-calling error scenarios built from BFCL and T-Eval with GPT-4o-based error injection, finds most LLMs rarely recover from tool-use errors, with GPT-4o best at 69.01 overall and tool-finetuned ToolLLaMA2 at 0.58.
-
A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations
A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.
-
Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines
The authors create a 1,000-query multi-modal RAG benchmark with GPT-4o-based metrics, show multi-stage generation beats single-stage, and report fine-tuned 7B-8B models beating GPT-4o only in the single-stage comparison.
-
Large Language Models Can Self-Improve in Long-context Reasoning
LLMs can improve their own long-context reasoning by fine-tuning on outputs selected by Minimum Bayes Risk consensus, without external annotations.