REVIEW 5 cited by
Self-Evaluation Improves Selective Generation in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Safe deployment of large language models (LLMs) may benefit from a reliable method for assessing their generated content to determine when to abstain or to selectively generate. While likelihood-based metrics such as perplexity are widely employed, recent research has demonstrated the limitations of using sequence-level probability estimates given by LLMs as reliable indicators of generation quality. Conversely, LLMs have demonstrated strong calibration at the token level, particularly when it comes to choosing correct answers in multiple-choice questions or evaluating true/false statements. In this work, we reformulate open-ended generation tasks into token-level prediction tasks, and leverage LLMs' superior calibration at the token level. We instruct an LLM to self-evaluate its answers, employing either a multi-way comparison or a point-wise evaluation approach, with the option to include a ``None of the above'' option to express the model's uncertainty explicitly. We benchmark a range of scoring methods based on self-evaluation and evaluate their performance in selective generation using TruthfulQA and TL;DR. Through experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based scores not only improve accuracy, but also correlate better with the overall quality of generated content.
Forward citations
Cited by 5 Pith papers
-
FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
A multimodal RAG system with point-in-time retrieval and calibrated abstention is presented, with only simulated evidence that refusal reduces selective error and drawdown.
-
AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows
A CI-integrated evaluator whose separation between executor and grader turns an agent's self-corrections from hidden repairs into auditable first-attempt failures.
-
HalluField: Detecting LLM Hallucinations via Field-Theoretic Modeling
HalluField flags LLM hallucinations using a hand-weighted temperature-perturbation of token-level 'free energy' (negative log-likelihood) and Shannon entropy.
-
Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities
Product-of-token-probabilities with Platt or isotonic rescaling is a strong, cheap confidence signal for text-to-SQL, beating minimum-token pooling but matching or losing to self-check on large Llama models.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
Discussion (0). Continue with ORCID to comment.