REVIEW 5 cited by
Improving Consistency in Large Language Models through Chain of Guidance
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Consistency is a fundamental dimension of trustworthiness in Large Language Models (LLMs). For humans to be able to trust LLM-based applications, their outputs should be consistent when prompted with inputs that carry the same meaning or intent. Despite this need, there is no known mechanism to control and guide LLMs to be more consistent at inference time. In this paper, we introduce a novel alignment strategy to maximize semantic consistency in LLM outputs. Our proposal is based on Chain of Guidance (CoG), a multistep prompting technique that generates highly consistent outputs from LLMs. For closed-book question-answering (Q&A) tasks, when compared to direct prompting, the outputs generated using CoG show improved consistency. While other approaches like template-based responses and majority voting may offer alternative paths to consistency, our work focuses on exploring the potential of guided prompting. We use synthetic data sets comprised of consistent input-output pairs to fine-tune LLMs to produce consistent and correct outputs. Our fine-tuned models are more than twice as consistent compared to base models and show strong generalization capabilities by producing consistent outputs over datasets not used in the fine-tuning process.
Forward citations
Cited by 5 Pith papers
-
Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
Models flip between correct and incorrect answers on over 23% of questions under meaning-preserving paraphrases, so single-prompt accuracy overstates reliable knowledge.
-
Leveraging LLMs for Automated Translation of Legacy Code: A Case Study on PL/SQL to Java Transformation
A case study with 10 PL/SQL to Java pairs finds that providing an LLM a domain model and similar examples can help translate legacy code, but the evidence is small-scale and mixed.
-
Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey
The paper surveys existing work on LLM meta-thinking and argues that multi-agent reinforcement learning is a promising missing ingredient for building self-correcting language models.
-
Transforming Expert Knowledge into Scalable Ontology via Large Language Models
An LLM-based taxonomy alignment framework reaches 0.97 F1 using many-shot prompting and expert calibration, but the claimed superiority over the 0.68 human benchmark is based on a non-comparable baseline.
-
Consistency in Language Models: Current Landscape, Challenges, and Future Directions
This review of 2019-2025 research on language model consistency concludes that the field needs standardized definitions, comprehensive multilingual benchmarks, and mixed human-automatic evaluation.
Discussion (0). Continue with ORCID to comment.