REVIEW 8 cited by
Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) are expected to be predictable and trustworthy to support reliable decision-making systems. Yet current LLMs often show inconsistencies in their judgments. In this work, we examine logical preference consistency as a foundational requirement for building more dependable LLM systems, ensuring stable and coherent decision-making while minimizing erratic or contradictory outputs. To quantify the logical preference consistency, we propose a universal evaluation framework based on three fundamental properties: transitivity, commutativity and negation invariance. Through extensive experimentation across diverse LLMs, we demonstrate that these properties serve as strong indicators of judgment robustness. Furthermore, we introduce a data refinement and augmentation technique, REPAIR, that enhances logical consistency while maintaining alignment with human preferences. Finally, we show that improving consistency leads to better performance in LLM-driven logic-based algorithms, reinforcing stability and coherence in decision-making systems.
Forward citations
Cited by 8 Pith papers
-
SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
SharedRep-RLHF learns a shared preference representation across groups to improve worst-case reward estimates for minority annotators, but the theoretical guarantees are undermined by proof errors.
-
A Design Space for the Critical Validation of LLM-Generated Tabular Data
A design space spanning analysis granularity and data source organizes existing approaches for critically validating LLM-generated tabular data and reveals unexplored combinations.
-
Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field
Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.
-
Voting with the Graph: Stable RLAIF via Topological Consistency Maximization
Removing minimum feedback arc sets from LLM-judge preference graphs yields small benchmark gains in RLAIF, but claims outrun the evidence.
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
-
Logical Reasoning in Large Language Models: A Survey
A survey of logical reasoning in large language models that organizes benchmarks, evaluations, and enhancement methods around formal and symbolic logic.
-
Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey
A literature review cataloging LLM-based augmentation methods across image, text, and speech, with a taxonomy of techniques, limitations, and suggested fixes.
-
A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods
A narrative review of LLM knowledge integration that categorizes techniques and compiles benchmarks, but lacks a systematic method and contains unreliable citations.
Discussion (0). Continue with ORCID to comment.