REVIEW 5 cited by
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench.
Forward citations
Cited by 5 Pith papers
-
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.
-
How Many Instructions Can LLMs Follow at Once?
IFScale measures instruction-following at densities from 10 to 500 constraints and finds that even top frontier models satisfy only about two-thirds of 500 simultaneous keyword instructions.
-
A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback subs...
-
QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.
-
Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
A lightweight grid-world benchmark shows that LLM agents pay a task-performance cost for following safety principles and that high adherence can mask inability rather than principled choice.
Discussion (0). Sign in to comment.