Pith. sign in

REVIEW 3 cited by

Evaluating Large Language Models on Controlled Generation Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.14542 v1 pith:673VRSFZ submitted 2023-10-23 cs.CL

Jiao Sun , Yufei Tian , Wangchunshu Zhou , Nan Xu , Qian Hu , Rahul Gupta , John Frederick Wieting , Nanyun Peng
show 1 more author
Xuezhe Ma
This is my paper · ORCID
classification cs.CL
keywords modelslanguagelargegenerationtasksbenchmarkincludingsmaller
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While recent studies have looked into the abilities of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc, there have been few studies looking into the controllability of large language models on generation tasks. We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models. We conclude that **large language models struggle at meeting fine-grained hard constraints**.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines

    cs.CL 2025-06 conditional novelty 7.0 of 10

    LongGuide automatically learns task-specific quality and length guidelines from small training sets, significantly improving LLM long-form generation.

  2. Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation

    cs.SE 2025-01 conditional novelty 6.0 of 10

    Sphinx, a multi-dimensional UI-navigation benchmark, shows current foundation models score low on goal understanding, planning, grounding, instruction following, and end-to-end success, with zero successes on industri...

  3. LCTG Bench: LLM Controlled Text Generation Benchmark

    cs.CL 2025-01 conditional novelty 5.0 of 10

    LCTG Bench is a new Japanese benchmark that scores LLM controllability on format, character count, keyword, and prohibited-word constraints across summarization, ad text, and pros/cons generation.

Pith tools