REVIEW 5 cited by
Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging. Our study assesses LLMs' proficiency in structuring tables and introduces a novel fine-tuning method, cognizant of data structures, to bolster their performance. We unveil Struc-Bench, a comprehensive benchmark featuring prominent LLMs (GPT-NeoX-20B, GPT-3.5, GPT-4, and Vicuna), which spans text tables, HTML, and LaTeX formats. Our proposed FormatCoT aids in crafting format-specific instructions from the intended outputs to populate this benchmark. Addressing the gap in task-centered evaluation, we propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score), to more accurately gauge LLM performance. Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains, outshining its LLM counterparts across most measures. In-depth error analysis and creating an ability map across six dimensions -- coverage, formatting, reasoning, comprehension, pragmatics, and hallucination -- highlight areas for future enhancements and suggest forthcoming research trajectories. Our code and models can be found at https://github.com/gersteinlab/Struc-Bench.
Forward citations
Cited by 5 Pith papers
-
Towards Efficient and Effective Alignment of Large Language Models
A thesis presenting Lion, WebR, LTE, BMC, and FollowBench, five empirical methods that together address LLM alignment data, training, and evaluation.
-
Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation
A new reflection-based AI method makes LLM-generated virtual students predict real students' future quiz performance better than deep learning knowledge-tracing baselines.
-
ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks
FormatBench evaluates LLM format adherence across ten tasks, and REFF uses format-checker rewards in PPO to raise format faithfulness substantially while keeping content quality roughly stable.
-
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios
RuleArena evaluates LLMs on realistic rule-guided reasoning and finds that even o1-preview solves only about half of the easiest problems and near zero of the hardest.
-
A Layered Architecture for Developing and Enhancing Capabilities in Large Language Model-based Software Systems
A layered architecture with model, inference, and application layers, plus a capability-mapping process, guides where to implement features like structured output and domain knowledge in LLM systems.
Discussion (0). Continue with ORCID to comment.