REVIEW 8 cited by
Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Despite recent efforts to develop large language models with robust long-context capabilities, the lack of long-context benchmarks means that relatively little is known about their performance. To alleviate this gap, in this paper, we propose \textbf{Counting-Stars}, a multi-evidence, position-aware, and scalable benchmark designed to evaluate the multi-evidence retrieval capabilities of long-context LLMs. \textbf{Counting-Stars} comprises two counting-based multiple pieces of evidence retrieval sub-tasks: searching and reasoning. Using Counting-Stars, we conduct experiments to evaluate several long-context LLMs, including GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1. Extensive experimental results demonstrate that Gemini 1.5 Pro achieves the best overall results, while GPT-4 Turbo exhibits the most stable performance across various tasks. Furthermore, our analysis of these LLMs, which have been extended to handle long-context scenarios, indicates that significant room for improvement remains as the length of the input context and the complexity of the tasks increase.
Forward citations
Cited by 8 Pith papers
-
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Ref-Long is a new long-context referencing benchmark on which all 13 tested LCLMs perform poorly, revealing a capability gap that simple retrieval benchmarks miss.
-
Minerva: A Programmable Memory Test Benchmark for Language Models
Minerva is a programmable memory-test benchmark showing that LLMs at 4k tokens perform well on search but drop sharply on editing, counting, state tracking, and composite tasks, revealing that retrieval ability does n...
-
MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning
MiMoTable is a real-world spreadsheet benchmark with 1,719 bilingual question-answer pairs and a meta-operation difficulty criterion on which the best LLM scores 77.4%.
-
NeMo: Needle in a Montage for Video-Language Understanding
NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.
-
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
MiniLongBench, a 237-sample compression of LongBench, is claimed to reproduce model rankings with a 0.97 Spearman correlation at 4.5% of the evaluation cost.
-
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.
-
Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale
FGO is a divide-and-conquer framework that runs LLM-based agent optimization on dataset subsets and recursively merges the optimized prompts or tool sets into a final agent system.
-
Federated In-Context LLM Agent Learning
FICAL shares tool-usage text summaries instead of model weights in federated LLM agent learning, reporting similar accuracy with far less communication, but the evidence is incomplete.
Discussion (0). Continue with ORCID to comment.