Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

LoCoBench claims long-context coding models still stumble on real software tasks

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:45 UTC pith:HU3CHMDJ

load-bearing objection LoCoBench is a substantial new benchmark resource, but its reported scores are arithmetically impossible and its headline degradation claim contradicts its own tables. the 4 major comments →

arxiv 2509.09614 v1 pith:HU3CHMDJ submitted 2025-09-11 cs.SE cs.AI

LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering

Jielin Qiu , Zuxin Liu , Zhiwei Liu , Rithesh Murthy , Jianguo Zhang , Haolin Chen , Shiyu Wang , Ming Zhu
show 9 more authors
Liangwei Yang Juntao Tan Zhepeng Cen Cheng Qian Shelby Heinecke Weiran Yao Silvio Savarese Caiming Xiong Huan Wang
This is my paper
classification cs.SE cs.AI
keywords long-context LLMssoftware engineering benchmarkcode generation evaluationmulti-file reasoningarchitectural coherencecontext windowLoCoBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LoCoBench, a benchmark with 8,000 evaluation scenarios spanning 10 programming languages and context lengths from 10K to 1M tokens. It argues that existing code benchmarks test short, isolated tasks, so they miss the long-context reasoning that real software engineering requires. LoCoBench covers eight task categories, from architectural understanding to security analysis, and scores models with 17 metrics including six new ones. The authors evaluate current long-context models and find that even the best ones score near the middle of the 0–5 LoCoBench scale, especially on expert-level, long-context scenarios. Their central claim is that long-context understanding in software development remains an unsolved problem that needs dedicated evaluation and improvement.

Core claim

The paper presents LoCoBench as a large, systematically generated benchmark for evaluating long-context language models on complex software engineering tasks. It demonstrates that current state-of-the-art models achieve only moderate LoCoBench Scores, with the top model scoring 2.312 out of 5, and that performance declines as task difficulty and context length increase. The benchmark's design includes a 5-phase pipeline that generates project specifications, synthetic codebases, evaluation scenarios, validation, and model scoring, with 8,000 scenarios across 10 languages and 36 domains. The authors argue this fills a gap left by benchmarks that test isolated code completion or bug fixing, an

What carries the argument

The central mechanism is the LoCoBench evaluation framework, which combines 17 metrics across four dimensions: Software Engineering Excellence (8 metrics), Functional Correctness (4 metrics), Code Quality Assessment (3 metrics), and Long-Context Utilization (2 metrics). The unified LoCoBench Score (LCBS) is a weighted linear combination of these dimensions, with weights 0.4, 0.3, 0.2, and 0.1, scaled to a 0–5 range. The new metrics, including Architectural Coherence Score, Dependency Traversal Accuracy, Cross-File Reasoning Depth, Incremental Development Capability, Information Coverage Utilization, and Multi-Session Memory Retention, are formally defined with equations, though the paper doe

Load-bearing premise

The benchmark's validity depends on the assumption that the new metrics (ACS, DTA, CFRD, ICU, MMR, IDC) can be reliably computed from model outputs, but the paper does not specify how the terms such as pattern adherence, contextual awareness, reasoning depth, and information retention are measured or who or what judges them.

What would settle it

Replicating the evaluation on a small sample of LoCoBench scenarios with human expert judges and comparing their scores to the automated metric scores would show whether the new metrics capture what they claim. If human and automated scores disagree substantially, the benchmark's measurements would not be trustworthy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If model scores on LoCoBench are valid, current long-context models are far from ready for enterprise-scale software engineering, with the best score at 2.312 out of 5.
  • The 100x context length variation (10K to 1M tokens) gives a way to measure where performance degradation begins for each model.
  • The task categories provide a taxonomy for future benchmarks to test long-context abilities beyond code completion.
  • The multi-session and architectural coherence metrics point toward evaluating models as persistent development agents, not just code generators.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's definition of the LCBS weights as 'empirically determined' without showing the procedure means a reader cannot verify whether the overall ranking would change under different weights.
  • Because the new metrics rely on terms like pattern adherence and contextual awareness that lack measurement protocols, the headline scores may be sensitive to how those terms are operationalized in practice.
  • The reliance on LLM-generated codebases and expected approaches means the benchmark measures performance on synthetic, not real, software; connecting this to real-world GitHub repositories could be a useful extension.
  • The reported success rates near 100% alongside low LCBS values suggest the score may be dominated by the subjective software engineering metrics rather than objective compilation or test results.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LoCoBench, a synthetic benchmark for evaluating long-context LLMs on complex software-engineering tasks. The authors describe a five-phase pipeline that generates 8,000 scenarios across 10 programming languages, with context lengths from 10K to 1M tokens, eight task categories, and 17 metrics aggregated into a LoCoBench Score (LCBS). They report experiments on 13 models and claim that state-of-the-art long-context models exhibit substantial performance gaps and clear degradation as context length and difficulty increase. The paper also releases six new metrics (ACS, DTA, CFRD, IDC, ICU, MMR) and an associated GitHub repository.

Significance. If the benchmark and its scoring pipeline were sound, LoCoBench would be a valuable resource: it offers a scale and language/task diversity that most existing long-context code benchmarks lack, and the idea of measuring architectural coherence, dependency traversal, and multi-session memory is genuinely useful for the community. The paper also deserves credit for attempting to make its evaluation framework systematic and for shipping code. However, the reported numerical results are internally inconsistent, the new metrics are not operationalized, and the central degradation claim is contradicted by the paper's own tables. As presented, the benchmark cannot be used as a measurement instrument, so the significance is currently prospective rather than actual.

major comments (4)
  1. [§5.2–§5.8 and Tables 11, 12, 14, 16] The reported LCBS values are arithmetically incompatible with Equation (18). For example, in Table 11 (Task Category), Gemini-2.5-Pro has SE=0.375, FC=0.356, CQ=0.768, and LCU=0.523 in every row, yet the stated LCBS totals vary from 2.272 to 2.421. A weighted sum of identical components is constant; no normalization or rounding can produce these different totals. The same pattern occurs in Tables 12, 14, and 16. Therefore the tables cannot have been generated by the stated scoring formula, and every conclusion drawn from those scores—including the claimed performance gaps—is unsupported.
  2. [§4.2–§4.5, Equations (1)–(6)] The new metrics are not measurable as defined. Equation (1) uses α(p,C), κ(p), and w(p); Equation (2) uses μ(d_ij), γ(d_ij,G), δ(d_ij); Equation (3) uses ρ and ι; Equation (4) uses ξ, σ, β; Equation (5) uses τ and φ; Equation (6) uses ψ and χ. The paper never states how any of these quantities are computed from model outputs, which model or judge fills them in, or what the reference standard is. Similarly, the LCBS weights in Equation (16) are described as “empirically determined” with no procedure shown. Without an operational definition, these scores carry no empirical content and cannot be reproduced or verified.
  3. [§5.6 and Figure 9; Appendix B.5.2] The central degradation claim is contradicted by the paper's own data. Figure 9 (upper left) reports mean LCBS increasing with difficulty: Easy 2.201, Medium 2.212, Hard 2.231, Expert 2.236, while the text in §5.6 states there is “clear performance degradation patterns as difficulty increases.” Additionally, Appendix B.5.2 reports Easy avg_lcbs=3.92 and Expert avg_lcbs=2.18 for GPT-4o, whereas Table 8 gives GPT-4o an overall LCBS of 2.073 and Table 7 gives a success rate near 100%. These numbers cannot all come from the same evaluation run, and the contradiction undermines the paper’s headline finding of long-context degradation.
  4. [§3.2, Phases 2–3; Appendix B.4] The benchmark’s ground truth and codebases are generated by LLMs without reported human validation. Phase 2 uses LLMs to write complete codebases, and Phase 3 uses LLMs to write “expected approach” answers and ground truths. Appendix B.4 describes validation as “purely automated” and reports no human spot-check, inter-annotator agreement, or qualitative audit. This makes the benchmark circular: it evaluates models against standards created by other models. At minimum, the authors should provide evidence that a sample of scenarios and expected approaches is correct, unambiguous, and solvable, e.g., human expert judgment and pilot runs with independent testers.
minor comments (5)
  1. [Abstract and §3.7] The abstract claims “8 new evaluation metrics” several times, while Table 6 and §3.7 state “6 new metrics.” Please reconcile the count.
  2. [Figure 5] The caption says “across 10 evaluation dimensions,” but the figure appears to show 8 task categories plus 2 long-context scores. Please clarify the axis labels and the definition of “dimension.”
  3. [Tables 7, 8, 9] Several table cells lack clear delimiters, e.g., Table 7(d) runs category names together and Table 8(a)–(c) has merged row labels. This makes the tables hard to verify and should be fixed if the paper is revised.
  4. [Appendix B.5.2] Model names are inconsistent (e.g., “Claude-4-Sonnet” vs. “claudesonnet4” vs. “Claude-Sonnet-4”), and the performance numbers in this appendix do not match the main tables.
  5. [References] Some references have formatting errors, such as “OW ASP” / “OWASP” and “rebench Team” as an author name. Please correct these.

Circularity Check

3 steps flagged

LoCoBench's central empirical claims are not derived from its own measurement pipeline: LLM-generated ground truth, definitionally decaying metrics, and arithmetically impossible LCBS totals.

specific steps
  1. other [Section 4.6, Eq. (18); Tables 11(a), 11(c)-(f)]
    "LCBS= 5·(0.4·SE+0.3·F C+0.2·CQ+0.1·LCU) (18); Table 11(a): Gemini-2.5-Pro 2.338 ... 2.272 ... 2.421 ...; Table 11(c)-(f): Gemini-2.5-Pro 0.375, 0.356, 0.768, 0.523 repeated identically in every task-category column."

    Under Eq. (18), LCBS is a fixed linear combination of the four reported dimension scores. Tables 11(c)-(f) give the same SE, FC, CQ, and LCU for a given model in every task category, so Eq. (18) forces a single LCBS value per model. The reported totals instead vary substantially (e.g., Gemini-2.5-Pro: 2.338, 2.272, 2.421). The published totals therefore cannot be outputs of the stated metric pipeline. The claimed 'substantial performance gaps' and 'performance degradation patterns' are not derived from the measured dimensions; they are unexplained inputs to the table, so the paper's headline empirical conclusion is not a consequence of its own scoring equations.

  2. self definitional [Appendix B.6.1 (Master Scenario Generation Template); Appendix B.3.1 (Integration Testing example)]
    "expected_approach: 'How an expert developer would approach this task', ground_truth: 'Expected solution or key insights' (B.6.1); expected_approach: 'An expert would recognize this as testing integration between Lambda execution environment and failure handling mechanism' (B.3.1)."

    Phase 3 builds each scenario's correctness criterion from an LLM prompt that instructs the generator to supply the 'expected_approach' and 'ground_truth'. Phase 5 then scores LLM outputs against these same LLM-written criteria; the paper reports no human validation of the expected approaches. Thus 'correct' is defined as agreement with the generating model's expected solution. Measuring LLMs against LLM-authored ground truth makes the performance result self-referential: the benchmark's 'substantial performance gaps' are a statement about agreement with its own generator, not an independent, externally grounded measurement of software-engineering ability.

  3. self definitional [Section 4.5, Eq. (6)]
    "MMR(S) = 1/|S| Σ_j ψ(s_j,C_{j-1})·χ(s_j,C_j) / log(j+1), where ... the logarithmic decay term log(j+1) models expected memory degradation over temporal distance."

    The Long-Context Utilization dimension that feeds LCBS includes MMR, which contains an explicit log(j+1) decay in its definition. Any decrease in MMR over sessions is therefore guaranteed by the formula, independent of actual model behavior. The paper's later conclusion that long-context models show 'performance degradation patterns' is partly baked into the metric: the metric assumes the very degradation phenomenon it is used to discover, rather than measuring it from model outputs alone.

full rationale

The paper's central empirical claims—that LoCoBench quantifies long-context degradation and reveals substantial performance gaps—are not supported by a self-contained derivation chain. The benchmark defines correctness through LLM-generated 'expected_approach' fields, so the evaluation loop is self-referential: LLM outputs are scored against other LLM outputs, with no human or external oracle reported. The new long-context metrics are non-operational: Eqs. (1)-(6) are filled with undefined quantities (pattern adherence α, contextual awareness γ, reasoning depth ρ, etc.), and no judge or automated measurement procedure is specified, so the scores have no independently reproducible empirical content. Most damaging, Tables 11(c)-(f) hold all four dimension scores constant across task categories while Table 11(a) reports varying LCBS totals, which is arithmetically incompatible with Eq. (18). The reported totals therefore cannot have been generated by the stated methodology, invalidating the derived rankings and degradation analyses. No load-bearing self-citation chain was found; the circularity is internal to the benchmark's construction and scoring, not imported from prior work. Because the headline result reduces to self-referential ground truth and a scoring pipeline whose published numbers contradict its own formula, a score of 6 is appropriate: one or more 'predictions' are effectively constructed by the benchmark's definitions and tables rather than derived from independent measurements.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The benchmark's validity rests on inputs that the paper neither derives nor validates: (1) the LCBS aggregation weights, stated as 'empirically determined' without a fitting procedure; (2) the quantities inside the six new metric formulas, which are never operationalized; (3) the assumption that LLM-generated codebases and expected approaches constitute valid ground truth; and (4) the assumption that automated compilation and complexity checks guarantee scenario quality. The internal inconsistency of the result tables additionally means the published numbers do not follow from the published formulas.

free parameters (3)
  • LCBS dimension weights w=[0.4,0.3,0.2,0.1] = 0.4 / 0.3 / 0.2 / 0.1
    Equation (16) states the weights are 'empirically determined' to reflect relative importance, but no fitting procedure or data is shown.
  • Metric-internal quantities in Eqs. 1-6 (alpha, kappa, mu, gamma, delta, rho, iota, xi, sigma, beta, tau, phi, psi, chi, = undefined
    Equations (1) to (6) define the new metrics through these terms without operational definitions or measurement procedures; they act as free parameters that determine every model score.
  • Difficulty thresholds and coverage target = 10K-100K, 100K-200K, 200K-500K, 500K-1M tokens; coverage > 0.7
    Section 3.4 fixes the four ranges by design; Section 3.5 filters scenarios until complexity and coverage targets are met, making the difficulty distribution a constructed outcome.
axioms (4)
  • domain assumption LLM-generated codebases and scenarios are valid evaluation material
    Phases 2 and 3 synthesize all projects and tasks with LLMs; validity is asserted by compilation checks, not by human review of the tasks or expected approaches.
  • domain assumption The six new metrics are computable from model outputs
    Section 4.2 through 4.5 introduces them; no computation procedure, judge, or calibration standard is given.
  • domain assumption Automated validation ensures scenario quality without LLM involvement
    Section 3.5 and Phase 4 state validation is 'purely automated... no LLM involvement to prevent bias'; this assumes compilation, tests, and complexity metrics catch the relevant flaws.
  • domain assumption Subjective quality metrics can be scored from classic references
    Table 6 maps STS, RS, CS, IS, SES, and SAS to ISO 25010, Kan, Glass, Buse and Weimer, and OWASP, but the paper does not describe how each reference is translated into a numeric score.
invented entities (2)
  • New long-context metrics: ACS, DTA, CFRD, IDC, ICU, MMR no independent evidence
    purpose: Measure architectural coherence, dependency traversal, cross-file reasoning, incremental development, information coverage, and memory retention; they drive most of the LCBS weight.
    No calibration against human judgment or known model behavior is provided; their values are determined by undefined terms, so they are the paper's own constructs without external support.
  • LoCoBench Score (LCBS) no independent evidence
    purpose: Single 0-5 aggregate for ranking models.
    Weighted average of four dimension scores with hand-set weights; no calibration set, no comparison to existing benchmarks, and the reported totals do not reproduce from the stated formula and component scores.

pith-pipeline@v1.3.0-alltime-deepseek · 52211 in / 21283 out tokens · 205532 ms · 2026-08-04T18:45:33.516709+00:00 · methodology

0 comments
read the original abstract

The emergence of long-context language models with context windows extending to millions of tokens has created new opportunities for sophisticated code understanding and software development evaluation. We propose LoCoBench, a comprehensive benchmark specifically designed to evaluate long-context LLMs in realistic, complex software development scenarios. Unlike existing code evaluation benchmarks that focus on single-function completion or short-context tasks, LoCoBench addresses the critical evaluation gap for long-context capabilities that require understanding entire codebases, reasoning across multiple files, and maintaining architectural consistency across large-scale software systems. Our benchmark provides 8,000 evaluation scenarios systematically generated across 10 programming languages, with context lengths spanning 10K to 1M tokens, a 100x variation that enables precise assessment of long-context performance degradation in realistic software development settings. LoCoBench introduces 8 task categories that capture essential long-context capabilities: architectural understanding, cross-file refactoring, multi-session development, bug investigation, feature implementation, code comprehension, integration testing, and security analysis. Through a 5-phase pipeline, we create diverse, high-quality scenarios that challenge LLMs to reason about complex codebases at unprecedented scale. We introduce a comprehensive evaluation framework with 17 metrics across 4 dimensions, including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our evaluation of state-of-the-art long-context models reveals substantial performance gaps, demonstrating that long-context understanding in complex software development represents a significant unsolved challenge that demands more attention. LoCoBench is released at: https://github.com/SalesforceAIResearch/LoCoBench.

Figures

Figures reproduced from arXiv: 2509.09614 by Caiming Xiong, Cheng Qian, Haolin Chen, Huan Wang, Jianguo Zhang, Jielin Qiu, Juntao Tan, Liangwei Yang, Ming Zhu, Rithesh Murthy, Shelby Heinecke, Shiyu Wang, Silvio Savarese, Weiran Yao, Zhepeng Cen, Zhiwei Liu, Zuxin Liu.

Figure 1
Figure 1. Figure 1: LoCoBench Pipeline Architecture. Our systematic 5-phase pipeline transforms high-level specifications [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LoCoBench Coverage Overview. Left: Programming language distribution showing equal representation (10% each) across 10 languages spanning diverse paradigms from systems programming (C, C++, Rust) to web development (JavaScript, TypeScript, PHP) to enterprise applications (Java, C#) to modern languages (Go, Python). Right: Hierarchical domain organization with 36 sub-categories grouped into 10 main categori… view at source ↗
Figure 3
Figure 3. Figure 3: Additional uniqueness factors in LoCoBench. Three independent factors provide comprehensive evaluation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LoCoBench’s evaluation projects analysis. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overall performance comparison of GPT-5, Gemini-2.5-Pro, and Claude-Sonnet-4 across 10 LoCoBench [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Model ranking and long-context utilization comparison. Left chart shows LoCoBench Score (LCBS) [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Programming language performance heatmap showing model performance across 10 programming [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Task category performance analysis. Top chart shows performance distribution across all models for each [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Context length and difficulty impact analysis. Upper left shows performance distribution by difficulty [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Domain specialization and performance analysis. Top chart shows model performance trends across 10 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Architecture pattern performance analysis. Top chart shows model performance trajectories across 10 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

    cs.SE 2026-06 unverdicted novelty 7.0

    StaminaBench evaluates coding agents over 100 procedurally generated change requests to a REST API, finding that tested models fail within 5-6 turns without feedback but improve up to 12x with test feedback and good h...

  2. LLM4CAD-Editor: An Intent-Aware Large Language Model Framework for Multi-Level Computer-Aided Design Editing

    cs.HC 2026-05 unverdicted novelty 7.0

    LLM4CAD-Editor introduces a feature-name DSL and 32B-parameter fine-tuned model that achieves 96.3% parsing accuracy on parameter edits and 82% intent satisfaction on functional edits, with 1.4x better robustness than...

  3. Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems

    cs.MA 2026-05 unverdicted novelty 6.0

    Meta-Team is a collaborative self-evolution framework that turns multi-agent execution experience into reusable improvements at agent, coordination, and team levels, outperforming baselines on six benchmarks.

  4. Teaching LLMs Program Semantics via Symbolic Execution Traces

    cs.SE 2026-05 unverdicted novelty 6.0

    Training Qwen3-8B on symbolic execution traces from Soteria improves violation detection in C programs by over 17 points, transfers across five property types, and shows superadditive gains with chain-of-thought.

Reference graph

Works this paper leans on

19 extracted references · 4 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Dependency Graph Analysis: Construct directed graph of file dependencies (imports, calls, inheri- tance)

  2. [2]

    Centrality Scoring: Compute PageRank and betweenness centrality to identify architecturally important files 3.Task-Specific Filtering: Apply task category filters to prioritize relevant functionality

  3. [3]

    Dependency Graph Construction: Build import/usage relationships before detailed implementation

  4. [4]

    "" def __init__(self): self._subscribers = \{\} self._async_subscribers = \{\} def subscribe(self, event_type: Type[Event], handler: EventHandler):

    Progressive Implementation: Generate files in dependency order, ensuring referential consistency 5.Integration Verification: Cross-reference validation to maintain architectural coherence 31 Real Generated Structure - Mercantilo E-commerce Suite: The Python expert-level e-commerce specification generates a complete Django monolith with 96 files: mercantil...

  5. [5]

    APPS provides 10,000 problems from coding compe- titions, while LiveCodeBench offers contamination-free evaluation with problems collected from ongoing contests

    focus on competitive programming problems. APPS provides 10,000 problems from coding compe- titions, while LiveCodeBench offers contamination-free evaluation with problems collected from ongoing contests. CodeContests (Li et al., 2022) extends this paradigm with competitive programming problems from Codeforces, AtCoder, and CodeChef. AlphaCode (Li et al.,...

    2022

  6. [6]

    While domain-specific, these benchmarks still primarily evaluate isolated function or script generation rather than comprehensive software development capabilities

    focuses on class-level code generation requiring understanding of object-oriented programming principles. While domain-specific, these benchmarks still primarily evaluate isolated function or script generation rather than comprehensive software development capabilities. Evaluation Methodology Advances:Recent work has focused on improving evaluation method...

    2020

  7. [7]

    LongICLBench (An et al., 2024) evaluates in-context learning capabilities at extreme lengths, while LongAlign (Bai et al., 2024a) addresses instruction following in long contexts

    pushing evaluation to 1M tokens. LongICLBench (An et al., 2024) evaluates in-context learning capabilities at extreme lengths, while LongAlign (Bai et al., 2024a) addresses instruction following in long contexts. BAMBOO (Dong et al., 2024b) provides comprehensive evaluation across multiple aspects of long-context understanding. Code-specific long-context ...

    2024

  8. [13]

    id": "java_api_graphql_easy_007_feature_implementation_expert_01

    Information Coverage Optimization: Balance between information completeness and context length constraints 5.Difficulty Calibration: Adjust context complexity based on target difficulty level 34 Diverse Scenario Examples Across Task Categories: Example 1 - Feature Implementation (Java GraphQL, Expert): \{ "id": "java_api_graphql_easy_007_feature_implement...

  9. [14]

    Context Preparation: Intelligent truncation for models with limited context windows using importance-based ranking 2.Prompt Engineering: Task-specific prompting strategies optimized for each model family

  10. [15]

    GPT-4o": \{

    Parallel Execution: Concurrent evaluation with configurable timeout (3600 seconds) and error recovery 4.Multi-Metric Assessment: Comprehensive scoring across all 17 evaluation metrics 5.Statistical Analysis: Confidence interval computation and significance testing B.5.2 Comprehensive Benchmark Statistics Multi-Model Evaluation Results Across Difficulty Le...

  11. [16]

    You MUST respond with valid JSON in the exact format shown below

  12. [17]

    Each file MUST contain complete, syntactically correct \{LANGUAGE\} code

  13. [18]

    Do NOT truncate your response - provide the complete solution

  14. [19]

    approach

    Use \{language_specific_best_practices\} **REQUIRED RESPONSE FORMAT **: ‘‘‘json \{ "approach": "Your solution strategy (keep under 200 words)", "files": \{ "filename1.\{ext\}": "complete file content with proper escaping", "filename2.\{ext\}": "complete file content with proper escaping" \}, "explanation": "Implementation details (keep under 300 words)" \...

    2025

  15. [2011]

    Zetian An, Chanakya Chen, Weiyan Zheng, Hendrik Geissler, Yiyang Qian, Peiyi Wang, Shuohang Chen, Tianyu Wang, Zhenguo Wu, and William Yang Wang

    Iso/iec 25010:2011 systems and software engineering – systems and software quality requirements and evaluation (square) – system and software quality models. Zetian An, Chanakya Chen, Weiyan Zheng, Hendrik Geissler, Yiyang Qian, Peiyi Wang, Shuohang Chen, Tianyu Wang, Zhenguo Wu, and William Yang Wang. 2024. Longiclbench: A comprehensive benchmark for lon...

    Pith/arXiv arXiv 2011

  16. [2022]

    InarXiv preprint arXiv:2203.13474

    Codegen: An open large language model for code with multi-turn program synthesis. InarXiv preprint arXiv:2203.13474. OpenAI, Anthropic, et al. 2024. Swe-bench-verified: A human-validated subset for more reliable code generation evaluation. InarXiv preprint. OWASP. 2021. Owasp top 10-2021: The ten most critical web application security risks. https://owasp...

    Pith/arXiv arXiv 2024

  17. [2023]

    InIEEE/ACM International Conference on Computer-Aided Design

    Verilogeval: Evaluating large language models for verilog code generation. InIEEE/ACM International Conference on Computer-Aided Design. Yanlin Wang, Lunjun Ding, Haoyu Luo, Luke Zettlemoyer, and Graham Neubig. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. InarXiv preprint arXiv:2212.10007. An Yen, Jiyuan Zhang, Yunzhi...

    Pith/arXiv arXiv 2022

  18. [2024]

    InarXiv preprint arXiv:2408.15296

    Helmet: A hierarchical lm-based evaluation method for text-to-sql. InarXiv preprint arXiv:2408.15296. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-swe-bench:...

    Pith/arXiv arXiv 2025

  19. [2025]

    unique_id

    evaluates coding LLMs at 1M context windows, demonstrating dramatic performance degradation. LongCodeU (Li et al., 2025) focuses on long code understanding across four aspects. LongCodeArena (Bo- gomolov et al., 2024) provides code-centric evaluation at 2M+ tokens. RepoQA (Liu et al., 2024) evaluates long-context code understanding through question answer...

    2025