Pith. sign in

REVIEW 5 cited by

AST-T5: Structure-Aware Pretraining for Code Generation and Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.03003 v4 pith:3YV4O7QG submitted 2024-01-05 cs.SE cs.CLcs.LG

Linyuan Gong , Mostafa Elhoushi , Alvin Cheung This is my paper
classification cs.SEcs.CLcs.LG
keywords codeast-t5tasksast-awarecode-relatedexactgenerationllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have made significant advancements in code-related tasks, yet many LLMs treat code as simple sequences, neglecting its structured nature. We introduce AST-T5, a novel pretraining paradigm that leverages the Abstract Syntax Tree (AST) for enhanced code generation, transpilation, and understanding. Using dynamic programming, our AST-Aware Segmentation retains code structure, while our AST-Aware Span Corruption objective equips the model to reconstruct various code structures. Unlike other models, AST-T5 avoids intricate program analyses or architectural changes, so it integrates seamlessly with any encoder-decoder Transformer. Evaluations show that AST-T5 consistently outperforms similar-sized LMs across various code-related tasks. Structure-awareness makes AST-T5 particularly powerful in code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in CodeXGLUE. Our code and model are publicly available at https://github.com/gonglinyuan/ast_t5.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structure-Aware Fill-in-the-Middle Pretraining for Code

    cs.CL 2025-05 conditional novelty 7.0 of 10

    AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.

  2. SGA: Plug&Play Geometric Verification for Educational Video Synthesis

    cs.AI 2026-07 conditional novelty 6.0 of 10

    An intercept-and-refine agent that symbolically checks Manim animation code for geometric collisions improves the authors' rendering-free MVQS layout score in 7 of 8 LLM×pipeline configurations, with the metric unvali...

  3. Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Mid-training on function-aware fill-in-the-middle data improves SWE-Bench by 2.8-5.4 points and reduces post-training capability loss.

  4. JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.

  5. Position Paper: Programming Language Techniques for Bridging LLM Code Generation Semantic Gaps

    cs.SE 2025-07 unverdicted novelty 2.0 of 10

    A position paper arguing that PL techniques, especially formal verification and structure-aware representations, should be deeply integrated into LLM code generation.

Pith tools