Pith. sign in

REVIEW 12 cited by

S*: Test Time Scaling for Code Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14382 v1 pith:2ALQ5Z2I submitted 2025-02-20 cs.LG cs.AI

Dacheng Li , Shiyi Cao , Chengkun Cao , Xiuyu Li , Shangyin Tan , Kurt Keutzer , Jiarong Xing , Joseph E. Gonzalez
show 1 more author
Ion Stoica
This is my paper
classification cs.LGcs.AI
keywords codemodelsscalingacrossmodelreasoningfurthergeneration
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S* extends the existing parallel scaling paradigm with sequential scaling to push performance boundaries. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions. We evaluate across 12 Large Language Models and Large Reasoning Model and show: (1) S* consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables non-reasoning models to surpass reasoning models - GPT-4o-mini with S* outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be available under https://github.com/NovaSky-AI/SkyThought.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Iterative Test-and-Repair Framework for Competitive Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    FixAudit improves LLM code generation on competitive programming benchmarks by training a shared model for iterative code-aware test generation and repair, achieving 35%+ gains in Pass@1 over baselines on the same 7B model.

  2. SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

    cs.DC 2026-07 accept novelty 6.5 of 10

    SiFAR cuts All-Reduce latency up to 52% and end-to-end decode throughput up to 18.6% at TP=8 by dual buffering, in-switch redundant pull, and speculative reduction with validation.

  3. PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.

  4. DeliCIR: Memory-Guided Test-Time Deliberation via Multi-Agent Collaboration for Composed Image Retrieval

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Proposes PDF, a hierarchical multi-agent Perception-to-Deliberation Framework that adds experience self-evolution and test-time scaling to composed image retrieval, claiming SOTA on CIRR, CIRCO, and FashionIQ.

  5. CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer

    cs.SE 2025-10 conditional novelty 6.0 of 10

    Using Python-generated test oracles and multi-temperature sampling, CodeChemist selects low-resource-language code by execution pass rate and improves Pass@1 up to 69.5% over vanilla sampling.

  6. SSRL: Self-Search Reinforcement Learning

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.

  7. Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

    cs.SE 2025-06 accept novelty 6.0 of 10

    An empirical study of five LLM-based program repair agents shows they excel on simple fixes but overfit tests and struggle with test generation, regression selection, and advanced tooling.

  8. ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation

    cs.AR 2025-06 conditional novelty 6.0 of 10

    A reasoning LLM fine-tuned on 3.5B tokens of RTL chain-of-thought data, with iterative test-time self-correction, sets new state-of-the-art results on VerilogEval and RTLLM.

  9. Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.

  10. AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A stochastic scheduling framework that modulates slow-to-fast reasoning in large reasoning models at test time, improving accuracy while reducing token usage.

  11. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling

    cs.SE 2025-07 conditional novelty 5.0 of 10

    Trae Agent combines parallel patch generation, hierarchical pruning, and agent-based majority-vote selection to reach 75.20% Pass@1 on SWE-bench Verified, the current leaderboard leader.

  12. First Finish Search: Efficient Test-Time Scaling in Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    First Finish Search launches n parallel reasoning traces and returns the shortest one, improving accuracy on AIME benchmarks while cutting token use.

Pith tools