REVIEW 7 cited by
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
read the original abstract
It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens when both models operate under the same budget? (e.g., compute, run-time). To address this question, we analyze code generation LLMs of various sizes and make comparisons such as running a 70B model once vs. generating five outputs from a 13B model. We consider a standard unit-test setup, which can be used to select the correct output from the smaller model. Our findings reveal that the repeated use of smaller models can yield consistent improvements, with gains of up to 15% across five tasks. On the other hand, in scenarios where unit-tests are unavailable, a ranking-based selection of candidates from the smaller model falls short of the performance of a single output from larger ones. Our results highlight the potential of using smaller models instead of larger ones, and the importance of studying approaches for ranking LLM outputs.
Forward citations
Cited by 7 Pith papers
-
Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning
CoT-PoT ensembling achieves self-consistency accuracy in LLMs with only two samples for 78.6% of tasks, reducing computation by 9.3x compared to standard methods.
-
Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning
Cross-modal agreement between chain-of-thought and program-of-thought reasoning enables self-consistency with only two LLM samples, reducing sampling cost by 9.3x while improving accuracy.
-
Acoda: Adversarial Code Obfuscation for Defending against LLM-based Analysis
Acoda uses a genetic algorithm to optimize eight obfuscation methods that reduce LLM code analysis success rates to as low as 30% while preserving original semantics.
-
Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
A Lagrangian-relaxation plus imitation-learning pipeline adaptively allocates test-time compute to LLMs, outperforming uniform baselines by up to 12.8% relative accuracy on MATH while staying within a fixed average budget.
-
CodeChemist: Functional Knowledge Transfer for Low-Resource Code Generation via Test-Time Scaling
Using Python-generated test oracles and multi-temperature sampling, CodeChemist selects low-resource-language code by execution pass rate and improves Pass@1 up to 69.5% over vanilla sampling.
-
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Repeated sampling scales problem coverage log-linearly with sample count, improving SWE-bench Lite performance from 15.9% to 56% using 250 samples.
-
Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomy
A 7-billion-parameter coding model with multi-sampling matches a 32-billion-parameter model on repository-level completion under the same time budget, and compiler-error feedback improves pass rates across all models tested.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.