Pith. sign in

REVIEW 4 major objections 5 minor 60 references

A 7-billion-parameter model, sampled repeatedly and corrected by compiler errors, can match a 32-billion-parameter model while using a quarter of the compute.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:13 UTC pith:ADQRRSUZ

load-bearing objection Useful empirical study on SLM-vs-LLM code generation, but the headline result rests on a weak similarity metric and the paper contains an unattributed verbatim block. the 4 major comments →

arxiv 2607.21677 v1 pith:ADQRRSUZ submitted 2026-07-23 cs.SE cs.AI

Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomy

Elisa Chiarotto , Jingbo Li , P. Chris Broekema , Rob V. van Nieuwpoort This is my paper
classification cs.SE cs.AI
keywords small language modelsmulti-sampling generationcompiler feedbackagentic AIcode completionretrieval augmented generationsustainable computingradio astronomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that small language models can be made competitive with much larger ones for code generation without extra energy cost, by spending the compute budget on many attempts instead of one big model. On repository-level code completion, generating several completions from a 7-billion-parameter model and picking the one with the highest log probability matched or beat a single completion from a 32-billion-parameter model under the same time budget, using one GPU instead of four. In a separate agentic pipeline, feeding compiler error messages back to the model as feedback raised pass rates by at least three to four percentage points on function-level code generation, across model families and sizes. The motivation is concrete: an upgraded radio telescope is expected to need forty times more processing, and the optimization process itself must stay within an energy budget. The paper's quality measure for the completion experiment is character n-gram cosine similarity to a reference solution, not execution-based correctness.

Core claim

The central claim is that two cheap mechanisms—repeated sampling with log-probability selection, and a compiler-in-the-loop revision cycle—can close most of the gap between small and large code models. The paper reports that the 7B model outperformed the 32B model on Python completion tasks in all ten task groups, with a statistically significant paired t-test, and was on par in C#. The compiler-feedback experiments show consistent gains of at least 3–4 percentage points for every model tested, with one model improving by 10 points, and the gain appears roughly independent of baseline performance. The authors interpret this as evidence that resource-efficient small language models are a viab

What carries the argument

The argument rests on two mechanisms. First, multi-sampling generation: under a calibrated generation-only wall-clock budget set by the larger model, the smaller model produces several completions (seven for Python, six for C#) and the candidate with the highest log probability is selected; a best-cosine upper bound is computed to show selection headroom. Second, an agentic revision loop: the model first writes baseline code with embedded assert-based unit tests, the code is run through an interpreter or compiler, and the resulting error output is fed back to a debugger role that rewrites the code; this repeats up to a few times. The quality metric for the first experiment is character n-gra

Load-bearing premise

The load-bearing assumption is that character n-gram cosine similarity to a reference solution measures code quality; a completion can be textually close to the reference yet be syntactically invalid or functionally wrong.

What would settle it

Re-run the completion experiment on the same benchmark but grade by executing the generated completions against unit tests (or checking that they compile) instead of comparing text similarity. If the 7B model's selected completions do not pass tests at least as often as the 32B model's, the claim that multi-sampling matches or beats the larger model is not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the multi-sampling result holds, teams can substitute one smaller model for a larger one on repository-level completion, cutting GPU count fourfold and lowering per-task generation runtime.
  • Compiler feedback gives a model-family-independent correctness boost of 3–4 percentage points on function-level code generation, even with a naive implementation.
  • Because the two mechanisms are independent, combining them should yield both benefits in a single pipeline—a testable prediction the paper makes.
  • The same agentic scaffolding is generic: later versions can add retrieval-augmented context, static-analysis warnings, and profiler feedback to move from correctness to performance optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's quality metric for the completion experiment is textual similarity, so the 'equivalent or higher quality' claim may understate or overstate functional correctness; replacing it with execution-based tests would either strengthen or qualify the result.
  • If the compiler-feedback gain is truly independent of model size, then the bottleneck for small models is not capacity but lack of a verification loop, which suggests that even smaller models could be useful in energy-constrained settings.
  • The best-cosine upper bound in the paper implies that a smarter selection strategy—for example, running generated unit tests and reranking—could push the 7B model further beyond the 32B baseline without extra sampling.
  • The same recipe could transfer to scientific code optimization beyond astronomy: any domain with a compiler or test harness available can use cheap models plus feedback instead of large models, which matters for labs with limited GPU budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates two mechanisms for making small language models (SLMs) competitive with larger models for code generation in a sustainable-radio-astronomy context. First, it compares multi-sampling Qwen2.5-Coder-7B against single-generation Qwen2.5-Coder-32B on CrossCodeEval Python and C# repository-level completion under a calibrated generation-only runtime budget, using character n-gram cosine similarity to ground truth as the quality metric, with log-probability selection and a best-cosine oracle upper bound. Second, it evaluates an agentic workflow that feeds compiler errors back to five different models on HumanEval, reporting consistent improvements of roughly 3-4 percentage points and an error analysis. The paper concludes that multi-sampling SLMs can match or exceed larger single-generation models at a quarter of the GPU resources and that compiler feedback improves all tested models.

Significance. If the central claims hold, the paper addresses a practically important problem: reducing the energy/compute cost of code-generation assistance for large scientific codebases. The experimental design has several strengths: the use of 10 seed-based task groups with paired comparisons, a runtime budget that deliberately favors the 7B model, public code availability, and an explicit lower-bound evaluation strategy for HumanEval that treats hallucinated examples as failures. The compiler-feedback agent is generic and cheap to implement. However, the primary multi-sampling 'quality' conclusion currently rests on a lexical similarity proxy rather than execution-based correctness, and the C# result is not statistically significant. The significance of the work will depend on whether the quality comparison can be validated with functional metrics or the claims appropriately narrowed.

major comments (4)
  1. [§IV-A, §V-A] The decisive claim that the 7B model 'achieved equivalent of higher quality code' is supported only by character n-gram cosine similarity between the generated completion and the ground truth. This is a textual-overlap measure: a syntactically invalid or functionally wrong completion can still score high, especially in repository-level tasks where boilerplate and imported symbols dominate. The paper itself notes (IV-A) that the best-cosine upper bound requires ground truth and thus is not a deployable metric, but the same proxy underlies the logP-selected comparison. Please add an execution-based evaluation (e.g., compile/pass@k on a CrossCodeEval subset) or demonstrate that cosine similarity correlates with functional correctness; otherwise the quality/resource claim is unsupported.
  2. [§V-A, §VII] For C#, the paired t-test is not significant (p=0.184941; 95% CI [-0.004436, 0.019845]), and the group-level results are 6 positive, 2 ties, 2 negative out of 10 groups. Section VII nevertheless states that the 7B model 'matches or outperforms the 32B model in code completion over repository-level tasks in Python and C#.' This overstates the evidence. The C# result should be reported as numerically comparable but not statistically significant, with the Python result kept distinct.
  3. [§IV-C, §V-B] The compiler-feedback improvement is load-bearing for the paper's second contribution, but the current evaluation does not isolate the effect of the error text. The baseline is a single generated attempt, while the agentic mode receives at least one revision opportunity; the gain could be due to additional inference, repeated attempts, or even a generic 'try again' effect. In addition, only 5 samples per task are used, with no confidence intervals or significance tests. Please report pass@1/pass@k with standard errors and include a control condition where the model revises without compiler feedback.
  4. [§IV-A, §V-A] The resource-efficiency and sustainability conclusions are based on GPU count (1 vs 4 A100s) and generation-only wall-clock time, explicitly excluding model loading, environment initialization, I/O, and energy. For a paper whose motivation is environmental impact, the 'one quarter of the resources' claim is a GPU-hour proxy, not a measured energy or total-job-time result. Either measure energy directly or limit the conclusion to generation-only GPU-hours and state that total environmental impact was not measured.
minor comments (5)
  1. [§V-A] Typo: 'equivalent of higher quality' should be 'equivalent or higher quality.' Also, 'outperforms the 32B model in 6 tasks' should read 'in 6 seed groups,' since each group contains 150 tasks.
  2. [§IV-C, §V-B] Please define whether the HumanEval results are pass@1, pass@5, or best-of-5. The text says 'with 5 samples per task' but does not specify the selection rule for the reported percentages.
  3. [§V-B] The claim that the performance gain is 'mostly independent from model size, class or family' is based on only five models (plus hybrid configurations); consider softening this generalization.
  4. [Fig. 2] Figure 2 appears to be reproduced from [39] with a different internal numbering. Please verify that permission/reproduction rights are in order and that the caption credits the source correctly.
  5. [§IV-B] The agentic experiments were run on a CPU-only AMD EPYC system, but the paper does not state the sampling temperature, max tokens, or decoding settings for the five models. Adding these details would improve reproducibility.

Circularity Check

0 steps flagged

No circularity: empirical benchmark comparisons are self-contained; self-citations are contextual only.

full rationale

The paper’s central claims are empirical evaluations against external benchmarks (CrossCodeEval for repository-level completion, HumanEval for compiler-feedback correctness), with explicit metrics described in the methodology. The multi-sampling experiment does not fit a parameter and then predict a quantity forced by that fit: the number of samples k is calibrated to a runtime budget, and the comparison of cosine similarity to ground truth is measured, not derived from the calibration. The text even acknowledges the runtime outcome as a constraint ('The 7B model is constrained to have a lower generation-only runtime than the 32B model'), so no fitted input is disguised as a prediction. The best-cosine upper bound is explicitly an oracle requiring ground truth and is not presented as a practical method. The compiler-feedback experiment evaluates held-out validation items, and the authors state that validation-set feedback is not given to the model, so the observed gains are not forced by construction. The minor self-citations ([14] for the SuperCode project and [59] for a future sustainability metric) provide project context and a pointer to future work; they are not load-bearing premises for the results. Even the possible weakness that character n-gram cosine similarity is a proxy, rather than execution-based correctness, is a validity limitation, not a circular reduction: the result is not equivalent to its inputs by definition. No circular step meets the required evidentiary bar.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are introduced; SuperRAG is a software system name. The load-bearing assumptions are the cosine-similarity quality proxy, the attribution of agentic gains to compiler feedback, and the use of GPU count/runtime as an environmental proxy. The central results are empirical, so the only tuned numbers are experimental sampling budgets.

free parameters (2)
  • k (number of multi-samples for the 7B model) = k=7 for Python, k=6 for C#
    Chosen so that 7B repeated generation stays within the 32B single-generation runtime budget; it directly determines the resource/performance trade-off claimed. It is not fitted to the outcome, but it is a hand-chosen experimental parameter.
  • HumanEval samples per task = 5
    Sampling budget used for all models in the compiler-feedback experiment; no sensitivity analysis is reported and the pass-rate aggregation is under-specified.
axioms (4)
  • domain assumption Character n-gram cosine similarity is a valid proxy for code-completion quality.
    Used in Section IV-A and V-A to decide whether the 7B multi-sampling model 'matches or outperforms' the 32B model; textual similarity is not functional correctness.
  • domain assumption A second debugger pass without compiler feedback would not produce the same gain, so the 3-4 percentage-point improvement is attributable to compiler feedback.
    No control condition in Section V-B; the agentic mode gives the model an extra attempt plus compiler errors, confounding the two effects.
  • domain assumption GPU count and generation-only runtime are sufficient proxies for environmental footprint.
    Sections I and VI infer sustainability from fewer GPUs and lower runtime; no energy, carbon, or water measurements are reported.
  • domain assumption HumanEval pass rate with assert-based self-tests measures code-generation correctness.
    Used in Section V-B; hallucinated assert examples are counted as failures, so the metric is a stated lower bound rather than an upper-bound quality estimate.

pith-pipeline@v1.3.0-alltime-deepseek · 15721 in / 12661 out tokens · 129889 ms · 2026-08-01T08:13:37.148335+00:00 · methodology

0 comments
read the original abstract

Recent Large Language Models (LLMs) can produce and optimize complex code. We investigate the use of LLMs to generate and optimize code for large-scale sciences, focusing on radio astronomy and sustainability. The LOFAR telescope is currently being upgraded, significantly increasing the sky area observed, while simultaneously processing more data faster. However, this is expected to increase the computational requirements 40-fold. This upgrade thus critically depends on rigorous performance optimization of existing software and widespread adoption of accelerators. The code base is very large, making this a daunting task. We therefore investigate and demonstrate an AI-driven approach meant to assist developers in evaluating and optimizing their code, including porting to hardware accelerators. The LOFAR community is committed to sustainable solutions, and needs to achieve these improvements without increasing the energy budget. We thus need to optimize existing codes or port them to accelerators, while making sure that the optimization process itself is also energy efficient. This poses a challenge, since LLMs are energy-intensive. We therefore propose to use Small Language Models (SLMs) instead to limit environmental impact. In this paper, we show how to enhance SLMs through the use of agentic AI. We extend the SLMs in two ways to improve code generation quality and performance: first with a multi-sampling generation strategy and second with incorporating compiler feedback. We demonstrate that multi-sampling SLMs can match or surpass larger single-generation models with fewer computational resources and that feeding compiler output back into the SLMs leads to consistent improvements across all tested models. Our approach is generic, and can also use Retrieval Augmented Generation (RAG) as well as static and dynamic analysis tools in the code generation pipeline.

Figures

Figures reproduced from arXiv: 2607.21677 by Elisa Chiarotto, Jingbo Li, P. Chris Broekema, Rob V. van Nieuwpoort.

Figure 1
Figure 1. Figure 1: The LOFAR central core showing 6 core stations. 5 more core stations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Comparison of Graph of Thoughts (GoT) to other prompting strategies. Fi2Ghil ttif CTCTSCTT d GT [39] [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Workflow for the multi-sampling generation vs large single generation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Graphical representation of the agentic workflow. Elements with the gray dashed line are not implemented yet, elements with the black dashed line [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean generation-only runtime across ten seed groups for 32B single [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Group-level mean cosine similarity across ten seed groups for Python [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance gain for different models (ordered by baseline perfor [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance for different models (ordered by performance). The [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 21 linked inside Pith

  1. [1]

    Cwm: An open-weights llm for research on code generation with world models,

    Q. Carbonneaux, G. Cohen, J. Gehring, J. Kahn, J. Kossen, F. Kreuk, E. McMilin, M. Meyer, Y . Wei, D. Zhanget al., “Cwm: An open-weights llm for research on code generation with world models,”arXiv e-prints, pp. arXiv–2510, 2025

    2025

  2. [2]

    Code graph model (cgm): A graph-integrated large language model for repository-level software engineering tasks,

    H. Tao, Y . Zhang, Z. Tang, H. Peng, X. Zhu, B. Liu, Y . Yang, Z. Zhang, Z. Xu, H. Zhanget al., “Code graph model (cgm): A graph-integrated large language model for repository-level software engineering tasks,” arXiv preprint arXiv:2505.16901, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen3-coder-next technical report,

    R. Cao, M. Chen, J. Chen, Z. Cui, Y . Feng, B. Hui, Y . Jing, K. Li, M. Li, J. Linet al., “Qwen3-coder-next technical report,”arXiv preprint arXiv:2603.00729, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    How much do language models memorize?

    J. X. Morris, C. Sitawarin, C. Guo, N. Kokhlikyan, G. E. Suh, A. M. Rush, K. Chaudhuri, and S. Mahloujifar, “How much do language models memorize?”arXiv preprint arXiv:2505.24832, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon,

    U. S. Prashanth, A. Deng, K. O’Brien, J. SV , M. A. Khan, J. Borkar, C. A. Choquette-Choo, J. R. Fuehne, S. Biderman, T. Keet al., “Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon,”arXiv preprint arXiv:2406.17746, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Scaling laws for fact memorization of large language models,

    X. Lu, X. Li, Q. Cheng, K. Ding, X.-J. Huang, and X. Qiu, “Scaling laws for fact memorization of large language models,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 11 263–11 282

    2024

  8. [8]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  9. [9]

    Towards reasoning ability of small language models,

    G. Srivastava, S. Cao, and X. Wang, “Towards reasoning ability of small language models,”arXiv preprint arXiv:2502.11569, 2025

    arXiv 2025

  10. [10]

    Why language models hallucinate,

    A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang, “Why language models hallucinate,”arXiv preprint arXiv:2509.04664, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,

    D. Cotroneo, C. Improta, and P. Liguori, “Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2025, pp. 252–263

    2025

  12. [12]

    Speed at the cost of quality: How cursor ai increases short-term velocity and long-term complexity in open-source projects,

    H. He, C. Miller, S. Agarwal, C. K ¨astner, and B. Vasilescu, “Speed at the cost of quality: How cursor ai increases short-term velocity and long-term complexity in open-source projects,”arXiv preprint arXiv:2511.04427, 2025

    arXiv 2025

  13. [13]

    Measuring the impact of early-2025 ai on experienced open-source developer productivity,

    J. Becker, N. Rush, E. Barnes, and D. Rein, “Measuring the impact of early-2025 ai on experienced open-source developer productivity,”arXiv preprint arXiv:2507.09089, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Supercode: Sustainability per ai-driven co-design,

    P. C. Broekema and R. V . van Nieuwpoort, “Supercode: Sustainability per ai-driven co-design,” inProceedings of the 22nd ACM International Conference on Computing Frontiers: Workshops and Special Sessions, 2025, pp. 141–149

    2025

  15. [15]

    Lofar: The low-frequency array,

    M. P. van Haarlem, M. W. Wise, A. Gunst, G. Heald, J. P. McKean, J. W. Hessels, A. G. de Bruyn, R. Nijboer, J. Swinbank, R. Fallows et al., “Lofar: The low-frequency array,”Astronomy & astrophysics, vol. 556, p. A2, 2013

    2013

  16. [16]

    Into the depths: Unveiling elais-n1 with LOFAR’s deepest sub-arcsecond wide- field images,

    J. de Jong, R. van Weeren, F. Sweijen, J. Oonk, T. Shimwell, A. Offringa, L. Morabito, H. R ¨ottgering, R. Kondapally, E. Escottet al., “Into the depths: Unveiling elais-n1 with LOFAR’s deepest sub-arcsecond wide- field images,”Astronomy & Astrophysics, vol. 689, p. A80, 2024

    2024

  17. [17]

    The lofar correlator: Implementation and performance analysis,

    J. W. Romein, P. C. Broekema, J. D. Mol, and R. V . van Nieuwpoort, “The lofar correlator: Implementation and performance analysis,”ACM Sigplan Notices, vol. 45, no. 5, pp. 169–178, 2010

    2010

  18. [18]

    Commodity compute-and data-transport system design in modern large-scale distributed radio telescopes,

    P. Broekema, “Commodity compute-and data-transport system design in modern large-scale distributed radio telescopes,” 2020

    2020

  19. [19]

    Lofar2.0 documentation

    ASTRON. Lofar2.0 documentation. [Online]. Available: https://www. lofar.eu/lofar2-0-documentation/

  20. [20]

    Analogue neuromorphic receiver signal processing for radio astronomy,

    H. Bos, “Analogue neuromorphic receiver signal processing for radio astronomy,” 2025

    2025

  21. [21]

    How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,

    N. Jegham, M. Abdelatti, C. Y . Koh, L. Elmoubarki, and A. Hendawi, “How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,”arXiv preprint arXiv:2505.09598, 2025

    arXiv 2025

  22. [22]

    Carbon emissions and large neural network training,

    D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,”arXiv preprint arXiv:2104.10350, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Making ai less’ thirsty’,

    P. Li, J. Yang, M. A. Islam, and S. Ren, “Making ai less’ thirsty’,” Communications of the ACM, vol. 68, no. 7, pp. 54–61, 2025

    2025

  24. [24]

    Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,

    R. Desislavov, F. Mart ´ınez-Plumed, and J. Hern ´andez-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,”Sustainable Computing: Informatics and Sys- tems, vol. 38, p. 100857, 2023

    2023

  25. [25]

    Quanti- fying the carbon emissions of machine learning,

    A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quanti- fying the carbon emissions of machine learning,”arXiv preprint arXiv:1910.09700, 2019

    Pith/arXiv arXiv 1910

  26. [26]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschelet al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  27. [27]

    How interpretable are reasoning explanations from prompting large language models?

    Y . W. Jie, R. Satapathy, R. Goh, and E. Cambria, “How interpretable are reasoning explanations from prompting large language models?” inFindings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 2148–2164

    2024

  28. [28]

    Deep sub-arcsecond wide-field imaging of the Lockman Hole field at 144 MHz,

    F. Sweijen, R. J. van Weeren, H. J. A. R ¨ottgering, L. K. Morabito, N. Jackson, A. R. Offringa, S. van der Tol, B. Veenboer, J. B. R. Oonk, P. N. Best, M. Bondi, T. W. Shimwell, C. Tasse, and A. P. Thomson, “Deep sub-arcsecond wide-field imaging of the Lockman Hole field at 144 MHz,” pp. 350–356, Jan. 2022

    2022

  29. [29]

    Astron receives C4.6 million to widen and sharpen lofar’s cosmic vision

    ASTRON. Astron receives C4.6 million to widen and sharpen lofar’s cosmic vision. [Online]. Available: https://www.astron.nl/lenss-awarded/

  30. [30]

    astrocamp: A community benchmark and co-design framework for sustainable ska-scale radio imaging,

    D.-A. Constantinescu, R. Rodr ´ıguez ´Alvarez, J. Morin, E. Orliac, M. Dardaillon, S. Wang, H. Miomandre, M. Pe ´on-Quir´os, J.-F. Nezan, and D. Atienza, “astrocamp: A community benchmark and co-design framework for sustainable ska-scale radio imaging,” inPlatform for Advanced Scientific Computing Conference, 2026, pp. 1–12

    2026

  31. [31]

    Celloai: Leveraging large language models for hpc software development in high energy physics,

    M. Atif, K. Chopra, O. Kilic, T. Wang, Z. Dong, C. Leggett, M. Lin, P. Calafiura, and S. Habib, “Celloai: Leveraging large language models for hpc software development in high energy physics,” inPlatform for Advanced Scientific Computing Conference, 2026, pp. 1–11

    2026

  32. [32]

    The larger the better? improved llm code-generation via budget reallocation,

    M. Hassid, T. Remez, J. Gehring, R. Schwartz, and Y . Adi, “The larger the better? improved llm code-generation via budget reallocation,”arXiv preprint arXiv:2404.00725, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Large language monkeys: Scaling inference compute with repeated sampling,

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V . Le, C. R ´e, and A. Mirhoseini, “Large language monkeys: Scaling inference compute with repeated sampling,”arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Codet: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,”arXiv preprint arXiv:2207.10397, 2022

    Pith/arXiv arXiv 2022

  35. [35]

    Lever: Learning to verify language-to-code generation with execution,

    A. Ni, S. Iyer, D. Radev, V . Stoyanov, W. tau Yih, S. I. Wang, and X. V . Lin, “Lever: Learning to verify language-to-code generation with execution,” 2023. [Online]. Available: https://arxiv.org/abs/2302.08468

    Pith/arXiv arXiv 2023

  36. [36]

    Metis: fast quality-aware rag systems with configuration adaptation,

    S. Ray, R. Pan, Z. Gu, K. Du, S. Feng, G. Ananthanarayanan, R. Netravali, and J. Jiang, “Metis: fast quality-aware rag systems with configuration adaptation,” inProceedings of the ACM SIGOPS 31st symposium on operating systems principles, 2025, pp. 606–622

    2025

  37. [37]

    Ragboost: Efficient retrieval-augmented generation with accuracy-preserving con- text reuse,

    Y . Jiang, Y . Huang, L. Cheng, C. Deng, X. Sun, and L. Mai, “Ragboost: Efficient retrieval-augmented generation with accuracy-preserving con- text reuse,”arXiv preprint arXiv:2511.03475, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Retrieval-augmented code generation: A survey with focus on repository-level approaches,

    Y . Tao, Y . Qin, and Y . Liu, “Retrieval-augmented code generation: A survey with focus on repository-level approaches,”arXiv preprint arXiv:2510.04905, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Demystifying chains, trees, and graphs of thoughts,

    M. Besta, F. Memedi, Z. Zhang, R. Gerstenberger, G. Piao, N. Blach, P. Nyczyk, M. Copik, G. Kwa ´sniewski, J. M ¨uller, L. Gianinazzi, A. Kubicek, H. Niewiadomski, A. O’Mahony, O. Mutlu, and T. Hoefler, “Demystifying chains, trees, and graphs of thoughts,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 12, pp. 10 967–10 989, 2025

    2025

  40. [40]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  41. [41]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,”arXiv preprint arXiv:2203.11171, 2022

    Pith/arXiv arXiv 2022

  42. [42]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,”Advances in neural information processing systems, vol. 36, pp. 11 809–11 822, 2023

    2023

  43. [43]

    Forest-of-thought: Scaling test-time compute for enhancing llm reasoning,

    Z. Bi, K. Han, C. Liu, Y . Tang, and Y . Wang, “Forest-of-thought: Scaling test-time compute for enhancing llm reasoning,”arXiv preprint arXiv:2412.09078, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Graph of thoughts: Solving elaborate problems with large language models,

    M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyket al., “Graph of thoughts: Solving elaborate problems with large language models,” inProceedings of the AAAI conference on artificial intelligence, vol. 38, no. 16, 2024, pp. 17 682–17 690

    2024

  45. [45]

    Everything of thoughts: Defying the law of penrose triangle for thought generation,

    R. Ding, C. Zhang, L. Wang, Y . Xu, M. Ma, W. Zhang, S. Qin, S. Rajmohan, Q. Lin, and D. Zhang, “Everything of thoughts: Defying the law of penrose triangle for thought generation,” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 1638– 1662

    2024

  46. [46]

    Buffer of thoughts: Thought-augmented reasoning with large language models,

    L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang, J. E. Gonzalez, and B. Cui, “Buffer of thoughts: Thought-augmented reasoning with large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 113 519–113 544, 2024

    2024

  47. [47]

    Lightrag: Simple and fast retrieval-augmented generation,

    Z. Guo, L. Xia, Y . Yu, T. Ao, and C. Huang, “Lightrag: Simple and fast retrieval-augmented generation,”arXiv preprint arXiv:2410.05779, vol. 2, no. 3, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Reversal of thought: Enhancing large language models with preference-guided reverse rea- soning warm-up,

    J. Yuan, D. Du, H. Zhang, Z. Di, and U. Naseem, “Reversal of thought: Enhancing large language models with preference-guided reverse rea- soning warm-up,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 19 442–19 459

    2025

  49. [49]

    Combining large language models with static analyzers for code review generation,

    I. Jaoua, O. B. Sghaier, and H. Sahraoui, “Combining large language models with static analyzers for code review generation,” in2025 IEEE/ACM 22nd International Conference on Mining Software Reposi- tories (MSR). IEEE, 2025, pp. 174–186

    2025

  50. [50]

    Static analysis as a feedback loop: Enhancing llm-generated code beyond correctness,

    S. Blyth, S. A. Licorish, C. Treude, and M. Wagner, “Static analysis as a feedback loop: Enhancing llm-generated code beyond correctness,” in2025 IEEE International Conference on Source Code Analysis & Manipulation (SCAM). IEEE, 2025, pp. 100–109

    2025

  51. [51]

    Producing wrong data without doing anything obviously wrong!

    T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney, “Producing wrong data without doing anything obviously wrong!”ACM Sigplan Notices, vol. 44, no. 3, pp. 265–276, 2009

    2009

  52. [52]

    E. Berger. Coz: Finding code that counts with causal profiling. [Online]. Available: https://github.com/plasma-umass/coz

  53. [53]

    Rapidly selecting good compiler optimizations using per- formance counters,

    J. Cavazos, G. Fursin, F. Agakov, E. Bonilla, M. F. O’Boyle, and O. Temam, “Rapidly selecting good compiler optimizations using per- formance counters,” inInternational Symposium on Code Generation and Optimization (CGO’07). IEEE, 2007, pp. 185–197

    2007

  54. [54]

    Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,

    Y . Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Rothet al., “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 46 701– 46 723, 2023

    2023

  55. [55]

    Langchain community

    Langchain-AI. Langchain community. [Online]. Available: https: //github.com/langchain-ai/langchain-community

  56. [56]

    The faiss library.(2024),

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar ´e, M. Lomeli, L. Hosseini, and H. J ´egou, “The faiss library.(2024),”arXiv preprint arXiv:2401.08281, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Qwen2. 5 technical report,

    A. Y . Qwen, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,”arXiv preprint, 2024

    2024

  58. [58]

    Coz: Finding code that counts with causal profiling,

    C. Curtsinger and E. D. Berger, “Coz: Finding code that counts with causal profiling,” inProceedings of the 25th Symposium on Operating Systems Principles, 2015, pp. 184–197

    2015

  59. [59]

    Job-level carbon and water footprint estimation for hpc: Bias assessment from runtime to full life cycle,

    X. Chen, C. Broekema, and R. van Nieuwpoort, “Job-level carbon and water footprint estimation for hpc: Bias assessment from runtime to full life cycle,” 2026. [Online]. Available: https://arxiv.org/abs/2607.19150

    Pith/arXiv arXiv 2026

  60. [60]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    2021