Pith. sign in

REVIEW 2 major objections 6 minor 34 references

Open-weight models deployed locally can reproduce human-curated data preparation scripts for longitudinal cohort studies with meaningful fidelity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:18 UTC pith:BII6KDEF

load-bearing objection A genuinely useful R-based benchmark for local LLM survey-data preparation, with an honest body and an abstract that oversells the detailed-prompt condition; worth serious peer review. the 2 major comments →

arxiv 2607.21482 v2 pith:BII6KDEF submitted 2026-07-23 cs.AI cs.CL

Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

Mack Nixon , Liam Wright , Yevgeniya Kovalchuk , Alison Fang-Wei Wu , Martin Danka , Andy Boyd , David Bann This is my paper
classification cs.AI cs.CL
keywords open-weight large language modelslocal deploymentagentic codingdata preparationlongitudinal cohort studiessurvey data harmonizationR scriptsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that open-weight large language models running entirely on local hardware can handle a large share of the tedious data preparation needed for longitudinal population studies—recoding survey responses, harmonizing categories across waves, and constructing derived variables—closely enough to reproduce human-curated cleaning scripts. Using a benchmark of 20 tasks drawn from a real British cohort, the best locally run model matched 85.9% of variables to ground truth, completed 75% of tasks fully in a single agentic pass, and produced data that yielded identical regression coefficients to the human-prepared data in six downstream analyses. The stakes are practical: research governance often forbids sending individual-level data to cloud AI services, so a model that works locally would let sensitive-data researchers use AI assistance without transmitting data. The paper is careful about scope: performance depends heavily on detailed prompts, drops to about two-thirds completion when the derivation recipe is withheld, and the authors recommend a reviewed copilot rather than an autonomous pipeline.

Core claim

The paper claims that an open-weight LLM running on a single high-end consumer GPU, operating as an agentic coder that writes and executes R scripts, can reproduce human-curated data-preparation pipelines for a longitudinal cohort study with strong fidelity: the top model produced 99.7% of the 102 expected variables, matched at least 95% of cells for 86.2% of them and effectively exact numeric reproduction for continuous ones, completed 75% of the 20 tasks with every variable correct in a single pass, and yielded regression coefficients identical to ground truth in all six downstream analyses. The claim is explicitly conditional: the benchmark's main condition supplied detailed prompts spell

What carries the argument

The load-bearing mechanism is the closed agentic loop: the LLM receives a task prompt plus machine-readable metadata (variable names, labels, value labels), drafts an R script, executes it over raw survey files, observes errors, and iterates until the script runs; an automated Python evaluator then compares the produced dataset cell-by-cell with human ground-truth output, scoring completeness and correctness, and multiplying the two into a 'balanced' performance score. The framework is built from 20 real tasks (creating 102 variables) spanning harmonization, multi-wave merging, recoding, and derived-measure construction, and it is portable to other tabular data.

Load-bearing premise

The claim of meaningful fidelity rests on treating the detailed prompt—which already spells out the derivation logic, outcome variables, and response categories—as the realistic way researchers will use these models; when that recipe is withheld, the best model's average task completion falls from about 88% to 67%, so the central claim would not survive in settings where the model must derive the logic from raw metadata alone.

What would settle it

Take the best-performing local model and run the same 20 tasks on a second cohort's data that the model has not seen in training, using only the lite-prompt format; if balanced variable performance falls below roughly 65% or any silent missing-value recoding flips a regression coefficient's sign, then the paper's fidelity claim is an artifact of its single benchmark and detailed prompts rather than a general capability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Trusted research environments and similar governance-restricted settings could deploy a locally run agent to produce first-pass cleaned datasets without transferring sensitive data to any external service.
  • The best local models are only viable on high-end consumer hardware; smaller models lose 20–30 percentage points or fail entirely to follow tool-calling protocols.
  • Because errors are silent—plausible-looking but wrong missing-value codes or recodings—a weaker model can produce data that looks clean yet materially shifts regression coefficients, so human verification of AI-prepared variables is essential.
  • Model-side improvements are not the only lever: since the detailed prompt was the strongest predictor of success, improving the machine-readability of survey metadata could yield larger gains than upgrading the model.
  • The benchmark and comparison pipeline are reusable, so other cohorts and languages (e.g., Stata, SAS, Python) can be tested without rebuilding evaluation tools.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline 85–88% numbers measure instruction-following and code execution, not autonomous derivation; the lite-prompt results (~67% for the best model) are the truer estimate of what a 'just look at the codebook' agent would achieve, so claims about autonomous capability should be scaled back.
  • A testable prediction follows from the paper's own complexity findings: if prompts were shortened by restructuring metadata rather than removing instructions, performance on the hardest tasks might improve more than from switching to a larger model, because prompt length correlated -0.75 with success.
  • The framework could double as a cheap pre-deployment quality gate: a research group could run a candidate model on a small set of their own historic cleaning scripts and reject models whose silent-error rate is unacceptably high for their downstream analyses.
  • Because the lightest model failed entirely at the orchestration format rather than at coding, agent-framework compatibility, not raw coding skill, may be the binding constraint for small local models; re-testing with a different agent scaffold could change size-tier rankings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces RRBench, an open-source framework for evaluating open-weight LLM agents on longitudinal survey data-preparation tasks drawn from the Next Steps cohort. It benchmarks eight open-weight models under two prompt conditions (detailed and 'lite') and reports variable-level completeness/correctness, task-level completion, sensitivity analyses (thresholds, missing-exclusion), and downstream regression consistency. The best model (Gemma 4 31B) achieves 85.9% balanced variable performance, 75% fully completed tasks, and an average task completion of 87.9% under detailed prompts; under lite prompts the best model drops to roughly 61% balanced performance and 67% average task completion. The paper concludes that locally deployable open-weight LLMs can reproduce human-curated data preparation scripts with meaningful fidelity, positioning the framework as a practical tool for governance-restricted research settings.

Significance. The benchmark is a timely and valuable contribution to the evaluation of local LLMs for sensitive-data research. Its strengths include: a realistic ground truth derived from a real cohort study with independent code review; pre-specified cell-level agreement thresholds; a public open-source framework; sensitivity analyses that probe threshold choice, missing-data handling, and prompt detail; and a downstream analysis that quantifies the material impact of silent data-preparation errors. The finding that the principal differentiator between models is correctness rather than completeness, and that even plausible-looking outputs can substantially distort regression estimates, is an important cautionary result for practitioners. If the central claim is suitably qualified, the framework will be a reusable resource for the community. The main limitation is interpretive: the headline results are obtained with prompts that embed derivation recipes, so they measure instruction-following and code execution rather than autonomous derivation from metadata. The paper's own lite-prompt sensitivity analysis and the Discussion's copilot framing partially address this, but the abstract and Discussi

major comments (2)
  1. [Methods, Tasks; Table S2; Discussion] The primary benchmark condition supplies a detailed prompt that 'specified code logic, required outcome variables, and response categories.' The headline result (87.9% avg task completion, Table 3) therefore measures the agent's ability to translate a near-complete derivation recipe into R code and execute it. Under the lite prompt (Table S2), which removes the code logic, the best model drops to 67.4% average task completion. The Discussion's claim that the framework 'deliberately measures LLMs' fully autonomous performance' is misleading; 'autonomous' holds only in the weak sense of no human intervention during execution. The abstract's 'almost saturated our benchmark' should be qualified as applying to the detailed-prompt condition, and the lite-prompt results should be given comparable prominence.
  2. [Conclusion; Discussion] The conclusion that local LLMs 'reproduce human-curated data preparation scripts ... with meaningful fidelity' is uncalibrated without a human-consistency baseline. The 85.9% balanced performance and 75% complete-task figures are absolute, but the relevant yardstick is the agreement between two human data scientists independently performing the same tasks. Because the ground truth is a single human-derived script (albeit code-reviewed), the benchmark does not show whether LLM outputs fall within the human range of variation. Please either add a human-baseline analysis (e.g., a second researcher deriving a subset of tasks from metadata only) or temper the conclusion to state that the models approximate the reference scripts, rather than that they match human-level fidelity.
minor comments (6)
  1. [Abstract; Table 3] The abstract cites 87.9% average task completion without stating that this is for the detailed-prompt condition; the lite-prompt result is roughly 67%. Please make the condition explicit in both the abstract and Table 3.
  2. [Results; Table S2] The text states that GPT-OSS-20B is 'the highest performing model' for lite prompts, but Gemma 4 31B has higher balanced performance (61.4 vs 60.8) and nearly equal average task completion (67.4 vs 67.5). Specify which metric was used for the comparison.
  3. [Downstream Analyses; Table 4] The selection of three variable groups by performance level (high, medium, low) is described but not detailed; it would be useful to state how the groups were chosen and whether the coefficients in Table 4 are representative across the three runs, since only one set of coefficients is shown.
  4. [Model Evaluation; Deployment] The evaluation was run on UCL's Unified AI Services Computing Facility, not on the consumer-grade hardware discussed in the abstract. State the actual hardware used and whether full-precision or quantized checkpoints were evaluated, to support the consumer-grade deployment claim.
  5. [Figure 4] The correlations in Figure 4 are described as Pearson r, but the variables are rank-normalized before computing the correlation. This is equivalent to Spearman's rank correlation; please re-label or compute on the raw values.
  6. [References] Reference 14 includes 'OFFICIAL_SENSITIVE' in the URL fragment; please verify that this is a suitable public citation and that the link is stable.

Circularity Check

0 steps flagged

No significant circularity; empirical benchmark against an externally fixed, independently reviewed ground truth rather than a derivation.

full rationale

This paper is an empirical benchmark, not a derivation chain. The claimed result — that open-weight LLMs can reproduce human-curated data-preparation scripts with meaningful fidelity — is evaluated by generating R scripts and output datasets and comparing them cell-by-cell against a fixed ground-truth dataset (Methods: 'The core dataset processing scripts were created by experienced cohort researchers and validated with independent code review'). No model parameter is fitted to the benchmark outputs, and no reported metric is defined in terms of the LLM's predictions. The closest concern is that the primary condition uses detailed prompts that specify code logic, required outcome variables, and response categories ('the detailed prompt specified code logic, required outcome variables, and response categories'). This means the headline performance measures instruction-following and code execution rather than fully autonomous derivation from metadata. That is a scope/validity limitation, not circularity: the LLM's output is not equivalent to the prompt by construction, and the paper directly addresses the issue by including a lite-prompt sensitivity analysis (Table S2) and by hedging its conclusion ('human code review is still recommended'). The authors also explicitly acknowledge prompt-design sensitivity in the Discussion: 'LLM performance is sensitive to prompt design, and we acknowledge the substantial researcher degrees of freedom this introduces.' Self-citations (e.g., the Next Steps cohort profile by Wu et al.) are descriptive references and are not load-bearing for the benchmark's validity or for any derived quantity. No circular step can be exhibited by quoting an equation where a prediction reduces to its input; therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No ungrounded entities are introduced. The scoring rules (thresholds, temperature, run count) and the domain assumptions above are the ledger items the central claim rests on. The benchmark itself is a shipped artifact, not a postulate; it is publicly released, which gives it independent falsifiability.

free parameters (5)
  • Categorical correctness threshold = 0.95 cell-level agreement
    Hand-chosen scorer: a variable counts as 'correct' if ≥95% of its cells exactly match ground truth. Sensitivity analysis (Fig. S4) shows performance is similar from 80-95% but declines steeply at 100%, so the headline numbers depend on the threshold being below 100%.
  • Continuous-variable tolerance = normalized RMSE ≤ 1e-4
    Hand-chosen scorer for continuous variables, calibrated to allow floating-point rounding while effectively requiring exact reproduction.
  • Sampling temperature = 0.8
    Chosen after observing that lower temperatures left the agent stuck on initial bugs; directly affects run-to-run variability and reported performance.
  • Number of runs per task = 3
    Each task run three times; metrics are pooled/averaged over runs with no reported variance.
  • Missing-value exclusion in sensitivity analysis = excluded NAs from comparison
    When missing values are excluded, correctness jumps to 94.4% for the best model (Table S4); the paper attributes most errors to missing-data coding differences, so this exclusion materially changes the interpretation of what the models get wrong.
axioms (5)
  • domain assumption The Next Steps Core ground-truth scripts are correct and reflect the intended research-ready transformations
    Every score in the paper measures agreement with these human-curated scripts. The paper cites independent code review, but the ground truth's correctness is not independently verifiable within the paper (Methods 'Ground-truth data'; author contributions).
  • domain assumption The automated matching/comparison routines correctly align LLM-produced variables and value labels to ground-truth counterparts
    The paper states 'we used a range of approaches to ensure we correctly matched outputs' and defers details to the GitHub repository; scoring therefore rests on an unspecified procedure (Methods 'Model Evaluation').
  • domain assumption Cell-level agreement with one human team's scripts is a valid definition of correctness for data preparation
    Alternative valid recoding choices (e.g., which sweep's labels to prefer) are scored as errors unless they pass the 95% threshold; no inter-human consistency measurement calibrates how large 85.9% fidelity actually is (Methods 'Variable-level Comparisons').
  • domain assumption The 20 tasks from Next Steps are representative of longitudinal population-study data preparation
    All tasks come from one British cohort; generalizability to other cohorts, instruments, languages, or governance settings is asserted as future work, not demonstrated (Discussion).
  • domain assumption Models released within 12 months before May 2026 represent the current open-weight state of the art
    Model-selection criterion in Methods 'LLM Choice'; benchmark scores are a dated snapshot by design.

pith-pipeline@v1.3.0-alltime-deepseek · 13422 in / 19525 out tokens · 181385 ms · 2026-08-01T07:18:31.905088+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ('average task completion' up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.

Figures

Figures reproduced from arXiv: 2607.21482 by Alison Fang-Wei Wu, Andy Boyd, David Bann, Liam Wright, Mack Nixon, Martin Danka, Yevgeniya Kovalchuk.

Figure 1
Figure 1. Figure 1: Overview of the evaluation framework. Each (human curated) ground-truth task comprises input data files, metadata, a processing script, and the expected output dataset produced by running the processing script on the input data files. The AI agent (SmolAgents) receives input files and metadata, then iteratively generates and executes an R script within a closed local environment. Agent performance is evalu… view at source ↗
Figure 2
Figure 2. Figure 2: Structure of the prompt provided to the AI agent. The prompt consists of two components: a task description instructing the agent to clean and preprocess raw survey datasets into a single research-ready file, and structured metadata in JSON format describing the input data files, including variable names, labels, value labels, and file-level information such as the number of variables and cases. In our pri… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Challenges in using data across governement

    National Audit Office. Challenges in using data across governement. https://www.nao.org.uk/wp-content/uploads/2019/06/Challenges-in-using-data-across- government.pdf (2019)

    2019

  2. [2]

    & Fiala, L

    Brodeur, A., Mikola, D., Cook, N. & Fiala, L. Computational Reproducibility and Robustness of Empirical Economics and Political Science Research Between 2022 and 2023. https://lrlusher.weebly.com/uploads/1/0/0/4/10048967/meta_paper_i__nature_v2___4_.p df (2025)

    2022

  3. [3]

    Tang, Z. et al. LLM/Agent-as-Data-Analyst: A Survey. Preprint at https://doi.org/10.48550/arXiv.2509.23988 (2025)

    doi:10.48550/arxiv.2509.23988 2025

  4. [4]

    Parkes, O. L. End User Licence Agreement. (2024)

    2024

  5. [5]

    Jimenez, C. E. et al. SWE-bench: Can Language Models Resolve Real-world Github Issues? in The Twelfth International Conference on Learning Representations (2024)

    2024

  6. [6]

    Austin, J. et al. Program Synthesis with Large Language Models. Preprint at https://doi.org/10.48550/arXiv.2108.07732 (2021)

    Pith/arXiv arXiv doi:10.48550/arxiv.2108.07732 2021

  7. [7]

    Hendrycks, D. et al. Measuring Coding Challenge Competence With APPS. Preprint at https://doi.org/10.48550/arXiv.2105.09938 (2021)

    Pith/arXiv arXiv doi:10.48550/arxiv.2105.09938 2021

  8. [8]

    Chen, M. et al. Evaluating Large Language Models Trained on Code. Preprint at https://doi.org/10.48550/arXiv.2107.03374 (2021)

    Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374 2021

  9. [9]

    & Torvik, V

    Li, L., Fang, L., Ludäscher, B. & Torvik, V. I. AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark. in Findings of the Association for Computational Linguistics: EMNLP 2025 (eds Christodoulopoulos, C., Chakraborty, T., Rose, C. & Peng, V.) 7766–7780 (Association for Computational Linguistics, Suzhou, China, 2025). doi:10.18653/...

    doi:10.18653/v1/2025.findings-emnlp.410 2025

  10. [10]

    Zhang, D. et al. DataSciBench: An LLM Agent Benchmark for Data Science. arXiv.org https://arxiv.org/abs/2502.13897v1 (2025). 27

    Pith/arXiv arXiv 2025

  11. [11]

    TIOBE Index for March 2026

    TIOBE Software. TIOBE Index for March 2026. TIOBE https://www.tiobe.com/tiobe- index/

    2026

  12. [12]

    Goldmann, E., Barnard-Mayers, R., Glymour, M. M. & Healey, M. A. Statistical software skills for master’s-level jobs in epidemiology: an analysis of 15 years of job posting data. Am J Epidemiol kwaf243 (2025) doi:10.1093/aje/kwaf243

    work page doi:10.1093/aje/kwaf243 2025

  13. [13]

    Wu, A. F.-W. et al. Cohort Profile: Next Steps—the longitudinal study of people in England born in 1989–90. Int J Epidemiol 53, dyae152 (2024)

    1989

  14. [14]

    Review of ESRC’s Cohort Studies 2024/25

    Economic and Social Research Council. Review of ESRC’s Cohort Studies 2024/25. https://www.ukri.org/wp-content/uploads/2025/12/081225-ESRC- OFFICIAL_SENSITIVE_Cohorts_Review_v1.2_Publication-WT.pdf (2025)

    2024

  15. [15]

    Qu, X. et al. A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyond. Preprint at https://doi.org/10.48550/arXiv.2508.11957 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.11957 2025

  16. [16]

    Roucher, A., Moral, A. V. del, Wolf, T., Werra, L. von & Kaunismäki, E. `smolagents`: a smol library to build great agentic systems. (2025)

    2025

  17. [17]

    The Effect of Sampling Temperature on Problem Solving in Large Language Models

    Renze, M. The Effect of Sampling Temperature on Problem Solving in Large Language Models. in Findings of the Association for Computational Linguistics: EMNLP 2024 7346–7356 (Association for Computational Linguistics, Miami, Florida, USA, 2024). doi:10.18653/v1/2024.findings-emnlp.432

    doi:10.18653/v1/2024.findings-emnlp.432 2024

  18. [18]

    & Pollin, R

    Herndon, T., Ash, M. & Pollin, R. Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics 38, 257– 279 (2014)

    2014

  19. [19]

    Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of- Experts Layer. Preprint at https://doi.org/10.48550/arXiv.1701.06538 (2017)

    Pith/arXiv arXiv doi:10.48550/arxiv.1701.06538 2017

  20. [20]

    Arseneault, L. et al. Exploring the world for longitudinal datasets: data resources for transformative mental health research. Int J Epidemiol 54, dyaf128 (2025)

    2025

  21. [21]

    O’Connor, M. et al. Better together: Advancing life course research through multi-cohort analytic approaches. Adv Life Course Res 53, 100499 (2022). 28

    2022

  22. [22]

    Bann, D. et al. Investigating change across time in prevalence or association: the challenges of cross-study comparative research and possible solutions. Discov Soc Sci Health 2, 18 (2022)

    2022

  23. [23]

    Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Preprint at https://doi.org/10.48550/arXiv.2307.03172 (2023)

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.03172 2023

  24. [24]

    https://www.trychroma.com/research/context-rot#conclusion

    Context Rot: How Increasing Input Tokens Impacts LLM Performance. https://www.trychroma.com/research/context-rot#conclusion

  25. [25]

    Tran, K.-T. et al. Multi-Agent Collaboration Mechanisms: A Survey of LLMs. Preprint at https://doi.org/10.48550/arXiv.2501.06322 (2025)

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.06322 2025

  26. [26]

    FAIR Principles

    GO FAIR Initiative. FAIR Principles. GO FAIR https://www.go-fair.org/fair-principles/f1- meta-data-assigned-globally-unique-persistent-identifiers/ (n.d.)

  27. [27]

    & Zhang, T

    Tong, W. & Zhang, T. CodeJudge: Evaluating Code Generation with Large Language Models. in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing 20032–20051 (Association for Computational Linguistics, Miami, Florida, USA, 2024). doi:10.18653/v1/2024.emnlp-main.1118

    doi:10.18653/v1/2024.emnlp-main.1118 2024

  28. [28]

    Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All

    Qwen Team. Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. (2026)

    2026

  29. [29]

    Qwen3.5: Towards Native Multimodal Agents

    Qwen Team. Qwen3.5: Towards Native Multimodal Agents. Qwen https://qwen.ai/blog?id=qwen3.5 (2026)

    2026

  30. [30]

    Gemma 4 31B

    Google DeepMind. Gemma 4 31B. https://huggingface.co/google/gemma-4-31B (2026)

    2026

  31. [31]

    Devstral Small 2 24B Instruct

    Mistral AI. Devstral Small 2 24B Instruct. https://huggingface.co/mistralai/Devstral-Small- 2-24B-Instruct-2512 (2025)

    2025

  32. [32]

    gpt-oss-120b & gpt-oss-20b Model Card

    OpenAI. gpt-oss-120b & gpt-oss-20b Model Card. (2025)

    2025

  33. [33]

    Ministral 3 14B Instruct

    Mistral AI. Ministral 3 14B Instruct. https://huggingface.co/mistralai/Ministral-3-14B- Instruct-2512 (2025)

    2025

  34. [34]

    Gemma 4 E4B

    Google DeepMind. Gemma 4 E4B. https://huggingface.co/google/gemma-4-E4B (2026). 29 Supplementary information for: Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks Mack Nixon1*, Liam Wright2, Yevgeniya Kovalchuk1, Alison Fang-Wei Wu2, Martin Danka2, Andy Boyd3, David Bann2* 1Centre for Ad...

    2026