Pith. sign in

REVIEW 3 major objections 6 minor 2 references

AI for NONMEM Coding in Pharmacometrics Research and Education: Shortcut or Pitfall?

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A six-point rubric, folded into the prompt, lifts two AI agents to top NONMEM code on all 13 tasks.

desk verdict Useful NONMEM benchmark with a real circularity problem: perfect scores certify rubric adherence, not runnable code, so the 'great accuracy' claim overstates what the data support. read the letter →

arxiv 2507.08144 v1 pith:SFXOPLFL submitted 2025-07-10 q-bio.OT

Wenhao Zheng , Wanbing Wang , Carl M.J. Kirkpatrick , Cornelia B. Landersdorfer , Huaxiu Yao , Jiawei Zhou This is my paper · ORCID
classification q-bio.OT
keywords NONMEMpharmacometricspopulationPK/PDmodelinglargelanguagemodelsAIcodegenerationpromptoptimizationrubric-basedevaluationineducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether current AI agents can write NONMEM code, the specialized language used for population pharmacokinetic/pharmacodynamic modeling, reliably enough to be useful in research and teaching. It tests seven AI agents on 13 coding tasks spanning basic and advanced model structures, develops a six-point rubric to grade the outputs, and turns that rubric into an optimized prompt. The central result is that, with the optimized prompt, two agents—o1 and gpt-4.1—reach the maximum rubric level on all 13 tasks, while every tested agent improves relative to its original prompt. The authors conclude that AI can supply a solid first draft for many NONMEM models, but they emphasize that expert review remains essential, especially for complex models and dataset-specific alignment.

What carries the argument

The central object is the six-point scoring rubric, a checklist with three levels: first, that the code includes the essential NONMEM control-stream blocks from $PROB through $TABLE; second, that every ETA has a corresponding THETA relationship in $PK; and third, that subroutine, differential-equation, and parameter settings follow PREDPP guide conventions. The optimized prompt is built by rewriting those criteria as explicit instructions to the AI agent, and it is this prompt transformation that the paper credits with the large performance gains.

What would settle it

Compile and run all 13 rubric-top outputs in NONMEM against simulated datasets that match each task; if a substantial fraction fail to execute, fail to converge, or fail to recover known parameter values, the central accuracy claim is contradicted.

Watch

Extended reading notes

Core claim

The paper's claim, on its own terms, is that a rubric-derived prompt is enough to make current AI agents produce structurally correct NONMEM code across a representative range of population PK/PD tasks. Specifically, o1 and gpt-4.1 both attained the maximum rubric level on all 13 tasks when rerun with the optimized prompt, and all seven agents improved relative to their first attempts. The rubric defines accuracy in three stacked levels: presence of the essential control-stream blocks, correct THETA–ETA relationships in the $PK block, and conformance of $SUBROUTINE, $DES, $PK and parameter settings with PREDPP conventions. The paper therefore asserts 'great accuracy' as structural and conventional correctness under expert rubric judgment, not as demonstrated successful execution of the code in NONMEM.

Load-bearing premise

The rubric is assumed to capture code accuracy well enough that a perfect rubric result means 'great accuracy,' but the paper never executes any generated code in NONMEM on a dataset.

Editorial extensions

If this is right

  • The optimized prompt is a ready-to-use template for generating initial NONMEM drafts, and the authors invite its reuse in research and teaching.
  • AI-generated scripts can serve as a solid starting point for model development, reducing the syntax burden for newcomers to pharmacometrics.
  • For basic model structures, such as one- and two-compartment PK and simple exposure-response models, AI output commonly meets the structural criteria of the rubric.
  • Complex models, including indirect response, TMDD, and cases requiring special dataset alignment, still need expert refinement before use.
  • In education, the emphasis can shift from technical coding toward quantitative thinking, prompt crafting, and critical assessment of AI output, while guarding against over-reliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test would be to run the rubric-perfect outputs in NONMEM against simulated data; successful execution and parameter recovery would harden the accuracy claim, while failures would narrow it to structure-only correctness.
  • Because each agent answered each task once, the results may reflect one sample of each model; repeated sampling would reveal run-to-run variability and whether the optimized prompt reduces it.
  • The same rubric-to-prompt strategy could plausibly transfer to other domain-specific modeling languages, though the paper does not test that.
  • The optimized prompt brought the largest gains on complex tasks like indirect response and lag-time absorption models, suggesting it may be most useful exactly where expert oversight is also most needed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript evaluates seven OpenAI large language models on 13 NONMEM coding tasks spanning basic and advanced population PK/PD model structures. The authors introduce a three-level scoring rubric (maximum score 6) that awards points for the presence of standard control-stream blocks, correct THETA-ETA wiring, and expert judgment of PREDPP conformance. They then construct an 'optimized prompt' by embedding the rubric criteria directly into the prompt and re-run all tasks. Under the optimized prompt, both o1 and gpt-4.1 receive perfect 6.0/6.0 scores on all 13 tasks, which the abstract summarizes as 'great accuracy.' The paper also discusses implications for pharmacometrics education and provides a public repository with the tasks, rubric, prompts, and generated code.

Significance. If the headline claim were supported, this would be a useful benchmark for AI-assisted NONMEM coding: the task set spans relevant model structures, the rubric systematizes qualitative code review, the optimized prompt is a practical artifact, and the public repository supports follow-up work. The educational discussion is thoughtful and appropriately cautionary. However, the central accuracy claim is not backed by any execution of the generated code in NONMEM, and the evaluation instrument is confounded with the intervention because the optimized prompt is a restatement of the scoring rubric. The paper is best read as a rubric-conformance benchmark rather than a demonstration of executable NONMEM accuracy; the authors should either add execution-based validation or substantially soften the accuracy claims.

major comments (3)
  1. [Results; Table 2] The abstract's claim that o1 and gpt-4.1 generated codes 'with great accuracy for all tasks' is not supported by the rubric-based evidence. The rubric in Table 2 awards 2 points for the presence of standard control-stream blocks, 2 points for THETA-ETA wiring, and 2 points for an expert judgment of PREDPP conformance. It does not check whether the code compiles in NONMEM, whether $INPUT matches a real dataset, whether initial estimates are plausible, whether $ESTIMATION converges, whether $COVARIANCE completes, or whether any parameter estimates are clinically meaningful. A 6/6 score can therefore be earned by code that does not run or that mis-specifies the model. The wording 'great accuracy' should be replaced by a more precise term such as 'rubric-conformant,' or the authors should supplement the rubric with execution-based validation on a test dataset, reporting convergence, covariance success, and parameter recovery.
  2. [Analysis Plan; Table 2] The optimized prompt is constructed by transforming the same scoring rubric into instructions, and the same rubric is then used to measure the improvement. This creates a direct overlap between the intervention and the outcome instrument: a model can raise its score by reproducing the block names and structural phrases explicitly requested in the prompt, without producing a valid or executable NONMEM control stream. The reported perfect 6/6 scores therefore partly measure prompt adherence rather than coding accuracy. To support the claim that the optimized prompt 'substantially enhances accuracy,' the authors should evaluate performance with an independent criterion (e.g., running the code in NONMEM, or expert review blind to whether the optimized prompt was used) or at least acknowledge that the evaluation measures rubric adherence and that this confound limits the interpretation.
  3. [Results; Figure 1] Each task-model combination is evaluated from a single LLM response. The boxplots in Figure 1 therefore show variation across the 13 tasks but not sampling variability within a task, so a single unlucky or lucky generation can change the reported score and, potentially, the ranking of models. The paper should either report results from multiple independent runs (e.g., 5–10 repetitions per task) with summary statistics, or explicitly state as a limitation that the results reflect one response per task and may not be stable. This is load-bearing for the comparison of models such as o1 and gpt-4.1, which are both reported as perfect under the optimized prompt.
minor comments (6)
  1. [Abstract and Results] The abstract says 'great accuracy for all tasks,' while the Results say 'perfect score of 6.0 in all 13 tasks'; these are not equivalent because the rubric does not measure execution accuracy. Please use consistent and cautious wording throughout.
  2. [Author affiliations] Affiliation 2 contains the typo 'Dvision of Pharmacotherapy' and should read 'Division of Pharmacotherapy.'
  3. [Figure 2 caption] The figure caption refers to an 'optimal algorithm' when the manuscript describes an 'optimized prompt'; please align the terminology.
  4. [Table 2 footnote] The sentence 'Table 1 from reference (10) were used as the input in this rubric' is grammatically incomplete and unclear; please rewrite to explain what role Table 1 from the NONMEM tutorial plays in the rubric.
  5. [Results] The text states that 'o1 demonstrated the best overall performance' and then reports that gpt-4.1 also achieved a perfect score under the optimized prompt; since both are tied at 6.0 across all tasks, the text should explicitly say they are tied rather than implying o1 is uniquely best.
  6. [Analysis Plan] The rubric is described as 'refined and validated through expert consensus,' but no inter-rater reliability statistics or detailed criteria for Level 3 scoring are provided; a quantitative measure of agreement would strengthen the reproducibility of the scores.

Circularity Check

1 steps flagged · score 5.0 of 10

The optimized prompt is built from the same rubric used to score the outputs, so the perfect 6/6 scores partly measure prompt adherence rather than independently verified NONMEM accuracy.

  1. self definitional [Analysis Plan; Results; Table 2]
    "To explore ways to improve AI performance, we transformed the predefined rubric into an optimized prompt, embedding its criteria directly into the prompt structure. ... We compared the results to determine whether the optimized prompt could improve the accuracy of AI-generated NONMEM codes."

    The intervention is constructed from the exact instrument used to evaluate it. The optimized prompt is the Table 2 rubric rewritten as instructions, and the maximum 6/6 score is awarded for meeting the rubric's criteria: presence of the listed control-stream blocks, THETA-ETA wiring, and PREDPP conformance. A model that follows the prompt by including the requested block names and wiring patterns is therefore rewarded by a scorer using the same criteria. The reported perfect 6/6 scores for o1 and gpt-4.1 under the optimized prompt thus certify rubric adherence, not executable NONMEM accuracy; the paper describes no compilation, dataset-alignment check, or successful run.

full rationale

The paper does not rely on self-citation or imported uniqueness theorems; the benchmark and the rubric are original and anchored to NONMEM manuals. However, the central improvement claim contains a structural circularity: the optimized prompt is explicitly the rubric embedded into the prompt, and the same rubric is then the sole accuracy measure. Because the rubric rewards block presence and nominal PREDPP conformance rather than execution, a perfect score under the optimized prompt is in part a measure of how well the model echoes the scoring criteria it was handed. The impact assessment itself concedes that AI agents 'may fail to implement differential equations correctly or misapply model compartments,' failure modes the 6/6 rubric does not test. This does not destroy the paper's value as a benchmark or prompt-engineering demonstration, but it prevents the abstract's 'great accuracy' claim from being an independent validation of NONMEM code correctness. Score 5 reflects partial, instrument-level circularity rather than a fully forced derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on three premises: the rubric validly measures accuracy, NONMEM conventions from cited guides are the correctness standard, and a single response per task represents each model. The rubric weights are hand-chosen, so the 'perfect 6/6' results are relative to an instrument the authors designed; the optimized prompt embeds that same instrument, which is the circularity burden scored separately.

free parameters (2)
  • Rubric point weights = Level 1: 1 point; Level 2: 2 points; Level 3: 3 points (max 6)
    The scoring weights that define 'perfect accuracy' are hand-chosen by the authors (Table 2); all headline scores are relative to this instrument.
  • Task set composition = 13 tasks selected by authors
    The specific 13 modeling tasks in Table 1 are a design choice; the 'all 13 tasks' claims hold only within this selection.
assumptions (3)
  • domain assumption Rubric criteria are a sufficient operationalization of NONMEM code accuracy.
    Table 2 defines accuracy as presence of required control-stream blocks and THETA-ETA correspondence; no execution or fit validation is performed.
  • domain assumption NONMEM 7.4 PREDPP conventions (refs 10, 11) define correct subroutine, $DES, and $PK structure.
    Level 3 of the rubric defers correctness to PREDPP guide VI; this is an expert judgment criterion, not a machine-checked one.
  • ad hoc to paper A single LLM response per task per model is a fair measurement of that agent's capability.
    No temperature, seed, or repeated sampling is reported; the boxplots in Figure 1 summarize one score per task per model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI for NONMEM Coding in Pharmacometrics Research and Education: Shortcut or Pitfall?." pith.science (2026). https://pith.science/paper/SFXOPLFL

@misc{pith2026250708144,
  author       = {Pith},
  title        = {Pith review of: AI for NONMEM Coding in Pharmacometrics Research and Education: Shortcut or Pitfall?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SFXOPLFL}},
  note         = {Machine review of arXiv:2507.08144}
}
read the original abstract

Artificial intelligence (AI) is increasingly being explored as a tool to support pharmacometric modeling, particularly in addressing the coding challenges associated with NONMEM. In this study, we evaluated the ability of seven AI agents to generate NONMEM codes across 13 pharmacometrics tasks, including a range of population pharmacokinetic (PK) and pharmacodynamic (PD) models. We further developed a standardized scoring rubric to assess code accuracy and created an optimized prompt to improve AI agent performance. Our results showed that the OpenAI o1 and gpt-4.1 models achieved the best performance, both generating codes with great accuracy for all tasks when using our optimized prompt. Overall, AI agents performed well in writing basic NONMEM model structures, providing a useful foundation for pharmacometrics model coding. However, user review and refinement remain essential, especially for complex models with special dataset alignment or advanced coding techniques. We also discussed the applications of AI in pharmacometrics education, particularly strategies to prevent over-reliance on AI for coding. This work provides a benchmark for current AI agents performance in NONMEM coding and introduces a practical prompt that can facilitate more accurate and efficient use of AI in pharmacometrics research and education.

Figures

Figures reproduced from arXiv: 2507.08144 by the authors.

Figure 1
Figure 1. Summary of AI Agent Performance. NONMEM codes generated by different AI agents for 13 tasks were evaluated using a predefined scoring rubric. Each boxplot displays the median score (vertical line within the box) and the interquartile range. The whiskers represent the minimum and maximum scores across the 13 tasks. The top-performing AI agents are indicated with # symbol. IQR, interquartile range [PITH_FULL_IMAGE:fi… view at source ↗
Figure 2
Figure 2. AI agent performance improved with optimal algorithm in writing NONMEM [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

  1. [1]

    Check My Work?

    (1) Huang, Y . et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:250214296, (2025). (2) Fostvedt, L. et al. Stronger together: a cross-SIG perspective on improving drug development. Journal of Pharmacokinetics and Pharmacodynamics 52, 1-8 (2025). (3) Shahin, M.H., Goswami, S., Loben...

    work page 2025

  2. [7]

    Acknowledgements This work is funded by a PharmAlliance Early Career Research Award and University of North Carolina at Chapel Hill

    Ellicott City: ICON Development Solutions, 1-61 (2010). Acknowledgements This work is funded by a PharmAlliance Early Career Research Award and University of North Carolina at Chapel Hill. Table

    work page 2010

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.