Pith. sign in

REVIEW 4 major objections 7 minor 47 references

Continuous variance-aware rubric rewards lift a 14B model to near-frontier accuracy on heart medical QA under GRPO.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 19:38 UTC pith:6HV7V3EK

load-bearing objection Solid applied GRPO recipe for heart QA with real held-out gains and code; the SFT-only control is missing, so the reward-design claim is only partly isolated. the 4 major comments →

arxiv 2606.05174 v1 pith:6HV7V3EK submitted 2026-04-17 cs.CL cs.AI

Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO

Arash Ahmadi , Parisa Masnadi Khiabani , Sarah Sharif , Charles Nicholson , David Ebert , Mike Banad This is my paper
classification cs.CL cs.AI
keywords Group Relative Policy Optimizationrubric-based rewardsmedical question answeringheart-focused clinical reasoningvariance-aware reward shapingLLM-as-judgeparameter-efficient fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

General-purpose language models are hard to deploy in real clinical settings because of privacy, cost, and edge constraints, so smaller models need better post-training. This paper shows that Group Relative Policy Optimization works for heart-focused medical question answering when the reward is built from multi-criterion clinical rubrics rather than a single overall score. The key move is to turn each criterion-level pass/fail judgment into a continuous, variance-aware reward that preserves partial credit, penalizes unsafe content, and scales with how hard the rubric is. On a held-out heart subset of HealthBench, the best such reward raises a 14B base model from 0.362 to 0.502 accuracy and from 0.532 to 0.668 F1, nearly matching a much larger 120B system while running on a single workstation GPU. The same design substantially outperforms the original fixed-weight and holistic Likert aggregations from the Rubrics-as-Rewards framework. A sympathetic reader cares because the result suggests that careful reward shaping, not just model size, can close much of the gap between local medical assistants and frontier models on multi-criteria clinical tasks.

Core claim

The paper establishes that continuous analytical rewards derived from criterion-level rubric outcomes, when used inside GRPO, give richer and more stable optimization signals than either weighted binary aggregation or a single overall Likert score, and that this variance-aware design produces large gains on held-out heart-related medical question answering for a 14B model.

What carries the argument

Variance-Aware Reward Framework: continuous functions (Complexity-aware power transform with logarithmic rubric-size bonus, and Hybrid base-plus-perfection bonus) that map signed criterion pass/fail outcomes into a scalar reward with non-zero within-group variance, partial credit, negative-criteria penalties, and modest complexity awareness.

Load-bearing premise

The binary yes/no decisions of a large language-model judge on each clinical criterion are accurate and unbiased enough that optimizing against them produces real gains on physician-derived labels.

What would settle it

A blinded physician re-grading of the same 500 heart HealthBench items that shows the GRPO models do not actually satisfy more rubric criteria than the base model, or a head-to-head where the same continuous rewards are driven by human criterion labels and the accuracy lift disappears.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper post-trains Qwen3-14B with LoRA under Group Relative Policy Optimization for heart-focused medical QA, using RaR-Medicine rubrics filtered to cardiac themes and criterion-level LLM judging (GPT-OSS-120B). It proposes a Variance-Aware Reward Framework that replaces RaR Explicit (weighted binary sum) and Implicit (holistic Likert) aggregation with continuous rewards: a Complexity-aware power transform with logarithmic rubric-size bonus (Eqs. 12–15) and a Hybrid partial-credit plus perfection-bonus form (Eq. 16). After an SFT format warm-start, GRPO is run with G=6 samples per prompt. On a held-out heart-related HealthBench subset (n=500, seed 42), COMPLEXITY reaches 0.502 accuracy / 0.668 F1 versus base 0.362 / 0.532, near GPT-OSS-120B (0.508 / 0.674), with McNemar tests and RaR ablations showing large gains over RaR-Explicit/Implicit (+38.7% vs +9.4%/+13.8% relative accuracy).

Significance. If the attribution holds, the work offers a practical, hardware-constrained recipe for rubric-aligned RL on multi-criteria clinical QA where exact verifiers are unavailable: continuous partial-credit rewards that preserve within-group variance for GRPO, plus a reproducible heart-filtered pipeline with code and data released. Closing much of the gap from a 14B local model to a 120B-class baseline on HealthBench heart items is of real deployment interest for privacy-preserving medical assistants. Strengths include held-out evaluation with CIs and McNemar tests, explicit RaR ablations, training dynamics, multi-model baselines, and open artifacts. The distinctive methodological claim—that continuous variance-aware shaping beats RaR aggregation under GRPO—is the main contribution beyond applying GRPO to medical dialogue.

major comments (4)
  1. §1.2 Tables 2–3 and Abstract attribute the full Base→COMPLEXITY jump (+0.140 Acc, +0.137 F1) to variance-aware GRPO rewards, but §3.3.1 and Discussion state SFT is required for structured tags and that reported gains reflect the end-to-end pipeline. There is no SFT-only (or SFT + fixed-format decoding) row on the same n=500 HealthBench split. RaR-Explicit/Implicit ablations share the SFT warm-start and therefore isolate reward shaping from each other, not SFT from GRPO. Without an SFT-only control, the headline Base comparison conflates format/imitation learning (including MedGemma synthetic traces) with continuous reward design. Please add SFT-only metrics on the identical evaluation split and restate claims so that (i) Base→pipeline and (ii) reward-form comparisons are cleanly separated.
  2. §3.4.2–3.4.5 and Discussion rest the optimization target on binary criterion decisions from GPT-OSS-120B, justified mainly by prior RaR results and Chatbot Arena Elo, not by agreement with physicians on the heart rubrics used here. Final metrics use physician-derived HealthBench labels, which avoids pure train–eval circularity, but residual judge–label correlation and systematic bias remain unquantified. A load-bearing premise of the central claim is that optimizing these judge decisions yields genuine clinical-rubric gains. At minimum, report judge–physician (or dual-judge) agreement on a stratified sample of criteria/completions, or a sensitivity analysis swapping judge models; otherwise qualify claims as “judge-aligned rubric satisfaction” rather than clinical correctness.
  3. §3.6 and Table 2 report Accuracy/Precision/Recall/F1 against “physician-derived binary labels,” while HealthBench and the training pipeline are multi-criteria rubric systems (Fig. 2, §3.4). How binary labels are obtained from multi-criterion rubrics (overall pass threshold, majority criteria, primary criterion, etc.) is not specified. This definition is load-bearing for interpreting 0.502 accuracy and for comparing to GPT-OSS-120B/Kimi-K2. Please define the label construction explicitly and, if feasible, also report mean rubric score / criterion-level satisfaction rates so results are not solely dependent on an underspecified binarization.
  4. §3.4.4 treats α=1.2, β=0.2, λ=0.5, B=15, P=5, nmax=25 as “theory-informed design constants” without sensitivity or ablation. The paper’s distinctive claim is that the continuous Complexity/Hybrid forms (Eqs. 14–16) drive the large gap over RaR aggregation. If performance is brittle to these free parameters, that claim is overstated. A small grid or leave-one-knob study (e.g., α∈{1.0,1.2,1.5}, β∈{0,0.2,0.5}, with/without perfection bonus) on a development slice would substantially strengthen the reward-design contribution.
minor comments (7)
  1. Fig. 1 caption mentions Hybrid and Complexity formulas; the figure body also shows R_comp / R_hybrid fragments that do not fully match Eqs. 14–16—align notation.
  2. §3.2.2 uses <start working out>/<end working out> while Fig. 1 and elsewhere use <reasoning>…</reasoning>; standardize the output schema throughout.
  3. Table 1 uses w for points while §3.1 uses wk; keep a single weight symbol.
  4. Supplementary figures are heavily referenced for dataset stats and radar charts; ensure the main text is self-contained for the primary accuracy/F1 claims.
  5. Discussion cites AlphaGo as [45] after earlier [30]; check reference numbering consistency for Silver et al.
  6. Clarify whether HealthBench multi-turn structure is collapsed to single-turn prompts for evaluation, and how that interacts with heart filtering.
  7. Report wall-clock and approximate judge-call counts more precisely; 26 h/run is useful but cost of criterion-level judging is central to reproducibility claims.

Circularity Check

0 steps flagged

No circularity: held-out HealthBench metrics are independent of the training reward construction; gains are empirical, not definitional.

full rationale

The paper’s load-bearing claim is an empirical comparison: SFT+GRPO with continuous variance-aware rewards (Complexity/Hybrid) raises Qwen3-14B accuracy/F1 on a held-out heart-related HealthBench subset (n=500, physician-derived labels) relative to the base model and to RaR Explicit/Implicit aggregations. Training rewards are built from criterion-level LLM-judge binary decisions on RaR-Medicine rubrics (Eqs. 8–16), then optimized with GRPO group-normalized advantages (Eqs. 4–7). Evaluation uses a different corpus (HealthBench) and physician-derived binary labels, not the training judge scores. Hyperparameters (α, β, λ, B, P, n_max) are stated as fixed design constants, not fitted to HealthBench. There is no self-definitional loop (reward ≠ reported metric by construction), no fitted-input-called-prediction, no load-bearing self-citation uniqueness theorem, and no renaming of a known result as a derivation. Residual concerns (LLM-judge proxy quality; missing SFT-only control) are experimental-validity issues, not circularity under the stated criteria. The derivation chain is self-contained against an external benchmark.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard RL/LLM machinery plus several hand-set reward constants and domain assumptions about judge reliability and heart filtering. No new physical entities are postulated; the invented pieces are reward functional forms and the filtered training pipeline. Free parameters are design constants, not data-fitted coefficients, but the claim still depends on their particular values.

free parameters (6)
  • reward exponent α = 1.2
    Set to 1.2 in the Complexity reward to mildly sharpen high scores; hand-chosen, not learned.
  • complexity coefficient β = 0.2
    Log-rubric-size bonus strength in Eq. 12/14; hand-chosen modest value.
  • negative penalty λ = 0.5
    Scales normalized negative ratio before nonlinear transform; design constant.
  • base scale rbase / Hybrid B and P = rbase=20, B=15, P=5
    rbase=20; Hybrid splits into continuous base B=15 and perfection bonus P=5 (and 4.5=0.3B negative cap).
  • nmax rubric normalization = 25
    Maximum rubric count used in log complexity term; set to 25 from training data scale.
  • GRPO group size G and LoRA rank r = G=6, r=16
    Sampling and adapter capacity choices that affect optimization dynamics and capacity.
axioms (5)
  • domain assumption GRPO group-normalized advantages require non-degenerate within-group reward variance to produce learning signal.
    Stated as design principle in §3.3.2 and §3.4.1; standard for GRPO but load-bearing for continuous rewards.
  • domain assumption An LLM judge can independently and reliably mark each rubric criterion present/absent for medical completions.
    §3.4.2 and Discussion; justified by RaR and Arena Elo, not by new physician agreement on these heart rubrics.
  • domain assumption Heart-related filtering (MedGemma classifier / keywords) yields a coherent cardiac QA training distribution without severe label noise.
    §1.1.1 data curation; errors here would mis-scope both training and evaluation.
  • ad hoc to paper SFT is only a format warm-start; reported gains can be attributed primarily to rubric-aligned GRPO in the end-to-end pipeline.
    Discussion §2; main tables do not fully isolate SFT-only vs GRPO on the same metrics.
  • standard math Autoregressive policy factorization and clipped importance-ratio GRPO objective are valid for this generation setting.
    Eqs. 1–7; standard RLHF/GRPO machinery.
invented entities (2)
  • Variance-Aware Reward Framework (Complexity-aware and Hybrid continuous rewards) no independent evidence
    purpose: Map multi-criterion binary rubric outcomes into continuous scalar rewards with partial credit, negative penalties, and optional complexity bonus for GRPO.
    Core methodological object of the paper; defined analytically in §3.4.3–3.4.5 rather than as a physical entity.
  • Heart-filtered RaR-Medicine + synthetic MedGemma reasoning traces pipeline no independent evidence
    purpose: Create SFT/GRPO splits specialized to cardiac questions with structured reasoning tags.
    Data-construction contribution enabling the experiments; not independently validated as a clinical corpus standard.

pith-pipeline@v1.1.0-grok45 · 22513 in / 3687 out tokens · 40197 ms · 2026-07-12T19:38:43.889304+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO." pith.science (2026). https://pith.science/paper/6HV7V3EK

@misc{pith2026260605174,
  author       = {Pith},
  title        = {Pith review of: Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6HV7V3EK}},
  note         = {Machine review of arXiv:2606.05174}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) have shown strong promise in healthcare applications. Yet deploying general-purpose models in real-world settings remains difficult due to data privacy constraints, inference costs, and limited suitability for edge or on-device use. These challenges motivate the development of smaller, more efficient models that require robust post-training strategies to ensure reliable medical reasoning. In this work, we investigate Group Relative Policy Optimization (GRPO) for post-training LLMs on heart-focused medical question answering with rubric-based supervision derived from RaR-Medicine. We propose a Variance-Aware Reward Framework that extends the Explicit Aggregation and Implicit Aggregation strategies of Rubrics as Rewards by replacing weighted binary criterion aggregation and single overall Likert-style scoring with continuous analytical reward functions derived from criterion-level rubric outcomes. This formulation provides richer optimization signals for feedback that is sparse, multi-criteria, and difficult to verify automatically, and enables more stable on-policy reinforcement learning. On a held-out heart-related subset of HealthBench, our best GRPO variant improves accuracy from 0.362 to 0.502 and F1 from 0.532 to 0.668 relative to the Qwen3-14B base model, while remaining competitive with GPT-OSS-120B (0.508 accuracy, 0.674 F1). Our findings show that carefully designed rubric-based rewards provide a practical strategy for improving heart-focused medical question answering in LLMs, with potential to extend to other rubric-based tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 10 linked inside Pith

  1. [1]

    Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine25, 44–56 (2019)

    2019

  2. [2]

    & Albadawy, M

    Khalifa, M. & Albadawy, M. Artificial intelligence for clinical prediction: exploring key domains and essential functions.Computer Methods and Programs in Biomedicine Update5, 100148 (2024)

    2024

  3. [3]

    Nature(2017)

    Esteva, A.et al.Dermatologist-level classification of skin cancer with deep neural networks. Nature(2017)

    2017

  4. [4]

    Gulshan, V.et al.Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA(2016). 24

    2016

  5. [5]

    M.et al.International evaluation of an ai system for breast cancer screening

    McKinney, S. M.et al.International evaluation of an ai system for breast cancer screening. Nature(2020)

    2020

  6. [6]

    & Ranganath, R

    Huang, K., Altosaar, J. & Ranganath, R. Clinicalbert: Modeling clinical notes and predicting hospital readmission.arXiv preprint arXiv:1904.05342(2019)

    Pith/arXiv arXiv 1904

  7. [7]

    Jerfy, A.et al.The growing impact of natural language processing in public health and healthcare: A narrative review.Frontiers in Public Health(2024)

    2024

  8. [8]

    Eguia, H.et al.Clinical decision support and natural language processing in health care: A systematic review.Journal of Medical Internet Research(2024)

    2024

  9. [9]

    Liu, L.et al.Using natural language processing to extract information from clinical text for populating clinical registries: a review.Journal of the American Medical Informatics Association(2025)

    2025

  10. [10]

    J.et al.Large language models in medicine.Nature Medicine29, 1930–1940 (2023)

    Thirunavukarasu, A. J.et al.Large language models in medicine.Nature Medicine29, 1930–1940 (2023)

    1930

  11. [11]

    Naghavi, M.et al.Global burden of 292 causes of death in 204 countries and territories and 660 subnational locations, 1990–2023: a systematic analysis for the global burden of disease study 2023.The Lancet406, 1811–1872 (2025)

    1990

  12. [13]

    Li, T.et al.Cancergpt for few shot drug pair synergy prediction using large pretrained language models.NPJ Digital Medicine7, 40 (2024)

    2024

  13. [14]

    Weber, S.et al.Using a fine-tuned large language model for symptom-based depression evaluation.npj Digital Medicine8, 598 (2025)

    2025

  14. [15]

    Su, X.et al.Large language models driven neural architecture search for universal and lightweight disease diagnosis on histopathology slide images.npj Digital Medicine8, 682 (2025)

    2025

  15. [16]

    Lopez, I.et al.Clinical entity augmented retrieval for clinical information extraction, npj digital medicine 8 (2025)

    2025

  16. [17]

    Wada, A.et al.Retrieval-augmented generation elevates local llm quality in radiology contrast media consultation.NPJ Digital Medicine8, 395 (2025)

    2025

  17. [18]

    Hao, Y.et al.Personalizing prostate cancer education for patients using an ehr-integrated llm agent.NPJ Digital Medicine8, 770 (2025)

    2025

  18. [19]

    Zheng, H.et al.Kt-llm: an evidence-grounded and sequence text framework for auditable kidney transplant modeling.npj Digital Medicine(2026). 25

    2026

  19. [20]

    & Bertsimas, D

    Petridis, P., Margaritis, G., Stoumpou, V. & Bertsimas, D. Holistic ai in medicine; improved performance and explainability.npj Digital Medicine(2026)

    2026

  20. [21]

    S., Barto, A

    Sutton, R. S., Barto, A. G.et al. Reinforcement learning: An introduction, vol. 1 (MIT press Cambridge, 1998)

    1998

  21. [22]

    Watkins, C. J. & Dayan, P. Q-learning.Machine learning8, 279–292 (1992)

    1992

  22. [23]

    Wang, Y.-H., Li, T.-H. S. & Lin, C.-J. Backward q-learning: The combination of sarsa algorithm and q-learning.Engineering Applications of Artificial Intelligence26, 2184–2193 (2013)

    2013

  23. [24]

    Mnih, V.et al.Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602 (2013)

    Pith/arXiv arXiv 2013

  24. [25]

    & Silver, D

    Van Hasselt, H., Guez, A. & Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, vol. 30 (2016)

    2016

  25. [26]

    Halat, S., Ebadzadeh, M. M. & Amani, K. Modified double-dqn: addressing stability. In2024 11th International Symposium on Telecommunications (IST), 697–702 (IEEE, 2024)

    2024

  26. [27]

    & Kautz, J

    Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J. & Kautz, J. Reinforcement learning through asynchronous advantage actor-critic on a gpu.arXiv preprint arXiv:1611.06256(2016)

    Pith/arXiv arXiv 2016

  27. [28]

    & Moritz, P

    Schulman, J., Levine, S., Abbeel, P., Jordan, M. & Moritz, P. Trust region policy optimization. InInternational conference on machine learning, 1889–1897 (PMLR, 2015)

    2015

  28. [29]

    & Klimov, O

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347(2017)

    Pith/arXiv arXiv 2017

  29. [30]

    Silver, D.et al.Mastering the game of go without human knowledge.nature550, 354–359 (2017)

    2017

  30. [31]

    Jumper, J.et al.Highly accurate protein structure prediction with alphafold.Nature596, 583–589 (2021)

    2021

  31. [32]

    Shao, Z.et al.Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  32. [33]

    Guo, D.et al.Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature 645, 633–638 (2025)

    2025

  33. [34]

    & Gabbrielli, M

    Pennino, F., Raimondi, B., Rondelli, M., Gurioli, A. & Gabbrielli, M. From reasoning to code: Grpo optimization for underrepresented languages.arXiv preprint arXiv:2506.11027(2025)

    Pith/arXiv arXiv 2025

  34. [35]

    & Zhou, J

    Wang, J., Meng, F. & Zhou, J. Deeptrans: Deep reasoning translation via reinforcement learning.Transactions of the Association for Computational Linguistics14, 47–63 (2026). 26

    2026

  35. [36]

    InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 337–347 (Springer, 2025)

    Pan, J.et al.Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 337–347 (Springer, 2025)

    2025

  36. [37]

    & Ngo, C

    Pham, T.-H. & Ngo, C. Rarl: Improving medical vlm reasoning and generalization with reinforce- ment learning and lora under data and hardware constraints.arXiv preprint arXiv:2506.06600 (2025)

    Pith/arXiv arXiv 2025

  37. [38]

    Wei, J.et al.Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  38. [39]

    Zheng, L.et al.Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36, 46595–46623 (2023)

    2023

  39. [40]

    Gunjal, A.et al.Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746(2025)

    Pith/arXiv arXiv 2025

  40. [41]

    Giuffr` e, M.et al.Expert of experts verification and alignment (eval) framework for large language models safety in gastroenterology.NPJ Digital Medicine8, 242 (2025)

    2025

  41. [42]

    Croxford, E.et al.Evaluating clinical ai summaries with large language models as judges.npj Digital Medicine8, 640 (2025)

    2025

  42. [43]

    Sellergren, A.et al.Medgemma technical report.arXiv preprint arXiv:2507.05201(2025)

    Pith/arXiv arXiv 2025

  43. [44]

    K.et al.Healthbench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775(2025)

    Arora, R. K.et al.Healthbench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775(2025)

    Pith/arXiv arXiv 2025

  44. [45]

    R., Beck, A

    Granter, S. R., Beck, A. H. & Papke Jr, D. J. Alphago, deep learning, and the future of the human microscopist.Archives of pathology & laboratory medicine141, 619–621 (2017)

    2017

  45. [46]

    InForty-first International Conference on Machine Learning(2024)

    Chiang, W.-L.et al.Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning(2024)

    2024

  46. [47]

    & Leibler, R

    Kullback, S. & Leibler, R. A. On information and sufficiency.The annals of mathematical statistics22, 79–86 (1951)

    1951

  47. [48]

    Moon, S.et al.A latency processing unit: A latency-optimized and highly scalable processor for large language model inference.IEEE Micro44, 17–33 (2024). 27

    2024