Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Rubric-guided reinforcement learning lifts a 4B medical model from a 7.0 to a 27.5 HealthBench-Hard score using only 2,000 training samples, and to 37.3 with 28,000 samples—surpassing far larger open-source models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:18 UTC pith:5YOAEEUF

load-bearing objection A coherent rubric-based RL recipe with real results, but the headline gain rests on an unvalidated LLM judge, and the paper's own table shows judge choice swings scores by nearly the same amount. the 3 major comments →

arxiv 2510.15859 v5 pith:5YOAEEUF submitted 2025-10-17 cs.CL cs.AI

InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training

Pengkai Wang , Pengwei Liu , Qi Zuo , Zhijie Sang , Congkai Xie , Hongxia Yang This is my paper
classification cs.CL cs.AI
keywords reinforcement learningrubric-based rewardopen-ended medical dialogueHealthBench-HardGRPOretrieval-augmented generationLLM alignmentinstruction following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that open-ended medical dialogue can be aligned to clinical quality not with a trained reward model but with per-case rubrics: each consultation gets a tailored checklist of criteria, an off-the-shelf judge LLM checks each criterion, and the weighted sum of those verdicts becomes the reward for incremental reinforcement learning. This lets a 4B-parameter model improve from 7.0 to 27.5 on HealthBench-Hard with only 2k training samples, and to 37.3 when scaled to 28k samples—a result the paper reports as state-of-the-art among open-source models. The core idea is that transparent, multi-dimensional, case-specific criteria can replace opaque scalar rewards, making RL tractable in domains where feedback is ambiguous and where reward hacking would be dangerous. A reader should care because medical consultation has resisted straightforward RL, and this result suggests that rubric-based rewards might generalize to other open-ended tasks like instruction following.

Core claim

ORBIT's central claim is that a rubric-based reward—not a learned reward model—can drive stable policy optimization in open-ended medical dialogue. Each query is paired with a dynamically generated, case-conditioned rubric: a checklist of positive and negative criteria with integer weights. A general-purpose instruction-following judge model (Qwen3-30B-Instruct-2507) issues a binary verdict for each criterion, and the reward is the weighted sum. With this signal, GRPO training lifts Qwen3-4B-Instruct from 7.0 to 27.5 on HealthBench-Hard using 2k samples, and to 37.3 with 28k samples, overtaking the 32B-parameter Baichuan-M2 under the same GPT-4.1 evaluation protocol. The paper argues that sm

What carries the argument

The load-bearing object is the rubric-aware reward function R(q, oi) = Σ_j 1[M_judge(oi, crit_j)=True]·w_j, where each criterion is a positive or negative clinical checklist item with an integer weight. Rubrics are generated per query by retrieval-augmented prompting over a seed rubric bank (HealthBench-4k non-Hard), then filtered by pass@k so only moderately difficult queries and non-trivial criteria remain. Training uses GRPO with two stability mechanisms: a variance-aware mask that drops batches with zero reward spread, and staged entropic restarts that raise sampling temperature between stages. Together these convert a judge's verdicts into a dense, interpretable reward that steers the p

Load-bearing premise

The entire training signal is a binary per-criterion verdict issued by one general LLM judge, so if those verdicts stop tracking genuine clinical quality as the policy shifts, the reinforcement learning will optimize the judge rather than the patient.

What would settle it

Take the trained InfiMed-ORBIT-4B model and have a panel of clinicians grade a sample of 100 HealthBench-Hard responses for safety and clinical appropriateness; if the clinicians find no meaningful improvement over the base Qwen3-4B-Instruct despite the +20.5 benchmark jump, then the rubric-judged reward is rewarding compliance with the judge, not with clinical care.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, a general LLM judge plus automatic rubric generation removes the need for task-specific reward models in open-ended alignment, cutting annotation and supervision costs.
  • The 7.0→27.5 jump with only 2k samples implies that rubric-guided RL is sample-efficient, making it feasible for domains where high-quality dialogue data is scarce.
  • Scaling to 28k samples raises the score to 37.3, suggesting that rubric coverage—not just model scale—is a controllable axis of improvement in medical dialogue models.
  • The same rubric-generation pipeline applied to InfoBench raises a 4B model's hard-split score from 42.0 to 82.9, supporting the paper's claim that rubric-based feedback transfers beyond medicine.
  • The distributional shift toward high rubric pass rates implies the method changes the underlying policy, not just the sampling luck, so it may raise the ceiling for small models in other open-ended tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: replace the Qwen3-30B judge with clinician-written verdicts on a held-out set of the same rubrics; if the HealthBench-Hard gain does not track human agreement (or vanishes under a clinician judge), then the score is being optimized against the judge's preferences rather than against clinical quality.
  • The paper's reliance on a seed bank of human-crafted rubrics suggests a natural next step: generating the initial rubrics from scratch, e.g., from clinical guidelines, to test whether the RAG seed is a strict necessity or merely a convenience.
  • The staged-entropy-restart mechanism may be a generic stabilizer for rubric-RL in any sparse-reward domain; applying the same pipeline to long-horizon agentic tasks (such as multi-step diagnosis workflows) would clarify how far the recipe generalizes.
  • Because the reward is a weighted sum of binary verdicts, the method inherits any systematic blind spots of the judge; scaling the rubric pool may eventually hit a ceiling set by what the judge can recognize as correct, regardless of model capacity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ORBIT (InfiMed-ORBIT), a rubric-based incremental reinforcement-learning framework for open-ended medical dialogue. Given a medical dialogue query, ORBIT retrieves similar cases and rubrics from a seed database built from HealthBench-4k (non-Hard) rubrics, uses a general-purpose LLM (DeepSeek-R1) to synthesize case-specific rubrics, filters queries and rubrics by pass@k difficulty, and trains the policy with GRPO where the reward is a weighted sum of binary rubric-verdicts from a judge LLM (Qwen3-30B-Instruct-2507). The central empirical claim is that, with only 2k training samples, ORBIT raises Qwen3-4B-Instruct's HealthBench-Hard score from 7.0 to 27.5, and with 28k samples to 37.3, surpassing much larger open-source models under the same GPT-4.1-judged protocol. The paper also reports gains on InfoBench, ablations of the filtering and restart mechanisms, and distributional analyses of rubric-compliance. Code is made available.

Significance. If the reported results hold up, the paper makes a practically valuable contribution: it demonstrates that automatically generated, case-conditioned rubrics can serve as a dense reward signal for RL in a domain where learned reward models are expensive and fragile. The framework is data-efficient, requires no task-specific judge fine-tuning, and the ablation studies (Tables 2, 4; Figures 3-4) generally support the contribution of difficulty filtering, multi-stage restarts, and judge/generator choice. The extension to InfoBench is a useful generality check. The central weakness is that the reward signal is an unvalidated LLM judge, and the paper's own data show a large judge-induced spread; the reported improvement may therefore be partly an artifact of judge-specific stylistic conformity. The evidence for data integrity and the absence of benchmark overfitting is also thin. These concerns are concrete and addressable, so the work is worth a major revision rather than rejection.

major comments (3)
  1. [§3.3, Eq. (3); §4.1; Appendix B, Table 3] The RL reward is the weighted sum of binary verdicts from Qwen3-30B-Instruct-2507, but the paper never validates this judge against human clinical judgment or against the official GPT-4.1 protocol on a common response set. This is load-bearing because Table 3 shows the same base model scores 7.0 under GPT-4.1, 8.1 under GPT-OSS-120B, 21.4 under DeepSeek-V3, and 24.4 under Qwen2.5-72B — a 17.4-point judge spread comparable to the 20.5-point reported gain. The final GPT-4.1 evaluation only shows that the trained policy satisfies GPT-4.1 rubrics better; it does not show that the training reward tracked clinical quality. Please add a correlation analysis between the training judge's per-response rewards and GPT-4.1/expert scores on a held-out response set, or train with multiple judges and show that the final GPT-4.1 improvements are consistent.
  2. [§4.2; Appendix D; Appendix F.2.2, Table 8 caption] All headline numbers come from a single training run per configuration. The paper itself states: "all results reported here are obtained without employing a multi-restart evaluation strategy" (Appendix D) and "these results are based on a single-seed optimization trajectory" (Appendix F.2.2). GRPO with dynamic filtering and staged entropy restarts has many stochastic components and thresholds, and the reported 293% gain is surprisingly large for a 4B model. Without multiple seeds and variance reporting, the absolute scores — and the "state-of-the-art" claim — are not statistically grounded. At minimum, run 3 seeds for the 2k and 8k configurations and report mean±std; if compute limits preclude this, state it explicitly and report checkpoint-level variance.
  3. [§3.2; Appendix A.2; Appendix H] The rubric-generation pipeline uses HealthBench-4k rubrics as retrieval seeds, and the generation prompt in Appendix A.2 explicitly requires balanced coverage of exactly the five HealthBench evaluation axes (Accuracy, Completeness, Communication, Context Awareness, Instruction Following). The t-SNE visualization in Appendix H shows global separation between Consensus and Hard rubric embeddings, but it does not rule out per-case thematic overlap, rubric rephrasing, or systematic similarity along the axes used in the target benchmark. Since the final evaluation is itself a GPT-4.1 rubric-scored HealthBench protocol, the training signal is aligned with the benchmark's structure. To support the claim of genuine alignment rather than benchmark overfitting, please (i) quantify nearest-neighbor distances between generated training rubrics and Hard split rubrics, and (ii) ablate by generating ru
minor comments (5)
  1. [Tables 1, 3] Typos: "Instrcut" in Table 1; "Infer model" in Table 3; the abstract says "HealthBench score" while the experiments are on HealthBench-Hard. Please standardize.
  2. [Eqs. (2), (4)-(5); §3.3] Numerical values for the thresholds τ_low, τ_high, τ_s, τ_r, δ, γ, T_max, and per-criterion weights w_j are never given in the main text or appendix. A consolidated hyperparameter table is needed for reproducibility.
  3. [Figure 1 caption] The caption contains the raw placeholder string "database (1).png". Please remove.
  4. [Appendix C] The citation "GPT-5-Chat (Arora et al., 2025)" appears to cite the HealthBench paper for a model name. Please use a proper model/system reference or note that the model is accessed via API without a specific citation.
  5. [Section 6] The Limitations section mentions only the dependence on seed rubrics. It should also acknowledge the unvalidated judge dependence and the single-run nature of the headline results, both of which the appendices already concede.

Circularity Check

0 steps flagged

No circularity: the ORBIT reward is a proxy for, not identical to, the HealthBench-Hard evaluation signal, and no load-bearing self-citation is present.

full rationale

The paper's central result (HealthBench-Hard 7.0→27.5/37.3) is an empirical outcome of an RL loop whose reward (Eq. 3) is a weighted sum of LLM-judge verdicts on RAG-generated rubrics, while the evaluation is the official HealthBench protocol using GPT-4.1 on the official Hard rubrics. These are not the same function: the training rubrics are generated for non-Hard medical dialogues with anti-copy constraints, and Hard samples are explicitly excluded from the seed pool (§4.1; Appendix H). Although the generation prompt (Appendix A.2) asks for coverage of the same five axes used by HealthBench, this makes the reward a designed proxy for the target metric, not a mathematical reduction of the test score to the reward. The empirical dependence on the rubric generator (Appendix C, Table 4: 12.3–20.3 across generators) shows the result is contentful rather than forced by construction. No load-bearing self-citations were found: GRPO, rubrics-as-rewards, and checklist papers are external; no uniqueness theorem or ansatz is imported from the authors' prior work. The two serious concerns—the unvalidated Qwen3-30B judge (Appendix B, Table 3: the same base model scores 7.0–24.4 depending on judge) and configuration selection on HealthBench-Hard (Appendix C chooses DeepSeek-R1 after comparing on the benchmark)—are validity/overfitting risks, not circularity. The limitation passage (§6) acknowledges reliance on a seed set of human-crafted rubrics, which further confirms the derivation is empirical rather than self-definitional.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claim rests on 7 identified free hyperparameters/thresholds and 6 domain assumptions. The most consequential are the judge-verdict assumption (the RL reward is whatever the judge says it is; Appendix B documents high judge sensitivity) and the benchmark-seed assumption (training rubrics are generated from the evaluation benchmark's own rubric corpus). None of the free parameters is fitted to produce a scientific constant; they are training-recipe choices, several ablated in Table 2. This is a typical load for an empirical ML paper — the concern is the close coupling between the chosen recipe and the chosen benchmark. The paper introduces no new physical or conceptual entities; 'rubrics' are structured text artifacts produced by an LLM, not independently postulated entities.

free parameters (7)
  • Sample difficulty band τ_low, τ_high = τ_low=0; τ_high=0.75 (moderate set) or 0.5 (strict set)
    Selects which queries are kept for RL ('solvable yet challenging'); §3.2 Eq. (1) and §4.3.2 retain 1,403 / 701 of 2,082 samples. Thresholds chosen by the authors, not derived.
  • Rubric pass threshold τ_s = not specified
    A rubric is 'passed' when S(y_i,r) ≥ τ_s (Eq. 2). The value is not reported in the text even though the filter's behavior depends on it.
  • Rubric filtering threshold τ_r = 0.25 / 0.50 / 0.75 variants
    Removes rubrics with pass rate ≥ τ_r; §4.3.2 reduces 25,020 rubrics to 10,055–14,411. Ablated in Table 2; reported main results use selected variants.
  • Variance mask threshold δ = not specified
    Batches with (max_i R − min_i R) ≤ δ are masked in Eq. (4); the exact δ is not reported.
  • Entropy restart schedule γ, T_max = not specified
    Temperature reset T_{k+1} = min(T_max, T_k·γ) at stage transitions (Eq. 5); γ>1 and T_max are not quantified.
  • Per-criterion weights w_j = integers in [-10,+10] assigned by DeepSeek-R1
    The reward in Eq. (3) weights each criterion by rubric 'points' produced by the rubric generator; these weights are a free choice of the generation model and shape the reward landscape.
  • Rollout count K / group size G = K=8 (filtering), K=40 (distributional analysis); G=8
    Rollout budget and GRPO group size are chosen configurations; Figure 4 shows behavior at K=40.
axioms (6)
  • domain assumption A general-purpose LLM judge's binary per-criterion verdicts are a valid dense reward for RL
    The entire pipeline (§3.3, Eq. 3) rewards responses by Qwen3-30B-Instruct judgments of rubric satisfaction. Appendix B shows different judges assign the same base model scores from 7.0 to 24.4, so this assumption is fragile.
  • domain assumption HealthBench-4k rubrics can seed generation of Hard-worthy rubrics without leaking Hard content
    §4.1 uses only the non-Hard HealthBench subset plus lexical/semantic filtering; Appendix H defends integrity via t-SNE separation, which is weak evidence.
  • domain assumption GPT-4.1 with the official HealthBench protocol is the ground-truth evaluation
    All headline numbers are produced by GPT-4.1 as judge following the HealthBench protocol (§4.2); the protocol is taken from Arora et al. 2025, not independently validated.
  • domain assumption Intermediate-difficulty samples produce the best learning signal
    The pass@k filters (§3.2) discard trivial and unsolved queries/rubrics based on this hypothesis; Table 2 supports it empirically but it is not independently derived.
  • standard math GRPO group-relative advantage estimation is a valid policy-gradient update
    Adopted from Shao et al. 2024 (§3.3); a published method, not derived here.
  • domain assumption Synthetic multi-turn dialogues from public medical datasets are clinically realistic enough to train on
    Dialogue construction (§3.1) relies on DoctorAgent-RL test-set seeds plus IMCS21/CHIP-MDCFNPC/MedDG/ReMeDi; no clinician verification of the training dialogues is described.

pith-pipeline@v1.3.0-alltime-deepseek · 26349 in / 22272 out tokens · 179450 ms · 2026-08-04T09:18:28.226040+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has powered many recent breakthroughs in large language models (LLMs), especially for tasks where rewards can be computed automatically, such as code generation. However, it is less effective in open-ended medical dialogue, where feedback is ambiguous, context-dependent, and difficult to simply summarize into a single scalar signal-often requiring heavily supervised reward models and creating risks of reward hacking. Thus, we introduce ORBIT, an open-ended rubric-based incremental training framework tailored for critical medical dialogues. ORBIT integrates medical dialogue construction with dynamically generated case-conditioned rubrics that serve as adaptive guides for incremental RL. Unlike approaches that rely on external medical knowledge bases or handcrafted rules, ORBIT uses rubric-guided evaluation and can be implemented with general-purpose instruction-following LLMs, avoiding task-specific judge fine-tuning. With only 2k training samples, ORBIT raises Qwen3-4B-Instruct's HealthBench-Hard score from 7.0 to 27.5, achieving state-of-the-art performance among similarly sized open-source models while maintaining strong consultation quality as rubric coverage broadens.

Figures

Figures reproduced from arXiv: 2510.15859 by Congkai Xie, Hongxia Yang, Pengkai Wang, Pengwei Liu, Qi Zuo, Zhijie Sang.

Figure 1
Figure 1. Figure 1: Overview of the ORBIT pipeline for rubric-guided, incremental alignment of LLMs on open-ended medical dialogue tasks. (a) Sampling medical consultation cases, truncating dialogues to form multi-turn realistic queries. (b) During the RL stage, we adopt the GRPO framework, where a dynamic, query-specific rubric is generated for each medical case. Each response is evaluated by an external judge model based on… view at source ↗
Figure 2
Figure 2. Figure 2: Multi-dimensional performance comparison of ORBIT models. This figure presents a comprehensive performance analysis categorized by Theme and Axis. The results are divided into two groups: (a, c) Internal methodological comparisons among the base instructor-tuned model (Qwen3-4B-Instruct), and our proposed models (InfiMed-ORBIT-4B with 2k training samples, 8k samples and 28k samples). (b, d) Large-scale ben… view at source ↗
Figure 3
Figure 3. Figure 3: From left to right, the figure illustrates the effects of rubric-based filtering and sample-based filtering. For rubric filtering, we track the growth in output length during training, while for sample filtering, we measure the corresponding change in overall training time. Both approaches substantially enhance training efficiency, and adjusting the filtering thresholds provides fine-grained control over t… view at source ↗
Figure 4
Figure 4. Figure 4: Distributional Analysis of Performance Metrics: Inference Scaling vs. Rubric-RL Alignment. We compare the off-the-shelf Qwen3-4B-Instruct baseline (at K = 8 and K = 40 rollouts) with our proposed InfiMed-ORBIT-4B (at K = 40 rollouts). From left to right, the subplots depict Kernel Density Estimates (KDE) and histograms for: (a) Average Normalized Score per query, (b) Max Normalized Score (Best-of-K), (c) R… view at source ↗
Figure 5
Figure 5. Figure 5: Overall details of the designed system prompt for the rubrics generation pipeline. A.2. Rubrics Generation Prompt A central challenge in this work lies in designing a task-specific prompt for each medical case that can effectively guide the model in generating highly contextualized and clinically grounded rubrics. To address this, we leverage the 5,000 curated rubrics from HealthBench as a seed dataset and… view at source ↗
Figure 6
Figure 6. Figure 6: Overall details of the designed rubrics generation prompt for the input medical case. Retrieved reference cases from the RAG system are used as in-context examples to guide a rubric-generation model in producing multi-dimensional scoring rubrics with both positive and negative criteria. 2 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance comparison of Average Normalized Scores under varying inference rollouts. The evaluation is conducted on the core experimental dataset comprising 2k multi-turn queries. We benchmark the Qwen3-4B-Instruct with K = 8 and K = 40 rollouts against the aligned InfiMed-4B model (K = 40). 0.0 0.2 0.4 0.6 0.8 1.0 Average Normalized Score (per query) 0 1 2 3 4 5 6 7 Normalized Frequency (a) Qwen3_4B: 8 r… view at source ↗
Figure 8
Figure 8. Figure 8: Distributional Analysis of Best-of-K Scores (K = 40). We compare the maximum normalized score per query achieved by the Qwen3-4B-Instruct baseline and InfiMed-ORBIT-4B. Moving beyond coarse-grained query-level scoring, we delve into the fine-grained dynamics of the Rubric-RL process by analyzing the pass rates of individual constraints. We maintain the same experimental setup (K = 40 rollouts) and employ t… view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of Average Rubric Pass Rates (K = 40). This subplot illustrates the average probability of satisfying a rubric across 40 rollouts for each query. Observation: InfiMed-ORBIT-4B (purple) exhibits a significant rightward shift compared to the Qwen3-4B-Instruct baseline (green and blue), indicating a higher and more consistent level of rubric compliance across the dataset. 0.0 0.2 0.4 0.6 0.8 1.0 … view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of Rubric Hit Rates (Best-of-N, N = 40). This subplot measures the proportion of rubrics that are satisfied (or penalties avoided) at least once within 40 rollouts for each query. Observation: InfiMed-ORBIT-4B (purple) demonstrates a substantially expanded hit rate boundary, with a high density of queries achieving near-perfect compliance, showcasing its enhanced capability to explore and dis… view at source ↗
Figure 11
Figure 11. Figure 11: The case study for Qwen3-4B-Instruct and our InfiMed-ORBIT-4B for the input medicine consultation dialogue problem. 10 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: t-SNE Visualization of Rubric Embeddings for HealthBench-Consensus. The embedding distribution forms dense, cohesive clusters, reflecting a high degree of semantic homogeneity. This indicates that the constraints in the Consensus subset share substantial structural similarities, which explains the stronger baseline performance and underscores their suitability as low-risk in-context exemplars. 11 [PITH_F… view at source ↗
Figure 13
Figure 13. Figure 13: t-SNE Visualization of Rubric Embeddings for HealthBench-Hard. In stark contrast to the Consensus subset, the Hard rubrics exhibit a sparse and highly fragmented topology. The pronounced dispersion across the semantic manifold reflects their semantic independence and structural complexity, quantitatively validating the elevated reasoning difficulty of this benchmark. 12 [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment

    cs.CY 2026-05 unverdicted novelty 6.0

    Scoping review of 134 studies on LLM-as-a-Judge in healthcare finds concentration in clinical decision support and NLP, frequent use of OpenAI models with prompt engineering, and moderate-to-strong human alignment whe...

Reference graph

Works this paper leans on

40 extracted references · 26 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Language models that think, chat better.arXiv preprint arXiv:2509.20357,

    Adithya Bhaskar, Xi Ye, and Danqi Chen. Language models that think, chat better.arXiv preprint arXiv:2509.20357,

    arXiv

  3. [5]

    Ace-rl: Adaptive constraint-enhanced reward for long-form generation re- inforcement learning.arXiv preprint arXiv:2509.04903,

    Jianghao Chen, Wei Sun, Qixiang Yin, Lingxing Kong, 9 InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training Zhixing Tan, and Jiajun Zhang. Ace-rl: Adaptive constraint-enhanced reward for long-form generation re- inforcement learning.arXiv preprint arXiv:2509.04903,

    arXiv

  4. [6]

    Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925,

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925,

    Pith/arXiv arXiv

  5. [7]

    Sft memorizes, rl generalizes: A com- parative study of foundation model post-training.arXiv preprint arXiv:2501.17161,

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A com- parative study of foundation model post-training.arXiv preprint arXiv:2501.17161,

    Pith/arXiv arXiv

  6. [8]

    Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  7. [9]

    Ultrafeedback: Boosting lan- guage models with scaled ai feedback.arXiv preprint arXiv:2310.01377,

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingx- iang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting lan- guage models with scaled ai feedback.arXiv preprint arXiv:2310.01377,

    Pith/arXiv arXiv

  8. [10]

    Multichallenge: A realistic multi-turn con- versation evaluation benchmark challenging to frontier llms

    Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn con- versation evaluation benchmark challenging to frontier llms. InFindings of the Association for Computational Linguistics: ACL 2025, pages 18632–18702,

    2025

  9. [11]

    Qa-lign: Aligning llms through constitution- ally decomposed qa.arXiv preprint arXiv:2506.08123,

    Jacob Dineen, Aswin RRV , Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, et al. Qa-lign: Aligning llms through constitution- ally decomposed qa.arXiv preprint arXiv:2506.08123,

    arXiv

  10. [12]

    Baichuan-m2: Scaling medi- cal capability with large verifier system.arXiv preprint arXiv:2509.02208,

    Chengfeng Dou, Chong Liu, Fan Yang, Fei Li, Jiyuan Jia, Mingyang Chen, Qiang Ju, Shuai Wang, Shunya Dang, Tianpeng Li, et al. Baichuan-m2: Scaling medi- cal capability with large verifier system.arXiv preprint arXiv:2509.02208,

    Pith/arXiv arXiv

  11. [14]

    Rubrics as rewards: Re- inforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Re- inforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Pith/arXiv arXiv

  12. [15]

    m1: Unleash the potential of test-time scaling for medical reasoning with large language models

    Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models. arXiv preprint arXiv:2504.00869,

    arXiv

  13. [16]

    Compute as teacher: Turning inference compute into reference-free supervision.arXiv preprint arXiv:2509.14234,

    Dulhan Jayalath, Shashwat Goel, Thomas Foster, Parag Jain, Suchin Gururangan, Cheng Zhang, Anirudh Goyal, and Alan Schelten. Compute as teacher: Turning inference compute into reference-free supervision.arXiv preprint arXiv:2509.14234,

    Pith/arXiv arXiv

  14. [17]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001,

    Pith/arXiv arXiv

  15. [18]

    Agent hospital: A simulacrum of hospital with evolvable medical agents.arXiv preprint arXiv:2405.02957,

    Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, et al. Agent hospital: A simulacrum of hospital with evolvable medical agents.arXiv preprint arXiv:2405.02957,

    Pith/arXiv arXiv

  16. [19]

    Wild- bench: Benchmarking llms with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770,

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wild- bench: Benchmarking llms with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770,

    Pith/arXiv arXiv

  17. [20]

    Beyond distillation: Pushing the limits of medical llm reasoning with minimalist rule-based rl

    10 InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training Che Liu, Haozhe Wang, Jiazhen Pan, Zhongwei Wan, Yong Dai, Fangzhen Lin, Wenjia Bai, Daniel Rueckert, and Rossella Arcucci. Beyond distillation: Pushing the limits of medical llm reasoning with minimalist rule-based rl. arXiv preprint arXiv:2505.17952, 2025a....

    Pith/arXiv arXiv

  18. [22]

    Large language models: A survey.arXiv preprint arXiv:2402.06196,

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jian- feng Gao. Large language models: A survey.arXiv preprint arXiv:2402.06196,

    Pith/arXiv arXiv

  19. [23]

    Infobench: Evaluating instruc- tion following ability in large language models.arXiv preprint arXiv:2401.03601,

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sang- woo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruc- tion following ability in large language models.arXiv preprint arXiv:2401.03601,

    Pith/arXiv arXiv

  20. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  21. [25]

    Rl’s razor: Why online reinforcement learning forgets less.arXiv preprint arXiv:2509.04259,

    Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. Rl’s razor: Why online reinforcement learning forgets less.arXiv preprint arXiv:2509.04259,

    Pith/arXiv arXiv

  22. [26]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Pith/arXiv arXiv

  23. [27]

    Reasonmed: A 370k multi-agent gen- erated dataset for advancing medical reasoning

    Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai, et al. Reasonmed: A 370k multi-agent gen- erated dataset for advancing medical reasoning. InPro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26457–26478,

    2025

  24. [28]

    Medagents: Large language models as collab- orators for zero-shot medical reasoning

    Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collab- orators for zero-shot medical reasoning. InFindings of the Association for Computational Linguistics ACL 2024, pages 599–621,

    2024

  25. [29]

    Large language models in medicine

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Ka- bilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940,

    1930

  26. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  27. [31]

    Check- lists are better than reward models for aligning language models.arXiv preprint arXiv:2507.18624,

    Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Check- lists are better than reward models for aligning language models.arXiv preprint arXiv:2507.18624,

    arXiv

  28. [32]

    Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs

    Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993,

    Pith/arXiv arXiv

  29. [33]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  30. [34]

    Dapo: An open-source llm re- inforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xi- aochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm re- inforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  31. [35]

    Qwen3 embedding: Ad- vancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Day- iheng Liu, Junyang Lin, et al. Qwen3 embedding: Ad- vancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Pith/arXiv arXiv

  32. [36]

    Group sequence policy optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071,

    Pith/arXiv arXiv

  33. [37]

    Ask patients with patience: Enabling llms for human-centric medical dialogue with grounded reasoning

    Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu, Fenglin Liu, and Junde Wu. Ask patients with patience: Enabling llms for human-centric medical dialogue with grounded reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2846– 2857,

    2025

  34. [38]

    While these components provide explicit procedural guidance and high-quality reference examples, the intrinsic capability of the underlying generation model remains a decisive factor influencing the quality of the resulting rubrics. To systematically evaluate this effect, we benchmark four state-of-the-art large language models on the rubric generation ta...

    2025

  35. [39]

    This subplot measures the proportion of rubrics that are satisfied (or penalties avoided) at least once within 40 rollouts for each query

    Distribution of Rubric Hit Rates (Best-of-N, N= 40 ). This subplot measures the proportion of rubrics that are satisfied (or penalties avoided) at least once within 40 rollouts for each query. Observation: InfiMed-ORBIT-4B (purple) demonstrates a substantially expanded hit rate boundary, with a high density of queries achieving near-perfect compliance, sh...

    2025

  36. [40]

    For all other models, the generation parameters were set to align with those specified in the official HealthBench protocol

    To enhance the evaluation stability, a low temperature of 0.1 was used for the Qwen series and other selected open-source models. For all other models, the generation parameters were set to align with those specified in the official HealthBench protocol. Table 6.Evaluation Model Parameter Settings Models Temperature top-p max token API Type GPT-4.1 0.5 – ...

    2023

  37. [2022]

    A generalist medical language model for disease diagnosis assistance.Nature medicine, 31(3):932–942, 2025e

    Xiaohong Liu, Hao Liu, Guoxing Yang, Zeyu Jiang, Shuguang Cui, Zhaoze Zhang, Huan Wang, Liyuan Tao, Yongchang Sun, Zhu Song, et al. A generalist medical language model for disease diagnosis assistance.Nature medicine, 31(3):932–942, 2025e. Meng Lu, Ho Brandon, Ren Dennis, and Xuan Wang. Triageagent: Towards better multi-agents collaborations for large lan...

    2024

  38. [2023]

    gpt-oss-120b & gpt- oss-20b model card.arXiv preprint arXiv:2508.10925,

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt- oss-20b model card.arXiv preprint arXiv:2508.10925,

    Pith/arXiv arXiv

  39. [2024]

    Doctoragent-rl: A multi-agent collaborative reinforce- ment learning system for multi-turn clinical dialogue

    Yichun Feng, Jiawei Wang, Lu Zhou, Zhen Lei, and Yixue Li. Doctoragent-rl: A multi-agent collaborative reinforce- ment learning system for multi-turn clinical dialogue. arXiv preprint arXiv:2505.19630,

    Pith/arXiv arXiv

  40. [2025]

    URL https: //hkunlp.github.io/blog/2025/Polaris. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Pre- ston Bowman, Joaquin Qui˜nonero-Candela, Foivos Tsim- pourlas, Michael Sharman, Meghan Shah, Andrea Val- lone, Alex Beutel, et al. Healthbench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775,

    Pith/arXiv arXiv 2025