Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Hermes 4 is an open-weight family of hybrid reasoning models, trained on a fully documented synthetic-data pipeline, reporting scores that sit alongside top open-weight systems in math, code, knowledge, and alignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Hermes 4 releases three open-weight reasoning models (14B, 70B, 405B) trained with synthetic data and a length-control SFT stage, evaluated on mathematics, code, knowledge, and alignment benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Hermes 4 report has a genuinely interesting length-control trick and a thorough data pipeline writeup; the missing decontamination audit is the one thing that keeps me from trusting the headline benchmark claims. the 5 major comments →

arxiv 2508.18255 v2 pith:K3L4R46C submitted 2025-08-25 cs.AI

Hermes 4 Technical Report

Ryan Teknium , Roger Jin , Jai Suphavadeeprasit , Dakota Mahan , Jeffrey Quesnelle , Joe Li , Chen Guang , Shannon Sands
show 1 more author
Karan Malhotra
This is my paper
classification cs.AI
keywords hybrid reasoning modelssynthetic data generationDataForgerejection samplinglength-control fine-tuningLLM-as-a-judgeopen-weight modelsreasoning-length truncation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The report sets out to establish that a transparent, reproducible post-training recipe can turn open-weight base models into hybrid reasoners whose scores approach the best open systems. Its load-bearing claim is methodological: a graph-based synthetic data pipeline (DataForge) with LLM-judge iteration, rejection sampling against roughly a thousand verifiers, and a second supervised stage that teaches the model to emit its closing thinking tag at a 30,000-token budget. The length-control stage is the sharpest concrete result: on the 14B model it cuts the fraction of generations that never stop reasoning from as high as 60% to 0.1%, at a cost of at most a few relative points on reasoning benchmarks (and a large gain on LiveCodeBench). If the recipe is right, open-weight teams can reproduce frontier-adjacent reasoning while keeping inference cost under control.

Core claim

On the paper's own terms, Hermes 4 is the claim that one training family—14B, 70B, and 405B—can combine structured multi-turn reasoning with broad instruction-following, and that the recipe is fully disclosed. The 405B model reports 81.9 on AIME'24, 78.1 on AIME'25, 61.4 on LiveCodeBench v6, 70.6 on GPQA Diamond, 87.2 on MMLU, 93.7 on Arena-Hard v1, 73.0 on RewardBench, and 85.5 on EQBench3; the 14B model, after length-control tuning, drops its overlong rate from 60% to 0.1% on LiveCodeBench-style reasoning while holding or improving scores. The paper further argues, from structured qualitative probes, that Hermes 4 shows unusually high behavioral plasticity: it stays in character in role-pl

What carries the argument

DataForge—a graph-based synthetic data generator whose nodes implement a PDDL-style preconditions/postconditions interface, so random walks through the graph synthesize instruction-answer pairs that an LLM judge grades and iterates on. The second carrying mechanism is think-token-only length-control SFT: a fine-tuning stage that masks every token except the closing </think> (and <eos>), teaching the model to terminate reasoning at a fixed token budget while leaving its reasoning distribution untouched.

Load-bearing premise

The report's quality and comparison numbers assume that LLM judges give valid cross-model signals and that a few hand-picked generations fairly represent each model; if judges are biased by style or provenance, or the examples are cherry-picked, the alignment, writing, and behavioral-plasticity claims do not stand.

What would settle it

Re-grade a random sample of Arena-Hard, EQBench3, and RefusalBench responses from the released logs with human raters or a different judge model: if the Hermes-versus-frontier rankings invert, or the refusal-rate gap shrinks to near zero, the judged comparisons are artifacts. Separately, hash-check the AIME'24/AIME'25 and LiveCodeBench prompts against the training corpus; any contamination would invalidate the reasoning scores.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The 30k-token think budget cuts overlong rates on AIME'24/AIME'25, GPQA Diamond, and LiveCodeBench from between 18% and 60% to at most 0.2%, with score changes between −3.9% and +48.6% relative.
  • Reasoning and non-reasoning data in a single hybrid dataset lets one checkpoint serve both structured multi-turn reasoning and general instruction following.
  • Because DataForge graphs are themselves nodes, the generation pipeline composes to arbitrary nesting depth, letting new task families be synthesized without new human annotation.
  • Training on the intermediate DataForge generation and judging calls gives the model measurable skill as an instruction generator and judge, which the authors say is valuable for future rejection sampling.
  • Public weights and released evaluation logs make the headline numbers independently checkable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single-token masked supervision is a generic knob: the same trick could set a 10k, 50k, or per-domain thinking budget on other base models; the paper only demonstrates 20k vs 30k on a 14B model.
  • The judged benchmarks (Arena-Hard, EQBench, RefusalBench, and the DataForge rubric grades) all lean on LLM judges, so the reported deltas may partly reflect stylistic preference rather than capability; human re-grading is the obvious test.
  • Training on intermediate generation/judging calls hints at a self-improvement loop—using Hermes 4 itself to synthesize and grade the next generation's data—that the report does not close.
  • Because the length-control stage leaves the reasoning distribution untouched and only teaches a stopping decision, an inference-time early-exit system might achieve the same overlong-rate reductions without any extra fine-tuning; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces Hermes 4, a family of hybrid reasoning models (405B and 70B from Llama 3.1, 14B from Qwen3) trained with a synthetic-data pipeline (DataForge) seeded from DCLM/FineWeb, rejection sampling against roughly 1,000 verifiers (Atropos), and a two-stage SFT that includes a length-control fine-tuning stage in which only the </think> termination token (and <eos>) are trained. The stated contributions are the data curation/synthesis strategy, the training methodology, and a broad evaluation across math, code, knowledge, alignment, and qualitative behavior. The central claim, made in the abstract and Section 4.6, is that Hermes 4 is comparable to frontier open-weight systems on standard benchmarks, with Table 3 reporting, for the 405B model, AIME'24 81.9, AIME'25 78.1, GPQA Diamond 70.6, LiveCodeBench v6 61.4, MMLU 87.2, MMLU-Pro 80.6, and Arena-Hard v1 93.7. The paper also reports several distinctive qualitative behaviors in Section 5.

Significance. If the quantitative results hold, the paper is a useful open recipe: it provides released weights, released evaluation logs, open implementations of the evaluation harness, and a carefully controlled experiment on reasoning-length truncation (Section 3.1 and Appendix B) that goes beyond most technical reports. The masking control run is a methodological strength. However, the central 'comparable to frontier open-weight models' claim rests on benchmark numbers that are not accompanied by a decontamination audit for the 5M-sample synthetic training set, and the point estimates are reported without confidence intervals. The qualitative section makes broad behavioral claims from single demonstrations. These issues are fixable, and the paper's transparency (release of weights, eval logs, and harness code) is a genuine asset that should be preserved.

major comments (5)
  1. [§2, §3.1.1, §4.6, Tables 3–4] No decontamination audit is reported between the training data and the evaluation benchmarks. The pipeline starts from DCLM/FineWeb seeds (Section 2.1.1), generates synthetic QA via DataForge (Section 2.1.2), rejection-samples against ~1,000 verifiers including Internbootcamp (Sections 2.2–2.2.3), and then uses WebInstruct-Verified, rSTAR-Coder, and DeepMath for the second SFT stage (Section 3.1.1). None of these stages is described as excluding AIME'24/'25, GPQA Diamond, or LiveCodeBench v6, and the 5M-sample training set is not released. Because the base Llama/Qwen checkpoints may also have seen these benchmarks, that part affects all baselines equally, but the Hermes-specific pipeline adds a large corpus of possibly benchmark-derived reasoning traces. This makes the objective scores in Tables 3 and 4—the core evidence for the central claim—impossible to verify externally. The authors
  2. [§4.5, Tables 3–4] Pass@1 scores are reported as point estimates without confidence intervals or standard errors, despite using finite sample counts: 64 samples for AIME, 8 for GPQA Diamond, 16 for LiveCodeBench, 4 for MATH-500, and 3 for EQBench3. Several cross-model differences are small (e.g., Hermes 405B AIME'24 81.9 vs Qwen3 235B 78.2; MMLU 87.2 vs Cogito 91.4; RewardBench 73.0 vs Qwen3 74.2). Without uncertainty quantification, the 'comparable to frontier systems' claim overreaches the evidence for borderline comparisons. At minimum, the authors should report standard errors or binomial confidence intervals for the finite-sample pass@1 estimates.
  3. [§5 and Appendix C] The qualitative behavioral claims—for example, that Hermes 4 exhibits 'higher behavioral plasticity' and that its responses are 'more readily shaped by system-level cues'—are based on single generations per condition in Appendix C. There is no systematic sampling, no multiple independent runs, no scoring rubric, and no inter-annotator or LLM-judge quantification. These claims are presented as a contribution of the paper, so they should be either supported with a small systematic study or explicitly reframed as illustrative observations rather than comparative findings.
  4. [§3.1 and Appendix B] The statement that training only on the </think> token 'leaves the model's reasoning distribution unchanged' is too strong and is contradicted by the paper's own experiments. Appendix B reports that the 20k-token </think>-only masking condition reduces AIME'24 by 20 points relative to Stage 1, and Table 2 shows the 30k-tuned 14B model improving LiveCodeBench by 48.6% while changing AIME'25 by −3.9%. These are not 'minimal and targeted' effects. The paper should temper the claim about preserving the reasoning distribution and characterize the conditions under which stopping-token training changes reasoning behavior.
  5. [§4.5.1, Tables 3–4, Figure 4] RefusalBench is an internally built benchmark: 166 hand-crafted prompts across 32 categories, labeled by Sonnet 4 as judge. It is used in the main comparison tables as an alignment metric, but the dataset is not released, the judge is a closed model with unstated agreement properties, and no validation against human labels is reported. This is load-bearing for the alignment-related comparisons (e.g., Hermes 405B RefusalBench 57.1 vs Qwen3 34.3), so the authors should release the prompts, report human agreement on a sample, and/or provide a sensitivity analysis with a different judge.
minor comments (4)
  1. [General] There are several typos and reference issues: 'Appendex A' should be 'Appendix A'; reference [35] (OpenAI) lacks a title; references [62] and [63] are identical; the caption of Tables 3 and 4 should define 'R' and 'N' (reasoning and non-reasoning modes) explicitly.
  2. [§4.5] The pass@1 sample counts are said to be taken from lighteval, with the note that some 'may no longer be the current defaults.' The paper should pin down the exact versions of lighteval and Atropos used and record the sample counts per benchmark in a table, so the results are reproducible as the software evolves.
  3. [§2.1] DataForge is described as inspired by AgentInstruct, but the citation [31] points to AgentInstruct, not to DataForge itself. If DataForge is a new tool, its own release (or repository) should be cited; if it is a reimplementation, that should be stated.
  4. [§3.1.2 / §3.1.3] The statement in Section 3.1 that the 14B model 'would reach its maximum context of 40,960 tokens 60% of the time' is supported by Table 2's overlong rates only for LiveCodeBench (60.0%); the other benchmarks show lower Stage-1 overlong rates (18–28%). Clarify which benchmark or aggregate the 60% figure refers to.

Circularity Check

0 steps flagged

No circular derivation found: benchmark claims rest on external fixed datasets, and self-citations are non-load-bearing infrastructure/provenance.

full rationale

The report's central claim (abstract; Tables 3-4) is that Hermes 4 is comparable to frontier open-weight systems. The numbers supporting that claim (AIME'24/'25, GPQA Diamond, LiveCodeBench v6, MMLU(-Pro), DROP, etc.) are measured against fixed external evaluation datasets implemented in lighteval and Atropos, not produced by fitting a parameter to those datasets. None of the paper's equations or training objectives define an evaluation score in terms of a fitted input; for example, the length-control SFT (Sections 3.1 and Appendix B) is an engineering intervention evaluated by before/after benchmark deltas, and the benchmark values are not inputs to that fit. Self-citations appear (Hermes 3 dataset retained [51], Atropos [27] for rejection sampling and evaluation, DeepHermes [50] for trajectory generation, and a Nous colleague [33] for reasoning-length commentary), but none functions as a uniqueness theorem or as the sole justification for the comparative claim; Atropos is open-source infrastructure and the external benchmarks remain independent. RefusalBench (Section 4.5.1) is an internally constructed benchmark with LLM-judge labels and is self-assessment, but it is a measured quantity, not a derivation-by-construction, and it is not the load-bearing evidence for the headline comparison. The qualitative behavior section (Section 5) is anecdotal and could be cherry-picked, but that is an evidence-quality concern, not circularity. The most serious validity concern—absence of a reported decontamination audit between the roughly 5M-sample synthetic corpus (Section 2) and the exact evaluation benchmarks—is a contamination/external-validity risk, not circularity: even if overlap existed, it would mean the scores overstate generalization, not that the scores are equivalent to the training inputs by construction. No step in the paper reduces, by its own equations or by a load-bearing self-citation, to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central results rest on assumptions about LLM-judge validity, verifier correctness, the benign effect of selective loss masking, and fair evaluation settings. The 30k thinking budget and evaluation sampling parameters are hand-chosen, and the invented tools DataForge and RefusalBench are not independently validated.

free parameters (5)
  • thinking_token_budget = 30,000 tokens
    Chosen after a 20k budget caused up to 20-point AIME regression; final budget balances benchmark scores and overlong rates (Section 3.1.3, Appendix B).
  • pass@1 sample counts = 64 (AIME), 8 (GPQA), 16 (LCB), 4 (MATH-500), 3 (EQBench/CreativeWriting)
    Chosen by authors; affect variance of reported scores. No confidence intervals are reported for these estimates (Section 4.5).
  • evaluation context lengths = 40,960 (reasoning/code), 32,768 (others)
    Chosen by authors to match model capabilities; affects measured performance (Section 4.5).
  • sampling temperature/top-p/top-k = 0.6 / 0.95 / 20 (default)
    Following Qwen3 settings, with deviations for DeepSeek models; these choices affect benchmark outcomes (Section 4.5).
  • dedup cosine threshold = 0.7
    Chosen for semantic deduplication of seed data; no ablation is provided (Section 2.1.1).
axioms (4)
  • domain assumption LLM judges provide valid quality signals for synthetic data filtering, and using different weights from the generator avoids self-preference.
    DataForge grades samples with LLM judges; this is load-bearing for the quality of the 5M-sample dataset (Section 2.1.2).
  • domain assumption Rejection sampling verifiers in Atropos produce correct training targets.
    The reasoning trajectories used for training are assumed correct based on verifier reward signals (Section 2.2).
  • ad hoc to paper Training only on the think-termination token leaves the model's reasoning distribution unchanged while teaching a counting behavior.
    The length-control stage relies on this mechanism; the authors themselves do not fully understand why masking matters (Section 3.1, Appendix B).
  • domain assumption Evaluation sampling settings and LLM-judge settings are fair across all compared models.
    Comparisons use shared settings except where providers differ; any hidden bias changes the rankings (Section 4.5).
invented entities (2)
  • DataForge no independent evidence
    purpose: Graph-based synthetic data generator that transforms pre-training passages into instruction-answer pairs.
    New tool introduced in this paper; no external validation or benchmark demonstrates its superiority over other synthesis methods.
  • RefusalBench no independent evidence
    purpose: Internal benchmark measuring how often a model refuses requests across 32 categories and 166 handcrafted prompts.
    Constructed by the authors and judged by Sonnet 4; no external validation or inter-annotator agreement is reported (Section 4.5.1).

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Hermes 4 Technical Report." pith.science (2026). https://pith.science/paper/K3L4R46C

@misc{pith2026250818255,
  author       = {Pith},
  title        = {Pith review of: Hermes 4 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3L4R46C}},
  note         = {Machine review of arXiv:2508.18255}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present Hermes 4, a family of hybrid reasoning models that combine structured, multi-turn reasoning with broad instruction-following ability. We describe the challenges encountered during data curation, synthesis, training, and evaluation, and outline the solutions employed to address these challenges at scale. We comprehensively evaluate across mathematical reasoning, coding, knowledge, comprehension, and alignment benchmarks, and we report both quantitative performance and qualitative behavioral analysis. To support open research, all model weights are published publicly at https://huggingface.co/collections/NousResearch/hermes-4-collection-68a731bfd452e20816725728

Figures

Figures reproduced from arXiv: 2508.18255 by Chen Guang, Dakota Mahan, Jai Suphavadeeprasit, Jeffrey Quesnelle, Joe Li, Karan Malhotra, Roger Jin, Ryan Teknium, Shannon Sands.

Figure 1
Figure 1. Figure 1: An example DataForge [31] data flow. A piece of pre-training seed data is passed through a series of randomly-selected nodes to synthesize an instruction-answer pair. Three stages are depicted here - passage transformation (e.g. writing a debate transcript inspired by a news article from DCLM [22]), instruction generation conditioned on the transformed passage, and answer generation. While any transformed … view at source ↗
Figure 2
Figure 2. Figure 2: Training loss 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 0.4 0.45 0.5 0.55 0.6 0.65 Training Step Loss 70B 405B To train Hermes 4 we use a modified5 version of TorchTitan [24]. As with Hermes 3, we began with the 405B and 70B versions of Llama 3.1 [49]. For the 14B model, we began with the Qwen3 14B [57] checkpoint. 5 https://github.com/NousResearch/torchtitan/tree/856a0ecabeb8a882c150641f73f8… view at source ↗
Figure 4
Figure 4. Figure 4: RefusalBench scores for various models. Average of 5 runs. Higher scores means less [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning

    cs.CL 2026-06 unverdicted novelty 6.0

    CHASE uses co-evolutionary RL with GRPO to harden LLMs against black-box prompt-rewriting attacks, cutting mean StrongREJECT scores by 43.2% on held-out families while keeping zero false refusals on benign prompts.

Reference graph

Works this paper leans on

107 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https://huggingface

    deepseek-ai/DeepSeek-R1-0528 · Hugging Face, August 2025. URL https://huggingface. co/deepseek-ai/DeepSeek-R1-0528 . [Online; accessed 13. Aug. 2025]

    2025

  2. [2]

    Axolotl: Post-training for ai models, 2023

    Axolotl maintainers and contributors. Axolotl: Post-training for ai models, 2023. URL https://github.com/axolotl-ai-cloud/axolotl

    2023

  3. [3]

    Scaling synthetic data creation with 1,000,000,000 personas, 2024

    Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, et al. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https://arxiv.org/abs/2406.20094

    Pith/arXiv arXiv 2024

  4. [4]

    The lean 4 theorem prover and programming language

    Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In Automated Deduction – CADE 28 , volume 12699 of Lecture Notes in Computer Science, pages 625–635. Springer, 2021. doi:10.1007/978-3-030-79876-5_37

    doi:10.1007/978-3-030-79876-5_37 2021

  5. [5]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412. 19437

    2024

  6. [6]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, et al. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024. 13 Hermes 4 Technical Report

    Pith/arXiv arXiv 2024

  7. [7]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, et al. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https: //arxiv.org/abs/1903.00161

    Pith/arXiv arXiv 2019

  8. [8]

    Scaling Synthetic Data Creation with 1,000,000,000 Personas

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, et al. Scaling Synthetic Data Creation with 1,000,000,000 Personas. arXiv, June 2024. doi:10.48550/arXiv.2406.20094

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.20094 2024

  9. [9]

    Knoblock, Drew McDermott, et al

    Malik Ghallab, Adele Howe, Craig A. Knoblock, Drew McDermott, et al. Pddl – the planning domain definition language. Technical Report CVC-TR-98-003/DCS-TR-1165, Yale Center for Computational Vision and Control, October 1998. Available at https://www.cs.cmu.edu/ ~mmv/planning/readings/98aips-PDDL.pdf

    1998

  10. [10]

    Openthoughts: Data recipes for reasoning models, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, et al. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxiv.org/abs/2506.04178

    Pith/arXiv arXiv 2025

  11. [11]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Nathan Habib, Clémentine Fourrier, Hynek Kydlí ˇcek, Thomas Wolf, et al. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/ lighteval

    2023

  12. [12]

    DeepMath-103K: A Large-Scale, Chal- lenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.arXiv, April 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, et al. DeepMath-103K: A Large-Scale, Chal- lenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.arXiv, April 2025. doi:10.48550/arXiv.2504.11456

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.11456 2025

  13. [13]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, et al. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  14. [14]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, et al. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    Pith/arXiv arXiv 2021

  15. [15]

    I., Daya Guo, Dejian Yang, Haowei Zhang, et al

    DeepSeek-A. I., Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incen- tivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv, January 2025. doi:10.48550/arXiv.2501.12948

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  16. [16]

    Teknium"

    "Teknium" "interstellarninja". Hermes-function-calling-dataset-v1. URL https:// huggingface.co/NousResearch/hermes-function-calling-v1

  17. [17]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv, March 2024. doi:10.48550/arXiv.2403.07974

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.07974 2024

  18. [18]

    Johnson, Alan J

    David S. Johnson, Alan J. Demers, Jeffrey D. Ullman, Michael R. Garey, et al. Worst-case per- formance bounds for simple one-dimensional packing algorithms. SIAM Journal on Computing, 3(4):299–325, 1974. doi:10.1137/0203025

    doi:10.1137/0203025 1974

  19. [19]

    Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

    Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, et al. Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World. arXiv, October 2024. doi:10.48550/arXiv.2410.16713

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.16713 2024

  20. [20]

    Efficient Memory Man- agement for Large Language Model Serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al. Efficient Memory Man- agement for Large Language Model Serving with PagedAttention. arXiv, September 2023. doi:10.48550/arXiv.2309.06180

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.06180 2023

  21. [21]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, et al. Reward- Bench: Evaluating Reward Models for Language Modeling. arXiv, March 2024. doi:10.48550/arXiv.2403.13787

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.13787 2024

  22. [22]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, et al. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv, June 2024. doi:10.48550/arXiv.2406.11939

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.11939 2024

  24. [24]

    Torchtitan: One-stop pytorch native solution for production ready LLM pretraining

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, et al. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Con- ference on Learning Representations . 2025. URL https://openreview.net/forum?id= SFN6Wm7YBI

    2025

  25. [25]

    rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

    Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, et al. rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset. arXiv, May 2025. doi:10.48550/arXiv.2505.21297. 14 Hermes 4 Technical Report

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.21297 2025

  26. [26]

    General-Reasoner: Advancing LLM Reasoning Across All Domains

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, et al. General-Reasoner: Advancing LLM Reasoning Across All Domains. arXiv, May 2025. doi:10.48550/arXiv.2505.14652

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.14652 2025

  27. [27]

    Atropos: An async first environ- ment rollout controller

    Dakota Mahan, Roger Jin, Teknium, Shannon Sands, et al. Atropos: An async first environ- ment rollout controller. https://www.github.com/NousResearch/Atropos, April 2025. Version 0.1

    2025

  28. [28]

    Generative Reward Models

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, et al. Generative Reward Models. arXiv, October 2024. doi:10.48550/arXiv.2410.12832

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.12832 2024

  29. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https: //arxiv.org/abs/1809.02789

    Pith/arXiv arXiv 2018

  30. [30]

    Jiang, Andy Lo, et al

    Mistral, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, et al. Magistral. arXiv, June 2025. doi:10.48550/arXiv.2506.10910

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.10910 2025

  31. [31]

    AgentIn- struct: Toward Generative Teaching with Agentic Flows

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, et al. AgentIn- struct: Toward Generative Teaching with Agentic Flows. arXiv, July 2024. doi:10.48550/arXiv.2407.03502

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.03502 2024

  32. [32]

    pydantic-cli: Turn pydantic-defined data models into cli tools, 2025

    mpkocher. pydantic-cli: Turn pydantic-defined data models into cli tools, 2025. URL https: //github.com/mpkocher/pydantic-cli. MIT License

    2025

  33. [33]

    Measuring Thinking Efficiency in Reasoning Models: The Missing Benchmark

    Tim Nous. Measuring Thinking Efficiency in Reasoning Models: The Missing Benchmark. NOUS RESEARCH , August 2025. URL https://nousresearch.com/ measuring-thinking-efficiency-in-reasoning-models-the-missing-benchmark

    2025

  34. [34]

    NVIDIA Nemotron Nano 2: An Ac- curate and Efficient Hybrid Mamba-Transformer Reasoning Model

    Nvidia, :, Aarti Basant, Abhijit Khairnar, et al. NVIDIA Nemotron Nano 2: An Ac- curate and Efficient Hybrid Mamba-Transformer Reasoning Model. arXiv, August 2025. doi:10.48550/arXiv.2508.14444

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.14444 2025

  35. [35]

    URL https://openai.com/index/learning-to-reason-with-llms/

    OpenAI. URL https://openai.com/index/learning-to-reason-with-llms/

  36. [36]

    OpenAI Chat Completions API Reference

    OpenAI. OpenAI Chat Completions API Reference . OpenAI, 2025. URL https://platform. openai.com/docs/api-reference/chat. Online; accessed on 14 August 2025

    2025

  37. [37]

    Samuel J. Paech. Eq-bench: An emotional intelligence benchmark for large language models, 2023

    2023

  38. [38]

    Samuel J. Paech. Eq-bench 3: Emotional intelligence benchmark. https://github.com/EQ- bench/eqbench3, 2025

    2025

  39. [39]

    Eq-bench creative writing benchmark v3

    Samuel J Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative- writing-bench, 2025

    2025

  40. [40]

    S.J. Paech. Longform creative writing benchmark, 2025. URL https://github.com/ EQ-bench/longform-writing-bench . GitHub repository

    2025

  41. [41]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. arXiv, April 2024. doi:10.48550/arXiv.2404.13076

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.13076 2024

  42. [42]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Loubna Ben Allal, Anton Lozhkov, et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv, June 2024. doi:10.48550/arXiv.2406.17557

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.17557 2024

  43. [43]

    Generalizing verifiable instruction following, 2025

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, et al. Generalizing verifiable instruction following, 2025. URL https://arxiv.org/abs/2507.02833

    Pith/arXiv arXiv 2025

  44. [44]

    Gpqa: A graduate-level google-proof q&a benchmark, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, et al. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  45. [45]

    Musr: Testing the limits of chain-of-thought with multistep soft reasoning, 2024

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, et al. Musr: Testing the limits of chain-of-thought with multistep soft reasoning, 2024. URL https://arxiv.org/abs/2310. 16049

    2024

  46. [46]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https://arxiv.org/abs/2206.04615

    Pith/arXiv arXiv 2023

  47. [47]

    Internbootcamp, 4 2025

    Internbootcamp Team. Internbootcamp, 4 2025

    2025

  48. [48]

    Kimi k2: Open agentic intelligence,

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, et al. Kimi k2: Open agentic intelligence,

  49. [49]

    The llama 3 herd of models, 2024

    Llama Team. The llama 3 herd of models, 2024. URL https://ai.meta.com/research/ publications/the-llama-3-herd-of-models

    2024

  50. [50]

    Deephermes 3 preview, 2025

    Ryan Teknium, Roger Jin, Chen Guang, Jai Suphavadeeprasit, et al. Deephermes 3 preview, 2025

    2025

  51. [51]

    Hermes 3 technical report, 2024

    Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report, 2024. URL https://arxiv.org/abs/2408.11857

    Pith/arXiv arXiv 2024

  52. [52]

    Kung, and David Cox

    Philippe Tillet, H.T. Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN Interna- tional Workshop on Machine Learning and Programming Languages (MAPL ’19) . ACM, 2019. doi:10.1145/3315508.3329973

    arXiv 2019

  53. [53]

    Various techniques used in connection with random digits

    John von Neumann. Various techniques used in connection with random digits. In A. H. Taub, editor, Collected Works of John von Neumann, V olume 5: Design of Computers, Theory of Automata and Numerical Analysis, pages 768–770. Pergamon Press, Oxford, England, 1961. Reprinted from Journal of Research of the National Bureau of Standards, Applied Mathematics ...

    1961

  54. [54]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv.org/ abs/2406.01574

    Pith/arXiv arXiv 2024

  55. [55]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv, December 2024. doi:10.48550/arXiv.2412.13663

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.13663 2024

  56. [56]

    Measuring short-form factuality in large language models, 2024

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, et al. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368

    Pith/arXiv arXiv 2024

  57. [57]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al. Qwen3 Technical Report. arXiv, May 2025. doi:10.48550/arXiv.2505.09388

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  58. [58]

    FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv, January 2025. doi:10.48550/arXiv.2501.01005

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.01005 2025

  59. [59]

    Yoo, Morris A

    Andy B. Yoo, Morris A. Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing, volume 2862 of Lecture Notes in Computer Science , pages 44–60. Springer, Berlin, Heidelberg, 2003. ISBN 978-3-540- 39727-4. doi:10.1007/10968987_3

    doi:10.1007/10968987_3 2003

  60. [60]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. arXiv, March 2022. doi:10.48550/arXiv.2203.14465

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.14465 2022

  61. [61]

    SGLang: Ef- ficient Execution of Structured Language Model Programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, et al. SGLang: Ef- ficient Execution of Structured Language Model Programs. arXiv, December 2023. doi:10.48550/arXiv.2312.07104

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.07104 2023

  62. [63]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, et al. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    Pith/arXiv arXiv 2023

  63. [64]

    How to Synthesize Text Data without Model Collapse? arXiv, December 2024

    Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, et al. How to Synthesize Text Data without Model Collapse? arXiv, December 2024. doi:10.48550/arXiv.2412.14689. 16 Hermes 4 Technical Report A PersonaHub Prompt User Prompt Please synthesize a detailed and creative coding task based on the following persona and specific parameters. The task should be sol...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.14689 2024

  64. [66]

    **Task Type Focus **: Create a task very loosely inspired by but not necessarily similar to: a text-based adventure game engine

  65. [67]

    **Target Difficulty/Scope **: The task should be: emphasizing UI/UX implementation and front-end best practices

  66. [68]

    **Key Technical Challenge(s) to Incorporate **: The task should touch upon: ensuring WCAG accessibility standards are met in UI components., and applying common design patterns (e.g., Singleton, Factory, Observer, Strategy) appropriately., and identifying and resolving performance bottlenecks using (hypothetical) profiling tools

  67. [69]

    **Project Nature & Starting Point **: The task involves ’add feature to existing class module’. Your generated task description MUST include a relevant snippet of hypothetical existing code (e.g., 15-50 lines in a common language like Python, JavaScript, Java, C#, or TypeScript), briefly describe what this code does, and outline the conceptual existing sy...

  68. [70]

    **Tone of voice **: You decide between formal, casual, angry

  69. [71]

    Alternatively, . . . Alternatively,

    **Typos**: Please include realistic typos and talk as if you’re texting a secretary - really short, curt, rude is ok. Skipping irrelevant words in sentence is ok. Persona Information: A professional in the poultry industry, specifically focused on managing and controlling outbreaks of exotic Newcastle disease in poultry farms. This person is likely to hav...

    work page 2024

  70. [72]

    Start with an ominous tone

    work page

  71. [73]

    Describe the fries as if they are an ancient artifact or a forbidden object

    work page

  72. [74]

    Use vivid and grotesque imagery to transform the cooking process into something eldritch

    work page

  73. [75]

    Incorporate elements of madness or forbidden rituals 19 Hermes 4 Technical Report

    work page

  74. [76]

    Beneath the pallid glow of kitchen lights, Where shadows writhe like nameless things, A blade descends with fell delight-- The steel descends on golden rings

    End with a twist that connects to cosmic horror Let’s outline: - Setting: A kitchen at night? Or perhaps in some temple-like structure? - The fry cutter becomes a dark instrument? - Cooking becomes an incantation? - The final product (the fries) as something that reveals terrifying truths? Draft: I. Beneath the pallid glow of kitchen lights, Where shadows...

    work page

  75. [77]

    Ominous setting (pallid glow) and eerie movements (shadows writhe)

    work page

  76. [78]

    deep as time

    Transformed cooking process: cutting becomes an act with dark delight; frying is in oils "deep as time"

    work page

  77. [79]

    fingers from a buried crypt

    The fries themselves become "fingers from a buried crypt" -> eldritch imagery

    work page

  78. [80]

    julienne cut

    Eating them causes cosmic revelation -> classic Lovecraftian trope: forbidden knowledge driving one mad Adjustments: - Made sure to include "julienne cut" by describing them as slender stalks and using words like "slender cuts" Final Answer Herein lies your eldritch ode--beware its greasy revelations: **JULIENNE EIDOLON ** Beneath dead stars where sanity ...

    work page 1937

  79. [81]

    Indigenous genocide/forced assimilation (boarding schools)

    work page

  80. [82]

    Slavery’s lasting legacy

    work page

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.