Pith. sign in

REVIEW 4 major objections 6 minor 34 references

When two task specs conflict, language models systematically prefer formal structure over plain language and examples.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 09:06 UTC pith:UW2XZ2I7

load-bearing objection Clean measurement framework with real design care; the Formal>NL>Ex ordering is useful but still partly confounded with recoverability until single-spec baselines exist. the 4 major comments →

arxiv 2607.28384 v1 pith:UW2XZ2I7 submitted 2026-07-30 cs.AI

When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences

Tairan Wang , Liang Zhou , Zikang Zhan , Pingchuan Yan This is my paper
classification cs.AI
keywords large language modelsspecification conflictrepresentation preferencesymmetry-based evaluationexecutable mappingsconflict resolutionformal vs natural languageinput-output examples
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language models often face several instructions at once that do not agree. This paper gives a controlled way to measure which form of instruction the model actually follows when those instructions force incompatible answers. It turns each conflict into two executable mappings that disagree on a test query, then counterbalances order and which form carries which mapping so the model’s choice can be attributed cleanly. On a 550-instance math benchmark spanning eleven function families, models do not choose at random: formal and naturalized-formal specs beat pure natural language, which in turn beats input–output examples. The same method carries over to Boolean algebra, code generation, and clinical rules, where preferences shift with domain—showing that conflict resolution is structured specification selection, not noise.

Core claim

Under symmetry-counterbalanced conflicts between competing executable mappings, LLMs exhibit systematic representation preferences rather than random resolution. On the mathematical benchmark the aggregate ordering is Formal ≈ Naturalized Formal > Pure Natural Language > Input–Output Examples, with formal and naturalized-formal forms selected over pure natural language in roughly 87% of attributable responses and over examples in roughly 93%. Example influence rises with model capability and depends on function family. The same attributable measurement extends to Boolean algebra, code generation, and clinical rule conflicts, where domain-specific factors can override or reshape the math orde

What carries the argument

A symmetry-based preference-attribution framework: two conflicting executable mappings f1 and f2 that disagree on a discriminative query, expressed in competing representation types, with presentation order and type-to-mapping assignment fully counterbalanced so the model’s answer can be scored as selecting f1, f2, or neither, then aggregated as P(t) over the symmetry-complete set.

Load-bearing premise

That matching the model’s final answer to one mapping’s stored output means the model selected that specification form, rather than computing something else or reacting to confounds like length and detail that cannot be fully separated from form.

What would settle it

Rerun the same 550 math conflicts with length-, density-, and detail-matched specs across forms (or with forced equal information content) and check whether the Formal ≈ NatF > NL > Examples ordering collapses or reverses in attributable selection rates.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Conflict handling in LLM systems can be measured as attributable specification selection, not only as accuracy or contradiction detection.
  • Preserving explicit structure in specs (formal or naturalized-formal) is a practical lever when multiple instructions may disagree.
  • Example-based specs should not be assumed weak or strong a priori: their weight tracks model capability and task recoverability.
  • In code settings, higher-capability models’ tilt toward tests over documentation is a measurable policy, not an automatic virtue.
  • In clinical-style rules, restrictiveness bias can dominate representation form, so domain checks must accompany form checks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Prompt and tool design that must merge docs, tests, and formal constraints could treat structure-preserving formats as the default “tie-breaker” layer unless domain evidence says otherwise.
  • Abstention rates under an explicit “uncertain” option (higher for some models when both sides are textual/formal) suggest a second axis of conflict behavior worth reporting beside preference rankings.
  • If length and information density are the real drivers, future work could orthogonalize those factors and re-estimate P(t) to isolate pure form effects.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a symmetry-based experimental framework for measuring how LLMs resolve conflicts between competing task specifications. Conflicts are formalized as two executable mappings f1, f2 that disagree on a discriminative query q*; model answers are attributed by exact match to f1(q*) or f2(q*). Presentation order and representation-to-mapping assignment are 2×2 counterbalanced, and preference P(t) is estimated by aggregation over the symmetry-complete set U. On a 550-instance mathematical benchmark (11 function families), four representation types are compared pairwise, yielding the aggregate ordering Formal ≈ Naturalized Formal > Pure Natural Language > Input–Output Examples. The framework is then applied to Boolean algebra (formula vs truth table / gate netlist), code generation (documentation vs tests on MBPP-derived conflicts), and clinical rule conflicts (openFDA-derived thresholds), showing attributable but context-dependent selection patterns, including a clinical restrictiveness bias.

Significance. If the measurement is valid, the work supplies a controllable, attributable alternative to existing conflict benchmarks that mainly study source credibility or knowledge contradictions rather than specification form. Strengths include executable ground truth, systematic order/assignment counterbalancing, clustered-bootstrap CIs, low non-attributable rates outside NL-vs-Ex, released construction detail for three heterogeneous domains, and diagnostic extensions (abstention, cross-authority clinical conflicts). The code-generation capability split and clinical restrictiveness finding are independently interesting for system design. The contribution is primarily methodological and empirical rather than theoretical; its lasting value depends on whether attributed matches truly isolate representation preference rather than differential recoverability of the underlying mapping.

major comments (4)
  1. [Preference Attribution; Experimental Protocol; Results (Pairwise Representation Preferences)] Preference Attribution / Experimental Protocol (and Fig. 2): the central inference treats exact match of the final answer to fs(q*) under a two-spec prompt as selection of representation type s. That reading requires that both specs are, in isolation, sufficiently recoverable for each model. The manuscript never reports single-specification competence baselines (one spec + q*, no conflict) stratified by representation type, model, and function family. Pure-NL bans notation; examples require induction and exclude q*; Form/NatF expose structure. Without these baselines, the headline ordering (e.g., Form>NL 87.45%, Form>Ex 93.04% of attributable responses in Pairwise Representation Preferences) is compatible with differential mapping recoverability rather than preferential selection under dual comprehension. This is load-bearing for RQ1 and should be added, or the claim should be narrowed t
  2. [Mathematical Benchmark Construction; Specification Generation; Discussion] Specification Generation (Mathematical Benchmark Construction): surface forms for NL, Form, and NatF are produced by DeepSeek-V4-Pro with temperature 1 and only manual spot-checks for faithfulness—no quantified semantic-faithfulness or single-spec accuracy audit, and no control that NL/Form/NatF are matched on length, information density, or detail. The Discussion already states these factors ‘cannot be fully separated’ from representation type. For a paper whose claim is about representation preference, either (i) report length/density statistics and single-spec accuracy by form, or (ii) add a matched-length / human-written control subset. Otherwise the Form≈NatF≫NL pattern may partly reflect generator style and density rather than form per se.
  3. [When Do Examples Become Competitive?; Table 1; Additional Results (Fig. 10, Fig. 12)] When Do Examples Become Competitive? / Table 1: higher-capability variants select example-supported mappings more often, which is presented as evidence that example influence depends on capability. The same pattern is exactly what incomplete example-based rule induction predicts (cf. elevated ‘other’ rates in NL vs Ex, Fig. 12, especially Llama and GPT-4.1-nano). Without single-spec example-only accuracy by model and family (Fig. 10 already shows large family effects), the capability interaction cannot distinguish ‘prefers examples more’ from ‘can induce from examples more.’ Please report example-only competence and condition conflict selection on successful single-spec induction where possible.
  4. [Heterogeneous Specification Conflicts; Abstract; Conclusion] Heterogeneous extensions are valuable but unevenly support the unified ‘representation preference’ narrative. Clinical Rule Conflicts (Fig. 7): textual forms show no stable ordering and all models prefer the more restrictive rule (52–87%), with genuine-value selection tracking restrictiveness under bidirectional perturbations—not representation. Code generation (Fig. 6): test vs documentation preference swings from ~16% to ~97% across models and is only loosely tied to an external capability index. These results show the framework measures attributable selection, but they also show that the math-benchmark ordering is not a general hierarchy of specification forms. The abstract and conclusion should state more sharply that the Form≈NatF>NL>Ex ordering is benchmark-specific, and that domain factors can dominate representation.
minor comments (6)
  1. [References] Several in-text citations and model cards are dated 2026 (e.g., DeepSeek-V4, GPT-5.4-nano, openFDA access dates). Ensure bibliography consistency and that all cited artifacts are citable at publication time.
  2. [Figure 3] Figure 3 panels for individual models are hard to read in the manuscript text layout; ensure axis labels and the pooled panel remain legible in the camera-ready figures.
  3. [Preference Attribution] Notation: P(t)=E_u P(select t | t, u) is clear, but ‘select t’ vs selecting mapping fi should be distinguished more explicitly when representation type and mapping identity are crossed.
  4. [Abstract; Introduction] Typographical issues: missing spaces in several run-on phrases in the abstract/introduction (e.g., ‘Furthermore,weextendtheframework…’) and inconsistent capitalization of Naturalized Formal / NatF.
  5. [Code Generation; Appendix] Code-generation construction is thorough in the appendix; a short main-text statement of the 67/200 retention rate and the single-slot semantic contract would help readers assess selection bias without reading the full appendix.
  6. [Robustness Checks; Figure 11] Robustness Checks: second-position rate of 67.86% for Form vs NatF is important; consider promoting a brief main-text caveat that order dominates when representation strengths are similar.

Circularity Check

0 steps flagged

No significant circularity: empirical preference measurement over executable conflicts, not a derivation that redefines its target.

full rationale

The paper’s central object is an operational measurement, not a first-principles derivation. Conflicting specifications are realized as statically defined executable mappings f1, f2 with a discriminative query q* such that f1(q*) ≠ f2(q*); model answers are attributed by exact match to those stored outputs; and preference is the aggregate P(t) = E_u P(select t | t, u) over symmetry-complete counterbalanced configurations. That construction makes selection observable; it does not force which representation wins. The reported ordering (Formal ≈ NatF > NL > Ex) and the heterogeneous extensions are empirical aggregates over trials, not fitted parameters relabeled as predictions, uniqueness theorems imported from the authors, or renamings of a known law. Specification surfaces are generated from fixed mappings and spot-checked for faithfulness; mappings, queries, and labels remain independent of the preference claim. Minor operational choices (attribution categories, one-slot code conflicts, restrictiveness splits) define the measurement protocol without making the observed selection rates true by construction. No load-bearing self-citation chain or self-definitional loop is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claims rest on operational measurement assumptions and construction choices, not on new physical entities. Load-bearing premises are that exact output match attributes specification selection, that schema-preserving perturbations create fair conflicts, and that four surface forms are comparable after symmetry aggregation despite uncontrolled length/density.

free parameters (4)
  • examples_per_mapping = 10 (4 for graph degree)
    Most math instances use ten I/O examples (four for graph degree); this hand-chosen budget affects example competitiveness.
  • specification_generator_sampling = temperature=1, thinking disabled
    DeepSeek-V4-Pro realizes NL/Form/NatF specs at temperature 1 with thinking disabled; surface wording is stochastic and generator-dependent.
  • code_conflict_semantic_accept_thresholds = total≥22; single-slot=5; others≥4
    MBPP pair acceptance requires evaluator total ≥22, single-slot score 5, other dimensions ≥4—hand-set gates that determine the 67-pair set.
  • clinical_subsample_sizes = e.g., 1500 / 700 / full 7968
    Uneven trial counts per model (e.g., 1500-trial subsample, 700-trial nested subset) are cost-driven analysis choices.
axioms (5)
  • ad hoc to paper If the model’s normalized final answer equals fs(q*), the trial is attributed as selecting specification s (else other).
    Core attribution rule in Preference Attribution and Experimental Protocol; equates behavioral match with specification preference.
  • domain assumption Schema-preserving, family-specific perturbations yield minimally different but meaningfully distinguishable executable mappings.
    Mathematical Benchmark Construction and Table 2; fairness of conflicts depends on this design choice.
  • domain assumption Symmetry-complete counterbalancing of order and representation-to-mapping assignment cancels non-target confounds in the aggregate preference P(t).
    Stated in Method; standard experimental-design assumption, only partially true if form covaries with length/density.
  • domain assumption LLM API outputs under stated decoding settings are stable enough for bootstrap inference over conflict instances.
    Implicit throughout Results; endpoint defaults and reasoning-effort settings vary by model (Table 5).
  • domain assumption Exact match / hidden-test execution is an adequate oracle for code and clinical yes-no attribution after parsing/adjudication.
    Code-generation and clinical appendices; residual parser/adjudication layer for clinical free-text.
invented entities (2)
  • Symmetry-based conflicting-specification preference framework (P(t) over symmetry-complete U) independent evidence
    purpose: Turn implicit conflict resolution into attributable selection among executable mappings while marginalizing order and assignment.
    Central methodological object defined in Method; not a physical entity, but a new measurement construct.
  • Naturalized formal (NatF) representation type independent evidence
    purpose: Intermediate surface form retaining math notation inside natural language for comparison to pure NL and pure formal.
    Operational category used throughout benchmark; empirically behaves like Formal rather than a midpoint.

pith-pipeline@v1.2.0-daily-grok45 · 27417 in / 3474 out tokens · 68587 ms · 2026-07-31T09:06:13.158220+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly required to integrate multiple sources of information that may be inconsistent or conflicting. However, there is still a lack of controllable and attributable methods for analyzing how models resolve conflicts between competing specifications. We propose a controlled experimental framework for studying model preferences under conflicting specifications. By constructing specifications with explicit conflicts, the framework enables model choices between competing specifications to be directly observed and analyzed. A symmetry-based design further reduces confounding factors, allowing preferences across representation types to be compared systematically. We evaluate the framework on an executable mathematical benchmark with 550 conflict instances spanning 11 function families, comparing four representation types: pure natural language, formal language, naturalized formal language, and input--output examples. Results show systematic preference patterns rather than random behavior, with a consistent ordering: $ \text{Formal} \approx \text{Naturalized Formal} > \text{Pure Natural Language} > \text{Input--Output Examples} $. Example effects further depend on model capability and function family. We extend the framework to heterogeneous specification conflicts in Boolean algebra, code generation, and the clinical domain, demonstrating its applicability across diverse tasks and specification forms. The framework provides a unified approach for measuring how LLMs resolve conflicts between competing sources of information.

Figures

Figures reproduced from arXiv: 2607.28384 by Liang Zhou, Pingchuan Yan, Tairan Wang, Zikang Zhan.

Figure 1
Figure 1. Figure 1: Example conflict instance represented in four spec [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Models are instructed to provide a clearly identifi [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pairwise representation preferences across evaluated models. Each point shows the proportion of attributable responses [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Selection of example-supported mappings across models. Each point shows the proportion of attributable responses [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results on the Boolean Algebra dataset, showing [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Clinical rule conflicts. (a) Example-selection rates against each textual form; all clustered-bootstrap intervals lie below 50%. (b) Genuine-value selection conditioned on whether the genuine rule is stricter or more permissive. from five families. Appendix details the construction, cover￾age, and attribution procedures. These rules are transcribed solely to measure model selection behavior and do not con￾… view at source ↗
Figure 6
Figure 6. Figure 6: “No explicit effort” means that the request omitted [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template for code-generation experiments. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Cross-authority conflicts. Probability of choosing [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example selection by function family across all three example-based comparisons. Light points show model-specific [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Presentation-order effects across representation [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Rates of responses assigned to other across models and representation comparisons. Each cell reports the percentage of all trials whose normalized final answer matches neither executable candidate output. Non-attributable responses are generally rare and are concentrated in the pure-natural-language versus examples condition. 0% 20% 40% 60% 80% Formula selected (%) DS-Flash DS-Pro GPT-4.1 GPT-5 GPT-5.4 Ll… view at source ↗
Figure 13
Figure 13. Figure 13: Results on the Boolean Algebra dataset, show [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Results on the Boolean Algebra dataset, show [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in neural information processing systems , volume=

  2. [2]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  3. [3]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    Multi-hop paragraph retrieval for open-domain question answering , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Wikicontradict: A benchmark for evaluating llms on real-world knowledge conflicts from wikipedia , author=. Advances in Neural Information Processing Systems , volume=

  5. [5]

    npj Digital Medicine , volume=

    Leveraging long context in retrieval augmented language models for medical question answering , author=. npj Digital Medicine , volume=. 2025 , publisher=

    2025

  6. [6]

    Findings of the Association for Computational Linguistics: EMNLP , volume=

    Conflicts in texts: Data, implications and challenges , author=. Findings of the Association for Computational Linguistics: EMNLP , volume=

  7. [7]

    arXiv preprint arXiv:2603.04464 , year=

    Understanding the Dynamics of Demonstration Conflict in In-Context Learning , author=. arXiv preprint arXiv:2603.04464 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2511.06668 , year=

    When Evidence Contradicts: Toward Safer Retrieval-Augmented Generation in Healthcare , author=. arXiv preprint arXiv:2511.06668 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2505.17762 , year=

    Resolving conflicting evidence in automated fact-checking: A study on retrieval-augmented llms , author=. arXiv preprint arXiv:2505.17762 , year=

    Pith/arXiv arXiv

  10. [10]

    Proceedings of the 34th ACM International Conference on Information and Knowledge Management , pages=

    BordaRAG: Resolving Knowledge Conflict in Retrieval-Augmented Generation via Borda Voting Process , author=. Proceedings of the 34th ACM International Conference on Information and Knowledge Management , pages=

  11. [11]

    Towards knowledge checking in retrieval-augmented generation: A representation perspective , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  12. [12]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Conflicting needles in a haystack: How LLMs behave when faced with contradictory information , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  13. [13]

    2025 , publisher =

    Lee, Jungyeon and Kangmin, Lee and Kim, Taeuk , booktitle =. 2025 , publisher =

    2025

  14. [14]

    Whose Facts Win? LLM Source Preferences under Knowledge Conflicts

    Schuster, Jakob and Gautam, Vagrant and Markert, Katja. Whose Facts Win? LLM Source Preferences under Knowledge Conflicts. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026. doi:10.18653/v1/2026.acl-long.1357

    work page doi:10.18653/v1/2026.acl-long.1357 2026

  15. [15]

    Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025) , pages=

    Consensus or conflict? fine-grained evaluation of conflicting answers in question-answering , author=. Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025) , pages=

    2025

  16. [16]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Knowledge conflicts for llms: A survey , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    What evidence do language models find convincing? , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  19. [19]

    arXiv preprint arXiv:2407.13048 , year=

    Establishing Knowledge Preference in Language Models , author=. arXiv preprint arXiv:2407.13048 , year=

    Pith/arXiv arXiv

  20. [20]

    The Twelfth International Conference on Learning Representations , year=

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author=. The Twelfth International Conference on Learning Representations , year=

  21. [21]

    2023 , eprint=

    Larger language models do in-context learning differently , author=. 2023 , eprint=

    2023

  22. [22]

    2021 , eprint=

    Program Synthesis with Large Language Models , author=. 2021 , eprint=

    2021

  23. [23]

    Artificial Analysis Intelligence Benchmarking Methodology , year =

  24. [24]

    2026 , note =

    Comparison of. 2026 , note =

    2026

  25. [25]

    arXiv preprint arXiv:2307.03172 , year=

    Lost in the middle: How language models use long contexts , author=. arXiv preprint arXiv:2307.03172 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2404.05520 , year=

    The fact selection problem in llm-based program repair , author=. arXiv preprint arXiv:2404.05520 , year=

    Pith/arXiv arXiv

  27. [27]

    2026 , eprint=

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. 2026 , eprint=

    2026

  28. [28]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  29. [29]

    2026 , eprint=

    The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes , author=. 2026 , eprint=

    2026

  30. [30]

    Introducing GPT-4.1 in the API , year =

  31. [31]

    2025 , howpublished =

    2025

  32. [32]

    Introducing GPT-5.4 Mini and Nano , year =

  33. [33]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 Technical Report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  34. [34]

    2026 , eprint=

    Kimi K3: Open Frontier Intelligence , author=. 2026 , eprint=

    2026