Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Routing among language models is meaningful only when the society is behaviourally diverse and the router is stable under surface-form query variants.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 04:45 UTC pith:WGQT2BZK

load-bearing objection Solid empirical framing of when multi-LLM routing is meaningful; the accuracy–robustness divergence is real but partly confounded by the untested optimal-actor assumption. the 3 major comments →

arxiv 2607.09197 v1 pith:WGQT2BZK submitted 2026-07-10 cs.MA

When is Routing Meaningful? Diversity and Robustness in Language Model Societies

Fantine Huot , Michael Kaisers , Mirella Lapata This is my paper
classification cs.MA
keywords LLM routingHierarchic Social Entropybehavioural diversityrouting robustnessmulti-agent societiesspecialisationperturbation stabilitycoreset selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most evaluations of multi-model routers care only about task accuracy and cost. This paper argues that two structural properties, independent of raw performance, decide whether routing is actually meaningful. The society of models must be behaviourally differentiated; if every actor answers the same way, the router has nothing real to choose. The routing policy must also be stable: paraphrases, typos, and other surface variants of a query should still go to the same actor, otherwise specialisation cannot emerge. The authors adapt Hierarchic Social Entropy from robotics to measure behavioural diversity from model outputs, and introduce a five-level perturbation robustness score for routing decisions. On EmbedLLM and RouterBench they show that diversity saturates quickly—a curated subset of fewer than ten models recovers most of the diversity in a large pool—and that KNN routers can look accurate on specialist societies while collapsing under perturbation, whereas prompted routing stays stable. Accuracy and meaningfulness can therefore diverge sharply.

Core claim

High task accuracy is compatible with vacuous routing: a router can score well while operating over a redundant society of near-identical models or while assigning surface-form variants of the same query to different actors. Behavioural diversity measured by Hierarchic Social Entropy and assignment stability under meaning-preserving perturbations are therefore necessary diagnostics that accuracy alone cannot supply.

What carries the argument

Hierarchic Social Entropy (HSE) adapted to language-model societies: behavioural vectors of scores on an evaluation set are compared by cosine distance, hierarchical single-linkage clustering is performed at every distance threshold, and simple social entropy is integrated across all taxonomic levels to yield a single diversity score. Paired with a per-query robustness metric that counts how often five classes of surface perturbation are routed to the same actor as the original.

Load-bearing premise

The paper assumes that a meaning-preserving surface change never legitimately changes which model is best for a query, so any reassignment under perturbation is pure router instability rather than a real change in suitability.

What would settle it

Construct a set of paraphrases or syntactic variants that systematically change which model is most accurate (for example by shifting domain difficulty or required expertise) and check whether the robustness metric still labels those reassignments as failures; if it does, the central diagnostic collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that multi-model routing is meaningful only when two properties orthogonal to accuracy hold: (i) the actor society is behaviourally differentiated, and (ii) the routing policy is stable under surface-form query variants. It adapts Hierarchic Social Entropy (HSE) to LM societies via cosine distance on behavioural score vectors and hierarchical clustering, and defines a perturbation-based robustness metric ρ over a five-level taxonomy. On EmbedLLM and RouterBench, HSE shows strong diminishing returns (a curated subset of fewer than ten agents recovers most diversity), specialist synthetic societies are far more diverse than real model pools, and KNN routers gain clean accuracy on high-HSE specialist societies while collapsing under perturbation, whereas prompted routing remains comparatively stable—illustrating that accuracy and meaningfulness can diverge.

Significance. If the claims hold, the work supplies two practical, independently measurable diagnostics for multi-LLM and multi-agent system design that current accuracy/cost-only evaluations miss. The coreset-style finding that most behavioural diversity is captured by a small curated subset is immediately actionable for society design. The HSE adaptation is carefully validated with sanity checks (identical vs orthogonal societies; redundancy does not inflate HSE) and is shown to be qualitatively stable across distance/linkage choices (Appendix E). The explicit separation of diversity (society-only) from robustness (policy-dependent) is a clean conceptual contribution to the multi-agent and routing literature.

major comments (3)
  1. [Section 5.2, Figure 4] Section 5.2 states the load-bearing assumption that meaning-preserving perturbations do not change which actor is most accurate: any change in π is treated as pure policy instability. This is never validated. On the synthetic SA/RD societies (binary domain indicators, oracle accuracy 1.0 by construction), a paraphrase or rambling rewrite that crosses a subject or reasoning-depth boundary legitimately changes the optimal actor; ρ then scores correct reassignment as brittleness. Even on real models, surface rewrites can shift difficulty. Without ground-truth optimal-actor labels (or at least accuracy of the original assignee) on the perturbed set, the headline accuracy–robustness divergence in Figure 4 is partly confounded. Please either (a) report how often the originally optimal actor remains optimal under each perturbation class, or (b) qualify the KNN “collapse” claims, especially on s
  2. [Section 7, Figure 4; Limitations] The strongest empirical claim—that KNN gains accuracy from specialist societies but collapses in robustness—rests primarily on the synthetic SA/RD/RD+SA societies with maximally sharp behavioural boundaries (Section 6, Figure 4). The Limitations section notes that real specialists will have softer profiles and that brittleness may be attenuated, but the main text and abstract present the divergence as a general illustration. Please either add a parallel analysis restricted to real-model max-HSE subsets (where Figure 5 already shows a more mixed KNN picture on EmbedLLM) as the primary evidence, or clearly mark the synthetic results as an upper-bound stress test rather than the central demonstration.
  3. [Section 4, Figure 3; Limitations] HSE is defined over a fixed evaluation set E (Section 3–4). When E is not representative of the deployment query distribution, behavioural vectors and therefore diversity rankings can be misleading (Limitations). The paper does not report sensitivity of the max-HSE subset or the diminishing-returns plateau to the choice of E (e.g., train vs held-out, or domain-stratified subsets). A short sensitivity check would substantially strengthen the coreset heuristic claim that “fewer than ten agents recover most available diversity.”
minor comments (5)
  1. [Section 4.3–4.4] Normalisation of HSE is used throughout the figures but the exact normaliser (maximum achievable HSE for society size N) is only briefly stated; a short formula or reference to the integral bound would help readers reproduce the y-axes of Figures 3 and 7.
  2. [Figure 4] In Figure 4, solid vs dashed lines for original vs perturbed accuracy are easy to miss in greyscale; consider markers or a clearer legend.
  3. [Appendix B.1] Appendix B.1 uses Gemini 3.1 Flash-Lite to generate perturbations; a brief note on whether human spot-checks confirmed semantic preservation (especially for Level 5 rambling) would increase confidence in the taxonomy.
  4. [Section 7; Appendix F] Table 7 (Appendix F) is valuable for the joint HSE+accuracy coreset but is not referenced in the main-text discussion of society design; a one-sentence pointer in Section 7 would help.
  5. [Section 2–3] Typo/consistency: “Emay be set equal toQ” (Section 3) needs a space; “therouting decisionis” (Section 2) appears to have missing spaces in the source.

Circularity Check

0 steps flagged

No significant circularity: HSE and robustness are independently defined metrics measured on external benchmarks and synthetic validations, with no predictions reducing to fitted inputs or self-definitional claims.

full rationale

The paper introduces two structural metrics (adapted HSE via hierarchical clustering and Shannon entropy over behavioural vectors from cosine distances, and a simple empirical fraction ho of stable actor assignments under surface perturbations) and reports empirical measurements on EmbedLLM/RouterBench plus purpose-built synthetic societies. HSE is taken from Balch (2000) with an explicit adaptation (cosine + single linkage) justified by domain needs and ablated in Appendix E; sanity checks (Tables 1–2, Figure 3) confirm expected boundary behaviour by construction but are presented only as validation, not as derived predictions. Synthetic SA/RD societies have oracle accuracy 1.0 by design to isolate diversity effects, which is stated openly and used for controlled comparison rather than to force the main claims about diminishing returns or accuracy–robustness divergence. Those claims are observational (Figures 3–6, Table 7): measured HSE curves, accuracy, and ho values under different routers and societies. There are no fitted parameters later called predictions, no uniqueness theorems imported from overlapping authors, no ansatz smuggled via self-citation, and no renaming of known results as novel derivations. The load-bearing assumption about meaning-preserving perturbations (Section 5.2) is an untested modelling premise that may affect interpretation of ho, but it is not circularity: the metric is still an independent empirical quantity. The work is self-contained against the cited external benchmarks and does not reduce its central findings to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

This is an empirical methodology paper. Load-bearing content is the adaptation of HSE, the definition of ρ, and the assumption that surface-form variants share an optimal actor. Free parameters are design choices (k, p, linkage, greedy size) rather than fitted physical constants. Invented entities are the adapted metric and the routing-specific robustness measure; they have no independent evidence outside this evaluation framework.

free parameters (4)
  • KNN neighbourhood size k = 1, 3, 10
    Evaluated at k∈{1,3,10}; accuracy and robustness conclusions depend on this design choice, though patterns are reported across all three.
  • Number of perturbations per query p = 5
    One perturbation per level yields p=5; ρ averages over these variants and is sensitive to how many and which rewrites are kept after deduplication.
  • Max-HSE subset size (greedy) = 9 (EmbedLLM), 4 (RouterBench)
    Coreset claim uses greedily selected sizes (9 on EmbedLLM, 4 on RouterBench); not guaranteed globally optimal (Limitations).
  • Few-shot exemplar margin δ and count k = δ=0.4, k=4
    Prompted router samples exemplars with score margin δ=0.4 and k=4 per actor; affects prompted accuracy and robustness.
axioms (4)
  • domain assumption Behavioural diversity of heterogeneous LM actors is well captured by cosine distance over score vectors on a fixed evaluation set E, not by parameter-space distance.
    Section 3–4.2; justifies measuring diversity from B rather than weights and preferring cosine over Euclidean/Wasserstein.
  • domain assumption Meaning-preserving surface-form variants of a query share the same optimal actor assignment.
    Section 5.2; required for interpreting any routing change under perturbation as policy instability rather than legitimate reassignment.
  • standard math Hierarchic Social Entropy (integral of Shannon entropy over single-linkage partitions at all distance thresholds) is a valid scalar of multi-agent behavioural diversity.
    Adapted from Balch (2000); Shannon entropy and hierarchical clustering are standard; the LM-specific distance and linkage choices are paper-specific.
  • domain assumption Stable query assignment is a necessary condition for specialisation to emerge in LM societies.
    Sections 2 and 5; drawn from multi-robot specialisation literature (Labella et al., Nitschke et al.) and applied to routing.
invented entities (3)
  • HSE for language-model societies (cosine behavioural distance, single linkage) no independent evidence
    purpose: Scalar measure of behavioural differentiation of a multi-model society independent of any routing policy.
    Adaptation of Balch’s HSE; no external falsifiable prediction beyond the paper’s own benchmarks.
  • Routing robustness metric ρ (and per-actor ρ^(k)) no independent evidence
    purpose: Fraction of surface-form query variants assigned to the same actor as the original, as a necessary condition for specialisation.
    Defined in Section 5.2; measures assignment stability, not answer correctness.
  • Five-level perturbation taxonomy for routing decisions no independent evidence
    purpose: Fine-grained diagnosis of where a routing policy breaks (character, word, syntax, paraphrase, rambling).
    Table 3 and Appendix B; generation via prompted LM, not an independently validated linguistic resource.

pith-pipeline@v1.1.0-grok45 · 23637 in / 3407 out tokens · 57548 ms · 2026-07-13T04:45:36.424413+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of When is Routing Meaningful? Diversity and Robustness in Language Model Societies." pith.science (2026). https://pith.science/paper/WGQT2BZK

@misc{pith2026260709197,
  author       = {Pith},
  title        = {Pith review of: When is Routing Meaningful? Diversity and Robustness in Language Model Societies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGQT2BZK}},
  note         = {Machine review of arXiv:2607.09197}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Routing policies for multi-model systems are evaluated almost exclusively on task accuracy and inference cost. We argue that two properties, orthogonal to performance, determine whether routing is meaningful. First, the society of actors must be behaviourally differentiated: if all actors respond identically, routing is vacuous. Second, the routing policy must be stable: surface-form variants of a query should be assigned to the same actor. High task accuracy is compatible with violating both properties, since a router can operate over a redundant society or assign queries inconsistently, preventing specialisation regardless of performance. We adapt Hierarchic Social Entropy (HSE) to language-model societies and introduce a perturbation-based robustness metric to diagnose these failure modes. Applied to EmbedLLM and RouterBench, we find that HSE exhibits strong diminishing returns, suggesting that a curated subset of fewer than ten agents recovers most available diversity in a large pool -- a practical coreset heuristic for society design. We further find that KNN routers gain accuracy from specialist societies but collapse in robustness under perturbation, while prompted routing remains stable across all perturbation types -- illustrating that accuracy and meaningfulness can sharply diverge.

Figures

Figures reproduced from arXiv: 2607.09197 by Fantine Huot, Michael Kaisers, Mirella Lapata.

Figure 1
Figure 1. Figure 1: Simple social entropy computed on societies [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Normalised HSE as a function of society size [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task accuracy (a) and robustness ρ (b) across EmbedLLM societies ordered by HSE. Solid lines show original queries; dashed lines in (a) show perturbations. KNN routers gain accuracy on specialist societies but lose robustness; the prompted router maintains both. actor falls sharply after the first few. For the default society, the curve plateaus after approximately nine agents. Synthetic specialist societi… view at source ↗
Figure 5
Figure 5. Figure 5: Routing robustness ρ on the default society vs. the max-HSE subset for EmbedLLM (a) and Router￾Bench (b). Switching to the max-HSE subset improves the prompted router on both benchmarks, and substan￾tially improves all KNN routers on RouterBench. On EmbedLLM, KNN-1 robustness decreases. that make specialist societies easy to route correctly on clean queries make them equally easy to route in￾correctly when… view at source ↗
Figure 7
Figure 7. Figure 7: Normalised HSE as a function of society size across all combinations of distance metric (rows: cosine, [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 6 linked inside Pith

  1. [1]

    Autonomous robots , volume=

    Hierarchic social entropy: An information theoretic measure of robot group diversity , author=. Autonomous robots , volume=. 2000 , publisher=

    2000

  2. [2]

    Autonomous Agents and Multi-Agent Systems , volume=

    Quantifying the effects of environment and population diversity in multi-agent reinforcement learning , author=. Autonomous Agents and Multi-Agent Systems , volume=. 2022 , publisher=

    2022

  3. [3]

    arXiv preprint arXiv:1910.05512 , year=

    Influence-based multi-agent exploration , author=. arXiv preprint arXiv:1910.05512 , year=

    Pith/arXiv arXiv 1910

  4. [4]

    International Conference on Machine Learning , pages=

    Policy diagnosis via measuring role diversity in cooperative multi-agent rl , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    Towards unifying behavioral and response diversity for open-ended learning in zero-sum games , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    International conference on machine learning , pages=

    Trajectory diversity for zero-shot coordination , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  7. [7]

    , author=

    The value of life: biological diversity and human society. , author=

  8. [8]

    Journal of Machine Learning Research , volume=

    System neural diversity: Measuring behavioral heterogeneity in multi-agent learning , author=. Journal of Machine Learning Research , volume=

  9. [9]

    Journal of applied ecology , volume=

    Beyond species: functional diversity and the maintenance of ecological processes and services , author=. Journal of applied ecology , volume=. 2011 , publisher=

    2011

  10. [10]

    Biological Reviews , volume=

    A guide to phylogenetic metrics for conservation, community ecology and macroecology , author=. Biological Reviews , volume=. 2017 , publisher=

    2017

  11. [11]

    Nature Machine Intelligence , volume=

    Designing neural networks through neuroevolution , author=. Nature Machine Intelligence , volume=. 2019 , publisher=

    2019

  12. [12]

    ACM Transactions on Autonomous and Adaptive Systems (TAAS) , volume=

    Division of labor in a group of robots inspired by ants' foraging behavior , author=. ACM Transactions on Autonomous and Adaptive Systems (TAAS) , volume=. 2006 , publisher=

    2006

  13. [13]

    Intelligent complex adaptive systems , pages=

    Emergent specialization in biologically inspired collective behavior systems , author=. Intelligent complex adaptive systems , pages=. 2008 , publisher=

    2008

  14. [14]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  16. [16]

    and Zettlemoyer, Luke , booktitle=

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel S. and Zettlemoyer, Luke , booktitle=

  17. [17]

    Transactions of the Association of Computational Linguistics , volume=

    Natural Questions: A Benchmark for Question Answering Research , author=. Transactions of the Association of Computational Linguistics , volume=

  18. [18]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  19. [19]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Teaching Machines to Read and Comprehend , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  20. [20]

    Transactions on Machine Learning Research , year =

    FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance , author =. Transactions on Machine Learning Research , year =

  21. [21]

    ICML 2024 Workshop on Agentic Markets , year =

    RouterBench: A Benchmark for Multi-LLM Routing System , author =. ICML 2024 Workshop on Agentic Markets , year =

    2024

  22. [22]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =

    Yue, Yanwei and Zhang, Guibin and Liu, Boyang and Wan, Guancheng and Wang, Kun and Cheng, Dawei and Qi, Yiyan , editor =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =. doi:10.18653/v1/2025.acl-long.757 , url =

    doi:10.18653/v1/2025.acl-long.757 2025

  23. [23]

    Gonzalez and M

    Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M. Waleed Kadous and Ion Stoica , booktitle =. 2025 , url =

    2025

  24. [24]

    2023 , url =

    Dongfu Jiang and Xiang Ren and Bill Yuchen Lin , booktitle =. 2023 , url =

    2023

  25. [25]

    International Conference on Learning Representations (ICLR) , year =

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author =. International Conference on Learning Representations (ICLR) , year =

  26. [26]

    and Burger, Doug and Wang, Chi , booktitle =

    Wu, Qingyun and Bansal, Gagan and Zhang, Jieyu and Wu, Yiran and Li, Beibin and Zhu, Erkang and Jiang, Li and Zhang, Xiaoyun and Zhang, Shaokun and Liu, Jiale and Awadallah, Ahmed Hassan and White, Ryen W. and Burger, Doug and Wang, Chi , booktitle =. 2024 , url =

    2024

  27. [27]

    The Twelfth International Conference on Learning Representations , year =

    Hong, Sirui and Zhuge, Mingchen and Chen, Jonathan and Zheng, Xiawu and Cheng, Yuheng and Wang, Jinlin and Zhang, Ceyao and Wang, Zili and Yau, Steven Ka Shing and Lin, Zijuan and Zhou, Liyang and Ran, Chenyu and Xiao, Lingfeng and Wu, Chenglin and Schmidhuber, J. The Twelfth International Conference on Learning Representations , year =

  28. [28]

    2023 , url =

    Li, Guohao and Hammoud, Hasan Abed Al Kader and Itani, Hani and Khizbullin, Dmitrii and Ghanem, Bernard , booktitle =. 2023 , url =

    2023

  29. [29]

    2023 , eprint =

    Sirui Hong and Mingchen Zhuge and Jonathan Chen and Xiawu Zheng and Yuheng Cheng and Ceyao Zhang and Jinlin Wang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu and J. 2023 , eprint =

    2023

  30. [30]

    Neural Network Ensembles, Cross Validation, and Active Learning , url =

    Krogh, Anders and Vedelsby, Jesper , booktitle =. Neural Network Ensembles, Cross Validation, and Active Learning , url =

  31. [31]

    arXiv preprint arXiv:1504.04909 , year =

    Illuminating Search Spaces by Mapping Elites , author =. arXiv preprint arXiv:1504.04909 , year =

    Pith/arXiv arXiv

  32. [32]

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How

    Melanie Sclar and Yejin Choi and Yulia Tsvetkov and Alane Suhr , booktitle =. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How. 2024 , url =

    2024

  33. [33]

    Findings of the Association for Computational Linguistics: EACL 2024 , pages =

    Prompt Perturbation Consistency Learning for Robust Language Models , author =. Findings of the Association for Computational Linguistics: EACL 2024 , pages =. 2024 , url =

    2024

  34. [34]

    2025 , url =

    Zhuang, Richard and Wu, Tianhao and Wen, Zhaojin and Li, Andrew and Jiao, Jiantao and Ramchandran, Kannan , booktitle =. 2025 , url =

    2025

  35. [35]

    The Bell system technical journal , volume=

    A mathematical theory of communication , author=. The Bell system technical journal , volume=. 1948 , publisher=

    1948

  36. [36]

    International Conference on Learning Representations , year =

    Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations , year =

  37. [37]

    2022 , url =

    Stephanie Lin and Jacob Hilton and Owain Evans , booktitle =. 2022 , url =

    2022

  38. [38]

    2019 , url =

    Maarten Sap and Hannah Rashkin and Derek Chen and Ronan Le Bras and Yejin Choi , booktitle =. 2019 , url =

    2019

  39. [39]

    2020 , url =

    Yonatan Bisk and Rowan Zellers and Ronan Le Bras and Jianfeng Gao and Yejin Choi , booktitle =. 2020 , url =

    2020

  40. [40]

    2022 , url =

    Ankit Pal and Logesh Kumar Umapathi and Malaikannan Sankarasubbu , booktitle =. 2022 , url =

    2022

  41. [41]

    2019 , doi =

    Aida Amini and Saadia Gabriel and Shanchuan Lin and Rik Koncel-Kedziorski and Yejin Choi and Hannaneh Hajishirzi , booktitle =. 2019 , doi =

    2019

  42. [42]

    2020 , url =

    Jian Liu and Leyang Cui and Hanmeng Liu and Dandan Huang and Yile Wang and Yue Zhang , booktitle =. 2020 , url =

    2020

  43. [43]

    Bowman , booktitle =

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle =. 2024 , url =

    2024

  44. [44]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =

    A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers , author =. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =. 2020 , url =

    2020

  45. [45]

    2019 , url =

    Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi , booktitle =. 2019 , url =

    2019

  46. [46]

    2021 , url =

    Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi , journal =. 2021 , url =

    2021

  47. [47]

    Think You Have Solved Question Answering?

    Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , journal =. Think You Have Solved Question Answering?. 2018 , url =

    2018

  48. [48]

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P.\ Xing and Hao Zhang and Joseph E.\ Gonzalez and Ion Stoica , booktitle =. Judging. 2023 , url =

    2023

  49. [49]

    arXiv preprint arXiv:2503.07891 , year=

    Gemini embedding: Generalizable embeddings from gemini , author=. arXiv preprint arXiv:2503.07891 , year=

    Pith/arXiv arXiv

  50. [50]

    Mountain View, CA: Google , year=

    A new era of intelligence with gemini 3 , author=. Mountain View, CA: Google , year=