{"id":"dd2c8078-cb9c-4543-b703-f81c5d2804f1","arxiv_id":"2607.09197","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Routing is meaningful only with behavioural diversity (adapted Hierarchic Social Entropy) and assignment stability under query perturbations; accuracy alone can hide vacuous or brittle routing, and fewer than ten models often recover most diversity.","lead":"Routing among language models is only meaningful when the models behave differently and when similar questions are sent to the same model. The paper gives metrics for both properties and shows that a small curated set of models captures most diversity while accuracy-focused routers can be brittle under rewording.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Robustness metric rests on untested assumption that meaning-preserving perturbations never change the optimal actor.","rationale":"The reader correctly flagged the semantic-equivalence assumption in Section 5.2 as the weakest link. That assumption is load-bearing for the headline empirical result (accuracy and meaningfulness diverge). The paper supplies no direct check that optimal actors remain invariant under its own perturbations, and the synthetic societies maximise the risk of legitimate domain-boundary crossings. The proposed test is cheap given the existing matrices and would either confirm the metric or quantify the confounding. Other limitations (greedy coresets, fixed E, proprietary backends) are secondary; they do not undercut the central claim as directly. Verdict therefore stays CONDITIONAL, now with the concrete validation step made explicit. No stronger objection (internal inconsistency, circularity, or fabrication) is present.","tokens_in":19569,"tokens_out":524,"duration_ms":6211,"concrete_test":"On the EmbedLLM and RouterBench test sets, compute the true optimal actor for every original query and every generated perturbation (using the existing correctness matrix or re-scoring). Report the fraction of cases where the optimal actor actually changes under each of the five perturbation levels. If that fraction exceeds ~10–15 % (especially on SA/RD societies), recompute ρ only on the subset where the optimal actor is invariant; if the KNN collapse largely disappears, the divergence claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that accuracy and meaningfulness can diverge (KNN high accuracy + low ρ on specialist societies; prompted reverse) treats every routing change under perturbation as pure policy instability. Section 5.2 states the assumption explicitly: “if qi and qji are semantically equivalent, the optimal assignment should be the same for both.” The paper never validates this. Synthetic SA/RD societies have hard binary domain boundaries (oracle accuracy 1.0 by construction), so a paraphrase that crosses a subject or reasoning-depth boundary legitimately changes the best actor; the metric then scores that reassignment as brittleness. Even on real models, a surface rewrite can shift difficulty or domain enough that a different model is truly better. Without ground-truth optimal-actor labels on the perturbed set, the reported accuracy–robustness tradeoff is partly confounded by legitimate reassignment rather than pure router instability. This is the single most load-bearing untested premise for the strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that multi-model routing is meaningful only when two properties orthogonal to accuracy hold: (i) the actor society is behaviourally differentiated, and (ii) the routing policy is stable under surface-form query variants. It adapts Hierarchic Social Entropy (HSE) to LM societies via cosine distance on behavioural score vectors and hierarchical clustering, and defines a perturbation-based robustness metric ρ over a five-level taxonomy. On EmbedLLM and RouterBench, HSE shows strong diminishing returns (a curated subset of fewer than ten agents recovers most diversity), specialist synthetic societies are far more diverse than real model pools, and KNN routers gain clean accuracy on high-HSE specialist societies while collapsing under perturbation, whereas prompted routing remains comparatively stable—illustrating that accuracy and meaningfulness can diverge.","tokens_in":19825,"tokens_out":1260,"duration_ms":18703,"significance":"If the claims hold, the work supplies two practical, independently measurable diagnostics for multi-LLM and multi-agent system design that current accuracy/cost-only evaluations miss. The coreset-style finding that most behavioural diversity is captured by a small curated subset is immediately actionable for society design. The HSE adaptation is carefully validated with sanity checks (identical vs orthogonal societies; redundancy does not inflate HSE) and is shown to be qualitatively stable across distance/linkage choices (Appendix E). The explicit separation of diversity (society-only) from robustness (policy-dependent) is a clean conceptual contribution to the multi-agent and routing literature.","major_comments":[{"comment":"Section 5.2 states the load-bearing assumption that meaning-preserving perturbations do not change which actor is most accurate: any change in π is treated as pure policy instability. This is never validated. On the synthetic SA/RD societies (binary domain indicators, oracle accuracy 1.0 by construction), a paraphrase or rambling rewrite that crosses a subject or reasoning-depth boundary legitimately changes the optimal actor; ρ then scores correct reassignment as brittleness. Even on real models, surface rewrites can shift difficulty. Without ground-truth optimal-actor labels (or at least accuracy of the original assignee) on the perturbed set, the headline accuracy–robustness divergence in Figure 4 is partly confounded. Please either (a) report how often the originally optimal actor remains optimal under each perturbation class, or (b) qualify the KNN “collapse” claims, especially on s","section":"Section 5.2, Figure 4"},{"comment":"The strongest empirical claim—that KNN gains accuracy from specialist societies but collapses in robustness—rests primarily on the synthetic SA/RD/RD+SA societies with maximally sharp behavioural boundaries (Section 6, Figure 4). The Limitations section notes that real specialists will have softer profiles and that brittleness may be attenuated, but the main text and abstract present the divergence as a general illustration. Please either add a parallel analysis restricted to real-model max-HSE subsets (where Figure 5 already shows a more mixed KNN picture on EmbedLLM) as the primary evidence, or clearly mark the synthetic results as an upper-bound stress test rather than the central demonstration.","section":"Section 7, Figure 4; Limitations"},{"comment":"HSE is defined over a fixed evaluation set E (Section 3–4). When E is not representative of the deployment query distribution, behavioural vectors and therefore diversity rankings can be misleading (Limitations). The paper does not report sensitivity of the max-HSE subset or the diminishing-returns plateau to the choice of E (e.g., train vs held-out, or domain-stratified subsets). A short sensitivity check would substantially strengthen the coreset heuristic claim that “fewer than ten agents recover most available diversity.”","section":"Section 4, Figure 3; Limitations"}],"minor_comments":[{"comment":"Normalisation of HSE is used throughout the figures but the exact normaliser (maximum achievable HSE for society size N) is only briefly stated; a short formula or reference to the integral bound would help readers reproduce the y-axes of Figures 3 and 7.","section":"Section 4.3–4.4"},{"comment":"In Figure 4, solid vs dashed lines for original vs perturbed accuracy are easy to miss in greyscale; consider markers or a clearer legend.","section":"Figure 4"},{"comment":"Appendix B.1 uses Gemini 3.1 Flash-Lite to generate perturbations; a brief note on whether human spot-checks confirmed semantic preservation (especially for Level 5 rambling) would increase confidence in the taxonomy.","section":"Appendix B.1"},{"comment":"Table 7 (Appendix F) is valuable for the joint HSE+accuracy coreset but is not referenced in the main-text discussion of society design; a one-sentence pointer in Section 7 would help.","section":"Section 7; Appendix F"},{"comment":"Typo/consistency: “Emay be set equal toQ” (Section 3) needs a space; “therouting decisionis” (Section 2) appears to have missing spaces in the source.","section":"Section 2–3"}],"recommendation":"major_revision","confidential_remarks":"The conceptual framing is strong and the paper is a good fit for a multi-agent / systems venue. The main risk is overclaiming the accuracy–meaningfulness divergence from synthetic hard-boundary societies without validating the optimal-actor-stability assumption. If the authors add the suggested checks or carefully qualify, this should be a solid contribution; I would not reject on novelty or scope grounds."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: accuracy alone does not tell you whether routing is doing anything useful. The paper adapts Hierarchic Social Entropy to behavioural score vectors (cosine + single linkage) and adds a five-level perturbation metric that scores assignment stability rather than answer stability. Both are cleanly defined and independent of the conclusions.\n\nWhat is new and well done: the HSE sanity checks (identical vs orthogonal societies, duplicate models/prompts), the clear diminishing-returns curves on EmbedLLM and RouterBench (most diversity recovered with <10 agents), and the head-to-head showing KNN gaining clean accuracy on specialist societies while collapsing under perturbation while prompted routing stays flat. Appendix E shows the qualitative HSE pattern is stable across distance/linkage choices. Citations are appropriate (Balch, Bettini, RouterBench, etc.). Math is elementary Shannon + hierarchical clustering; no load-bearing errors.\n\nSoft spots, in proportion. The stress-test is right on the main one: Section 5.2 assumes meaning-preserving rewrites never change the optimal actor. Synthetic SA/RD societies have hard binary boundaries and oracle accuracy 1.0, so a paraphrase that crosses a domain edge is legitimately reassigned yet scored as brittleness. Even on real models the paper never supplies ground-truth optimal labels on the perturbed set. That confounds the strongest claim (accuracy and meaningfulness can sharply diverge) but does not erase the empirical pattern. Other limitations they already flag: greedy max-HSE, fixed evaluation set E, proprietary backends, no released code. These are real but secondary.\n\nThis is for people building or evaluating multi-model routers and multi-agent societies. It is not a theory paper and does not claim to be. The coreset heuristic and the two orthogonal metrics are immediately usable. I would send it to peer review; a referee can demand softer specialists and a check of the optimal-actor assumption without the contribution disappearing. Worth reading and citing if you work in this area.","headline":"Solid empirical framing of when multi-LLM routing is meaningful; the accuracy–robustness divergence is real but partly confounded by the untested optimal-actor assumption.","tokens_in":20439,"tokens_out":501,"would_cite":true,"duration_ms":5647,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Routing among language models is meaningful only when the society is behaviourally diverse and the router is stable under surface-form query variants.","keywords":["LLM routing","Hierarchic Social Entropy","behavioural diversity","routing robustness","multi-agent societies","specialisation","perturbation stability","coreset selection"],"falsifier":"Construct a set of paraphrases or syntactic variants that systematically change which model is most accurate (for example by shifting domain difficulty or required expertise) and check whether the robustness metric still labels those reassignments as failures; if it does, the central diagnostic collapses.","tokens_in":20456,"feed_emoji":"🔀","tokens_out":640,"duration_ms":7354,"temperature":0.7,"pith_summary":"Most evaluations of multi-model routers care only about task accuracy and cost. This paper argues that two structural properties, independent of raw performance, decide whether routing is actually meaningful. The society of models must be behaviourally differentiated; if every actor answers the same way, the router has nothing real to choose. The routing policy must also be stable: paraphrases, typos, and other surface variants of a query should still go to the same actor, otherwise specialisation cannot emerge. The authors adapt Hierarchic Social Entropy from robotics to measure behavioural diversity from model outputs, and introduce a five-level perturbation robustness score for routing decisions. On EmbedLLM and RouterBench they show that diversity saturates quickly—a curated subset of fewer than ten models recovers most of the diversity in a large pool—and that KNN routers can look accurate on specialist societies while collapsing under perturbation, whereas prompted routing stays stable. Accuracy and meaningfulness can therefore diverge sharply.","feed_headline":"Most diversity sits in fewer than ten models","feed_subtitle":"Accuracy can rise while routing becomes meaningless; stability under paraphrases is the missing test.","key_machinery":"Hierarchic Social Entropy (HSE) adapted to language-model societies: behavioural vectors of scores on an evaluation set are compared by cosine distance, hierarchical single-linkage clustering is performed at every distance threshold, and simple social entropy is integrated across all taxonomic levels to yield a single diversity score. Paired with a per-query robustness metric that counts how often five classes of surface perturbation are routed to the same actor as the original.","core_discovery":"High task accuracy is compatible with vacuous routing: a router can score well while operating over a redundant society of near-identical models or while assigning surface-form variants of the same query to different actors. Behavioural diversity measured by Hierarchic Social Entropy and assignment stability under meaning-preserving perturbations are therefore necessary diagnostics that accuracy alone cannot supply.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Most diversity packs into fewer than ten models","Accuracy can climb while routing stays vacuous","HSE plateaus fast: small coresets recover society diversity","KNN lifts scores yet collapses under paraphrase perturbation","Meaningful routing needs diversity plus assignment stability"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The paper assumes that a meaning-preserving surface change never legitimately changes which model is best for a query, so any reassignment under perturbation is pure router instability rather than a real change in suitability.","fun_headline_variants_meta":{"raw":{"variants":["Most diversity packs into fewer than ten models","Accuracy can climb while routing stays vacuous","HSE plateaus fast: small coresets recover society diversity","KNN lifts scores yet collapses under paraphrase perturbation","Meaningful routing needs diversity plus assignment stability"]},"model":"grok-4.5","effort":"low","cost_usd":0.007048,"raw_usage":{"total_tokens":1730,"prompt_tokens":737,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":70480000,"prompt_tokens_details":{"text_tokens":737,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":921,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":737,"tokens_out":72,"duration_ms":9293,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T04:45:36.424413+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Construct a set of paraphrases or syntactic variants that systematically change which model is most accurate (for example by shifting domain difficulty or required expertise) and check whether the robustness metric still labels those reassignments as failures; if it does, the central diagnostic collapses.","supporting_citations":[],"review_version":1}