{"id":"2a81e345-431b-486e-9013-ddf93915447a","arxiv_id":"2608.11027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A label-free behavioral map of 32 LLMs shows stable family clusters, gpt-2 as a global outlier, and decreasing cross-family distances over time; a token-level MMD check and three alternative encoders reproduce the patterns.","lead":"This paper maps how 32 language models from six families behave by comparing their answers to 10,000 shared prompts. It finds stable family clusters, a gpt-2 outlier, and a trend of newer models becoming more behaviorally similar across vendors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-sample responses confound dispersion with convergence: the decreasing cross-family distance may reflect shrinking within-model response spread rather than behavioral homogenization.","rationale":"The reader's CONDITIONAL verdict centers on missing variance estimates from single-sample responses. I agree this is the load-bearing gap, but the mechanism is more specific than general sampling noise: the single-sample distance is a biased estimator of the between-model mean-embedding distance, with bias growing with within-model generation spread (Eq. 16, Appendix A.4). The paper's own dispersion finding (Section 4, Fig. 5) shows that spread is decreasing for recent reasoning-oriented models, which is exactly the condition needed to manufacture the observed convergence trend even under stationary conditional means. The encoder-robustness and token-MMD checks are strong on representation, but they reuse the same outputs, so they cannot disambiguate this confound. This does not invalidate the framework or the theory; it makes the central empirical claim conditional on a dispersion correction that the paper does not supply. A multi-sample conditional-mean distance test would settle it. The verdict remains CONDITIONAL as the reader stated, with the condition sharpened to: demonstrate that the convergence and outlier findings survive when within-model dispersion is removed.","tokens_in":21825,"tokens_out":4747,"duration_ms":43161,"concrete_test":"For all 32 models and a 2,000-prompt subset of the shared bank, collect K=10 independent responses per prompt. Estimate per-prompt conditional mean embeddings mu_hat_m(x) by averaging the K response embeddings, and recompute the aligned distance D_mu(m,m') = (1/N) * sum_i ||mu_hat_m(x_i) - mu_hat_m'(x_i)||. Compare the release-date trend, family nearest-neighbor counts, and outlier rankings under D_mu versus the original single-sample Dmean; also compute the dispersion term omega_m as the average distance of single responses to their conditional mean, and regress (Dmean - D_mu) on release date. If the negative cross-family slope and gpt-2 outlier survive under D_mu, the single-sample concern is largely resolved; if they attenuate, the convergence claim is a dispersion artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's central convergence claim ('cross-family distances decrease over time') is computed from exactly one generated response per model-prompt pair. For such data, Dmean in Eq. (1) estimates E||Z_m - Z_m'||, which by Eq. (16) in Appendix A.4 is controlled by the mean-embedding distance plus within-prompt spread terms omega_m + omega_m'. The paper's own response-cloud dispersion analysis (Fig. 5 bottom) shows that several recent reasoning-oriented models (claude-opus-4-7-think, gpt-5.2) have comparatively compact response clouds, i.e., smaller within-model spread. If within-model spread decreases over time, then Dmean will decrease even when the conditional mean behavior ||mu_m(x) - mu_m'(x)|| is constant or increasing. The token-level MMD cross-check (rho=0.98) and the three alternative encoders all re-encode the same single responses, so they inherit the same dispersion dependence and cannot separate it from genuine behavioral convergence. The sufficient-condition theory in Section 6 explicitly carries the omega terms (Eq. 16), but no estimate or correction is provided. Therefore the headline empirical pattern, and the gpt-2 outlier, which may reflect high generation entropy, is not yet distinguished from a dispersion artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a label-free framework for comparing LLM behavior directly from generated text. Using 32 models from six families responding to a shared bank of 10,000 prompts, it constructs three sentence-level dissimilarities (a mean per-prompt distance, a PCA-compressed summary of prompt-wise disagreement, and a Gromov–Wasserstein discrepancy) plus a token-level per-prompt MMD cross-check. These are used to build behavioral maps, study family-wise drift, hierarchical clustering, cross-family convergence, and response-cloud dispersion. The paper claims that model families form coherent clusters, that gpt-2 is a global outlier, that cross-family distances decrease over time, and that several recent reasoning-oriented models have comparatively compact response clouds. It also proves a sufficient condition linking behavioral similarity to effective target-distribution similarity, small excess population log-loss, and inference-prompt coverage, while explicitly stating that the training-side quantities are unobserved. The main empirical results are robust across four response encoders, three projection methods, and token-level versus sentence-level representations, but all of these reuse exactly one generated response per model–prompt pair.","tokens_in":22085,"tokens_out":4929,"duration_ms":48913,"significance":"If the empirical findings survive additional sampling scrutiny, the paper would make a useful contribution: a label-free, closed-model-compatible way to measure behavioral lineage and temporal homogenization would complement benchmark leaderboards and be of broad interest to the LLM evaluation community. The robustness checks across encoders, projections, and the token-level MMD are genuine strengths, and the measure-theoretic framing and the sufficient-condition theory in Section 6 are correct and carefully scoped. The paper is also unusually transparent about its limitations. The central caveat is that the headline convergence claim rests on single-sample distances with no uncertainty quantification, and the paper's own Appendix A.4 shows that those distances carry within-prompt generation variability; until that variability is quantified or controlled, the decreasing cross-family distance is not fully separated from a dispersion artifact. This is an addressable issue rather than a fatal flaw, so the result warrants major revision rather than rejection.","major_comments":[{"comment":"The headline convergence claim (\"cross-family distances decrease over time\") rests entirely on distances computed from exactly one generated response per model–prompt pair. As the paper itself shows in Eq. (16), the expected single-sample per-prompt distance is controlled not only by the mean-embedding discrepancy eta_sem but also by within-prompt spread terms omega_m + omega_m'. Because Fig. 5 (bottom) documents that within-model response-cloud dispersion changes across releases, with several recent reasoning-oriented models markedly more compact, the decreasing cross-family distance in Fig. 5 (top) is not yet distinguished from a decreasing-dispersion artifact. The token-level MMD cross-check and the alternative-encoder checks re-encode exactly the same single responses, so they inherit the same dependence on generation variability and cannot separate it from genuine behavioral convergence. I request either an explicit dispersion control (e.g., estimating omega_m per model from repeated sampling), a repeated-sampling experiment for at least a subset of models, or a variance-decomposition analysis showing that the observed trend survives after conditioning on within-model dispersion.","section":"Section 4 and Appendix A.4, Eq. (16)"},{"comment":"The central statistics—nearest-neighbor counts, Spearman correlations, fitted convergence slopes, and the identity of the two outliers—are reported as point estimates with no uncertainty quantification. In particular, Fig. 5 (top) shows a fitted slope of -1.37e-02/day with no confidence interval, and Table 7 reports trend correlations without intervals. Because the 32 models are not independent draws, many release dates in Table 3 are approximate, and each distance uses a single response per prompt, a bootstrap over prompts, a sensitivity analysis over approximate release dates, and repeated sampling for at least a subset of models are needed before the decreasing trend can be treated as established. This is load-bearing for the paper's central empirical claim, not a presentation detail.","section":"Section 4 and Section 5, Fig. 5 and Table 7"}],"minor_comments":[{"comment":"The phrase \"sign orientation, symmetrization, and rescaling\" is not a precise definition; please spell out the exact operation used to obtain the displayed D_pca matrix, since this is a key construction.","section":"Section 3, Eq. (2)"},{"comment":"The response-cloud dispersion panel has no fitted trend or uncertainty; consider adding a slope estimate and a caption stating whether the apparent decline is statistically significant.","section":"Figure 5 (bottom)"},{"comment":"The sentence \"The proof is given in Appendix A.1\" appears immediately before the proof that is already in Appendix A.1; please delete the redundant sentence.","section":"Appendix A.1"},{"comment":"The statement that the plug-in MMD estimator's finite-sample bias \"does not materially alter\" rankings is based on high empirical agreement; please report the distribution of response lengths per model so that readers can assess length-induced bias across families.","section":"Section 5"},{"comment":"The notation for the PCA construction defines x_bar but not the operator S; please define all symbols used in the equation.","section":"Section 3, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"I agree with the conditional assessment in the reader's report. The paper's strengths are real, and I would not reject it, but the central empirical claim about cross-family convergence needs repeated-sampling evidence or a dispersion control before it can be accepted. The issue is addressable within the manuscript's scope, so major revision is the appropriate decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a thoughtful, well-scoped empirical paper, but the headline convergence result is not yet separated from a sampling artifact. The strongest new thing is a release-date behavioral map of 32 models from six families, with family coherence, a gpt-2 outlier, and a downward cross-family distance trend that survives four encoders and a token-level MMD cross-check. That robustness is real and is the paper's best evidence. The three constructions are carefully distinguished—aligned mean, PCA-compressed disagreement, alignment-free GW—and the measure-theoretic framing is clear. The theorem (training-side sufficient condition) is correct but, as the authors say, a possible account rather than an empirical explanation.\n\nThe soft spot is load-bearing. Every distance is computed from exactly one generated response per model-prompt pair. For Dmean, that estimates E||Z_m - Z_m'||, which by the paper's own Eq. (16) is bounded by mean-embedding distance plus within-prompt spread terms omega_m + omega_m'. The paper's dispersion analysis shows recent reasoning models have more compact response clouds, i.e. smaller omega. So a decreasing cross-family distance is exactly what you'd expect if within-model spread shrinks over time, even with constant conditional mean behavior. The token MMD cross-check and the encoder swaps all run on the same single responses, so they inherit the same dependence and cannot separately identify convergence from shrinking dispersion. The authors acknowledge the single-response limitation in Section 8 but do not quantify it or show the trend survives repeated sampling. That's the main thing a referee should push on.\n\nLesser issues: no code or data released, which matters for a measurement paper; the GW subsample of 256 prompts is small; and some release dates are approximate. None of these are fatal in themselves.\n\nOn the plus side, the paper is transparent about what is descriptive and what is theoretical. It does not relabel a fit as a prediction. The encoder-robustness checks are well done, including using encoders with no vendor overlap with the evaluated models. The literature coverage (LLM DNA, LLMmap, behavioral fingerprinting, Platonic representation debate) is fair.\n\nBottom line: for someone working on LLM monitoring or behavioral similarity, this is a useful map and a good framing, but the convergence claim should be treated as a hypothesis until repeated sampling gives error bars. Send it to peer review with a request for uncertainty quantification and code/data release. I would not reject it.","headline":"A careful behavioral map of 32 LLMs, but the headline convergence trend is confounded by single-sample responses and shrinking response spread; deserves refereeing, not rejection.","tokens_in":22573,"tokens_out":2981,"would_cite":true,"duration_ms":28199,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper maps and measures the behavioral evolution of large language models from their outputs alone, finding lineage-based clusters, a gpt-2 outlier, and decreasing cross-family distances over time.","keywords":["behavioral evolution","large language models","label-free evaluation","response embeddings","Gromov–Wasserstein distance","maximum mean discrepancy","cross-family convergence","model families"],"falsifier":"Generate K independent responses per model-prompt pair (say K=10) on a subset of the prompt bank, recompute the mean per-prompt distance and token MMD for each replicate, and check whether the same-family nearest-neighbor fraction, the gpt-2 outlier status, and the negative cross-family release-date slope survive. If distances between two independent responses from the same model rival distances between different models, the reported geometry is a single-sample artifact.","tokens_in":21637,"feed_emoji":"🧠","tokens_out":12571,"duration_ms":94923,"temperature":0.7,"pith_summary":"This paper asks whether the output behavior of large language models can be mapped and measured without ground-truth labels, and it answers yes using 32 models from six families, each responding to the same 10,000 prompts. The authors compare responses through three complementary constructions on sentence embeddings (an aligned mean per-prompt distance, a PCA-compressed disagreement score, and an alignment-free Gromov–Wasserstein discrepancy), then cross-check the picture with a token-level maximum mean discrepancy analysis. On a release-date axis they find that model families form coherent behavioral clusters, that gpt-2 is a global outlier (with qwen-2.5-72b-instruct as a second outlier), that cross-family distances decrease over time, and that several recent reasoning-oriented models have comparatively compact response clouds. The patterns survive across three projections, four response encoders including one 73× smaller, and the token-level cross-check, which agrees with the sentence-level mean distance at Spearman rank correlation 0.98. The paper also proves a training-side sufficient condition under which small excess log-loss, good inference-prompt coverage, and similar effective target distributions force behavioral similarity, offering a possible explanation rather than an empirical account of the observed homogenization.","feed_headline":"LLM behavior maps: families cluster, distances shrink.","feed_subtitle":"A label-free study of 32 models finds lineage coherence, a gpt-2 outlier, and cross-family convergence over time.","key_machinery":"The load-bearing machinery is a set of pairwise dissimilarities computed from a shared, row-aligned prompt bank. The first, the aligned mean per-prompt distance $D^{\\mathrm{mean}}_{m,m'}=N^{-1}\\sum_i \\|E_m[i]-E_{m'}[i]\\|_2$, averages Euclidean distances between embeddings of the same prompt and is a pseudometric on observed response matrices, becoming a metric after quotienting out models with identical encoded responses. The second is a PCA-compressed score: each model pair's vector of prompt-wise distances is projected onto its leading principal direction, summarizing how pairs disagree across prompts. The third is the Gromov–Wasserstein discrepancy $D^{\\mathrm{gw}}_{m,m'}$, which compares each model's internal response geometry (the matrix of cosine dissimilarities among its own responses) under an optimized coupling, so it discards prompt alignment and is invariant to orthogonal transformations and positive rescaling of embedding vectors. The token-level cross-check replaces each response by the empirical measure of its token embeddings and computes the squared maximum mean discrepancy with an RBF kernel prompt by prompt. These constructions vary deliberately along representation, aggregation, and alignment, so their high rank agreement (Spearman 0.98 between the mean distance and token MMD) is the paper's main stability argument. The theoretical engine is Theorem 1, which bounds the expected total variation between two deployed models by a triangle inequality through their effective target distributions, with learning-error terms transferred from training prompts to inference prompts via a density-ratio coverage condition and Pinsker's inequality: $$\\mathbb{E}_{x\\sim\\xi}\\,\\mathrm{TV}(q_m(\\cdot|x),q_{m'}(\\cdot|x)) \\;\\le\\; \\$eta^{{\\mathrm{txt}}$}_{m,m'} + \\sqrt{\\kappa_m\\varepsilon_m/2} + \\sqrt{\\kappa_{m'}\\varepsilon_{m'}/2}.$$ Here $\\eta^{\\mathrm{txt}}_{m,m'}$ is the expected total variation between the two effective target distributions on the inference prompts, $\\varepsilon_j$ is each model's excess population log-loss, and $\\kappa_j$ bounds the inference-to-training density ratio. The bound is architecture-agnostic: conditional on these quantities it does not depend on initialization, optimizer, or architecture.","core_discovery":"The central discovery is that an unlabeled, output-only geometry over models reproduces what leaderboard scores hide: LLMs are organized by lineage, and that lineage structure partially dissolves among recent releases. Under the mean per-prompt distance, 21 of 32 models have a nearest neighbor from their own family, and gpt-2 has the largest average distance to all other models (about 88.5), making it the global outlier; qwen-2.5-72b-instruct separates second. Mean distance from each model to models of other families decreases with release date, giving a negative fitted trend (cross-family slope $-1.37\\times10^{-2}$ per day under the sentence-level construction), which the paper reads as descriptive evidence of behavioral homogenization rather than a causal mechanism. Several recent reasoning-oriented models, including claude-opus-4-7-think and gpt-5.2, show lower response-cloud dispersion under the chosen encoder. The token-level MMD matrix correlates with the sentence-level mean at Spearman $\\rho=0.98$, recovers the same two outliers and the same decreasing trend, and the rank geometry is preserved when every response is re-encoded with three further encoders, down to one 73 times smaller. The authors frame these as stable empirical patterns, not as claims about capability or about the training causes of drift.","pith_inferences":["If the single-sample distances were recomputed with repeated generation, the family clusters could prove either more stable or largely sampling noise; the paper's cross-encoder agreement does not bound this generation variability, so a direct variance study is the natural next check.","The cross-vendor Gromov–Wasserstein pairings of recent models (for example, gemini-3.1-pro-preview with qwen3.7-max and an opus-class Claude model with gpt-5.2) suggest that internal response-cloud geometry is converging even where prompt-aligned distances still separate families; testing whether these pairs share error patterns on benchmark categories would sharpen the convergence claim.","A practical extension would turn the convergence trend into a rolling monitoring statistic: recompute mean cross-family distance in sliding release-date windows; if the negative slope flattens or reverses, that would indicate a shift in training objectives or decoding policies. The paper does not propose such a monitor.","The theory's coverage condition points to a concrete diagnostic: if training-prompt distributions were available for each model, one could test whether models with smaller density-ratio bounds $\\kappa$ converge faster, as Theorem 1 predicts; this would convert the sufficient condition into a falsifiable prediction about training data."],"forward_implications":["Closed frontier models can be placed in the same behavioral geometry as open models using only their generated text, since every distance is computed from outputs rather than weights or activations.","The observed decrease in cross-family distance means models from different vendors are becoming harder to distinguish by their outputs; monitoring this trend could serve as an early signal of behavioral homogenization.","The compact response clouds of several recent reasoning-oriented models indicate that those models concentrate their outputs in a smaller embedding region for the same prompt bank, which is consistent with more deterministic decoding but is not itself a capability claim.","The low-rank structure of prompt-wise disagreement (leading component explains 63% of variance) implies that a small set of high-leverage prompts could reproduce much of the model geometry, enabling cheaper drift monitoring without running the full 10,000-prompt bank.","Because Theorem 1 is a sufficient condition, the empirical convergence could arise from other causes; conversely, models can be behaviorally close in the measured geometry without satisfying the training-side assumptions."],"supporting_citations":[{"why":"Defines the Gromov–Wasserstein distance that the paper uses as its alignment-free comparison of internal response geometries.","marker":"Mémoli 2011"},{"why":"Provides the optimal-transport formulation and computational grounding for computing the GW discrepancy.","marker":"Peyré and Cuturi 2019"},{"why":"Supplies the maximum mean discrepancy used as the independent token-level cross-check of the sentence-level picture.","marker":"Gretton et al. 2012"},{"why":"Establishes the sentence-embedding paradigm the paper follows when encoding each complete response.","marker":"Reimers and Gurevych 2019"},{"why":"Introduces GPT-2, the model identified as the global behavioral outlier, and anchors the GPT family's origin on the release-date axis.","marker":"Radford et al. 2019"},{"why":"Contributes MMLU, one of the 13 public benchmarks from which the shared 10,000-prompt bank is drawn.","marker":"Hendrycks et al. 2021a"},{"why":"Supplies the optimal-transport solver used to compute the Gromov–Wasserstein discrepancy in practice.","marker":"Flamary et al. 2021"}],"fun_headline_variants":["LLM lineage clusters, gpt-2 outlier, cross-family gaps shrink","Response geometry reveals LLM families and temporal convergence","Unlabeled behavioral maps track LLM drift and family ties","Behavioral maps: lineage dominates, then dissipates","32 LLMs mapped: families cluster, distances shrink over time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every distance in the main analysis is computed from exactly one generated response per model and prompt, so the entire behavioral map assumes that generation-to-generation variation is small relative to differences between models.","fun_headline_variants_meta":{"raw":{"variants":["LLM lineage clusters, gpt-2 outlier, cross-family gaps shrink","Response geometry reveals LLM families and temporal convergence","Unlabeled behavioral maps track LLM drift and family ties","Behavioral maps: lineage dominates, then dissipates","32 LLMs mapped: families cluster, distances shrink over time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2826,"prompt_tokens":1133,"completion_tokens":1693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":749,"completion_tokens_details":{"reasoning_tokens":1610}},"tokens_in":749,"tokens_out":1693,"duration_ms":10848,"temperature":1.0,"reasoning_tokens":1610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:47:54.460141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate K independent responses per model-prompt pair (say K=10) on a subset of the prompt bank, recompute the mean per-prompt distance and token MMD for each replicate, and check whether the same-family nearest-neighbor fraction, the gpt-2 outlier status, and the negative cross-family release-date slope survive. If distances between two independent responses from the same model rival distances between different models, the reported geometry is a single-sample artifact.","supporting_citations":[],"review_version":1}