{"id":"98f830ac-814f-481a-aa60-21b63bdeaa46","arxiv_id":"2606.07524","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"ABLE constructs model embeddings from gradient-based input attributions, enabling training-free LLM comparison across 239 models with theoretical stability guarantees.","lead":"ABLE creates compact 'fingerprints' of large language models by measuring how sensitive each model's output is to small changes in its input text. These fingerprints let you compare models, route queries, or predict benchmark scores without running full evaluations.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Theory-experiment gap: stability guarantees cover fixed-parameterization perturbations, but the central empirical contribution relies on cross-architecture comparisons where no such guarantee applies.","rationale":"The reader's verdict of CONDITIONAL with MODERATE confidence is appropriate. The reader correctly identified the most significant weakness: the theory covers fixed parameterizations while the experiments are cross-architecture. This is a real gap, but it does not invalidate the empirical contributions. The paper is transparent about this limitation (§A explicitly states the scope), and the empirical validation on 239 models with multiple downstream tasks provides substantial independent evidence. The interpolation experiment (Appendix C) offers a partial bridge by showing Lipschitz-like behavior within a fixed architecture, and the Mantel test (Appendix F) demonstrates sample stability. The concern is about the framing of the theory as supporting the central claim broadly, when it only supports a narrow slice. This warrants the CONDITIONAL verdict — the empirical work stands, but the theoretical contribution is narrower than the abstract implies. No code or data release further supports keeping the verdict conditional rather than fully accepting. The paper would benefit from either (a) explicitly narrowing the theoretical claims in the abstract to match the actual scope, or (b) providing a formal or semi-formal argument for why cross-architecture comparisons inherit any stability properties. Neither adjustment would change the core empirical findings. I note no additional concerns beyond what the reader identified: the relation prediction dataset (135 pairs) is small but adequate for a proof-of-concept, and the Gradient×Input attribution method is a reasonable choice given the efficiency-stability tradeoff discussed in the paper.","tokens_in":23730,"tokens_out":851,"duration_ms":198972,"concrete_test":"Construct a controlled experiment with two architecture families (e.g., Llama-2-7B and Qwen2-7B) that have documented derivative relationships within each family but not across families. Compute ABLE embeddings for all models and measure: (a) within-family pairwise distances for models with known parameter-space distances, and (b) cross-family distances for models with no documented relationship. If the Lipschitz constant estimated from within-family perturbations predicts cross-family distance ordering, the theory's scope may extend informally; if not, the theory-experiment gap is confirmed as material.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim combines two components: (1) ABLE captures model-specific input-sensitivity patterns via gradient-based attributions, and (2) under standard regularity assumptions, ABLE induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees. The theory (Theorem A.1, Proposition A.1) is explicitly stated for a shared fixed-dimensional parameterization — the paper itself acknowledges this in §A: 'the theory is stated for differentiable Transformer-style models under a shared fixed-dimensional parameterization; the heterogeneous cross-architecture comparisons in the main paper remain empirical.' This means the theoretical guarantee covers only the narrow case of perturbations within a single architecture family (e.g., interpolating between Llama-2-7B variants as in Appendix C), while the primary empirical contributions — relation prediction across 239 heterogeneous models (Table 1), model routing (Table 2), and benchmark score prediction (Table 3) — involve models with different architectures, different parameter dimensions, and different tokenizers. The Lipschitz constant L = B·M·L_φ from Theorem A.1 has no defined value when θ and θ' live in different-dimensional spaces. The reader correctly identified this gap. The concern is load-bearing because the paper's abstract frames the stability analysis as providing 'a principled stability guarantee for ABLE' without clearly scoping that this guarantee does not cover the paper's main experiments. The empirical results may well be valid — the interpolation experiment (Appendix C) and Mantel test (Appendix F) provide partial support — but the theory does not formally underpin the cross-architecture claims that constitute the paper's novelty. This is a scope mismatch between theory and experiments, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"ABLE proposes a model representation framework for heterogeneous LLM ecosystems based on gradient-based feature attributions. The method computes Gradient×Input attributions on a fixed probing set, aligns token-level scores to a unified word vocabulary, and projects the result to a compact embedding via Johnson-Lindenstrauss random projection. The paper validates the approach on 239 open-source LLMs across three tasks: relation prediction, model routing, and benchmark score prediction. A theoretical appendix establishes Lipschitz continuity of the parameter-to-embedding map and finite-sample concentration bounds under a fixed parameterization.","tokens_in":23932,"tokens_out":1202,"duration_ms":128610,"significance":"The paper addresses a genuine gap: existing parameter-space methods require architectural compatibility, while output-based methods may conflate models with similar behavior. The attribution-based approach is a reasonable middle ground. Strengths include the training-free design, the scale of evaluation (239 models), and the inclusion of a self-contained theoretical analysis (Appendix A) with an empirical interpolation check (Appendix C). The tokenizer-agnostic word-level alignment is a practical contribution. The benchmark ranking comparison against output-based baselines (Table 5) provides useful evidence that attribution signals carry information beyond surface outputs.","major_comments":[{"comment":"§Abstract and §1: The abstract states that ABLE 'induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees' without scoping this to a fixed parameterization. The main text (§1, contribution bullet 2) and Appendix A do scope the claim to 'differentiable Transformer-style models with a fixed parameterization,' and the paper explicitly acknowledges that 'the heterogeneous cross-architecture comparisons in the main paper remain empirical' (§A). However, the abstract's unqualified framing creates a mismatch between the stated guarantee and the primary empirical contributions (Tables 1–3), which involve heterogeneous architectures with different parameter dimensions. The Lipschitz constant L = B·M·L_φ from Theorem A.1 has no defined value when θ and θ' live in different-dimensional spaces. This is a presentation issue that affects how readers interpret","section":null},{"comment":"§4.3, Table 1: The relation prediction dataset contains only 135 model pairs, split 8:2 into train and test. This yields approximately 27 test pairs. With 5 random seeds, the standard deviations on Accuracy (±0.042) and AUC (±0.057) are substantial relative to the margins between ABLE (0.867) and the best baseline (0.721). The paper should report the exact number of test pairs and discuss whether the performance gap is statistically significant (e.g., via a paired test or bootstrap confidence interval). The current presentation does not allow the reader to assess whether the improvement over Log-Likelihood is robust or an artifact of small-sample variance.","section":null},{"comment":"§A.2, Proposition A.1, Step 3: The proof sketch for multi-layer composition states that 'perturbation analysis shows that the product ∇_x f remains Lipschitz in θ' but does not provide the actual perturbation bound. Steps 1 and 2 give explicit per-layer Lipschitz constants, but Step 3 only asserts that aggregating across L layers yields a global constant M without deriving how M depends on L, B_W, L_σ, etc. For an L-layer network, the product of Jacobians can lead to exponential dependence on L. The proposition should either state the explicit form of M (even if loose) or clarify that the bound is qualitative and the constant may be vacuously large for deep models.","section":null}],"minor_comments":[{"comment":"§3.4: The JL lemma requires K = O(ε^{-2} log M), but the paper states K = O(ε^{-2} log M) in the main text while Theorem A.2 derives K ≥ (4/ε²) ln(2m²/δ). These should be consistent; the main text omits the δ dependence.","section":null},{"comment":"Table 1: The 'Greedy' baseline description ('predicts that all within-organization pairs are related and cross-organization pairs are unrelated') is ambiguous — does 'organization' refer to the model creator's organization? This should be clarified.","section":null},{"comment":"§4.4, Table 2: The ABLE router accuracy (0.676) is described as 'slightly higher than the fully trained EmbedLLM router' (0.665). The difference is 0.011 with standard deviations of ±0.001 and ±0.003. The claim of superiority should be softened or supported by a significance test.","section":null},{"comment":"Figure 4: The t-SNE visualization is informative but the figure is difficult to read at normal zoom. Consider increasing font size for model names or providing an interactive version.","section":null},{"comment":"§A.1: The notation s_θ^(y)(x) is used in the appendix while the main text uses S_c. Aligning notation would improve readability.","section":null},{"comment":"Appendix B, Figure 7: The axis label 'ABLE Dimension' uses a non-standard character (uni00A0, a non-breaking space). This should be cleaned up.","section":null},{"comment":"§4.5: The ridge regression for benchmark score prediction uses leave-one-out cross-validation. It would be useful to report the regularization hyperparameter selection procedure.","section":null}],"recommendation":"major_revision","confidential_remarks":"The theory-experiment gap identified by the reader is real but the paper is transparent about it in the appendix. The main issue is that the abstract oversells the guarantee. If the authors scope the abstract claim and address the small-sample concern in Table 1, this is a solid contribution. The Proposition A.1 proof gap (Step 3) is the most technically substantive concern and should be addressed before acceptance."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive reading of our manuscript. The referee identifies three major issues: (1) an over-qualified theoretical claim in the abstract, (2) insufficient statistical reporting for the relation prediction experiment given the small test set, and (3) a missing explicit perturbation bound in the multi-layer composition step of Proposition A.1. We agree with all three points and will revise the manuscript accordingly. Below we address each comment in detail.","responses":[{"response":"The referee is correct. The abstract should scope the theoretical claim to fixed-parameterization models, consistent with the main text (§1, contribution bullet 2) and Appendix A, which explicitly state that the guarantee applies to 'differentiable Transformer-style models with a fixed parameterization' and that 'the heterogeneous cross-architecture comparisons in the main paper remain empirical.' We will revise the abstract to include this scoping qualifier. Specifically, we will change the relevant sentence to read approximately: 'under standard regularity assumptions for differentiable Transformer-style models with a fixed parameterization, ABLE induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees.' This ensures the abstract does not overstate the scope of the theoretical result relative to the heterogeneous empirical setting of Tables 1–3.","revision_made":"yes","referee_comment":"The abstract states that ABLE 'induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees' without scoping this to a fixed parameterization. The main text and Appendix A do scope the claim, but the abstract's unqualified framing creates a mismatch with the primary empirical contributions involving heterogeneous architectures. The Lipschitz constant L = B·M·L_φ has no defined value when θ and θ' live in different-dimensional spaces."},{"response":"The referee's concern is well-taken. The 8:2 split of 135 pairs yields exactly 27 test pairs (18 positive, 9 negative, given the 2:1 negative-to-positive sampling ratio). We agree that this is a small test set and that the current presentation does not allow readers to assess statistical significance. In the revision, we will: (1) report the exact number of test pairs (27) explicitly in §4.3; (2) add a paired bootstrap 95% confidence interval for the accuracy and AUC differences between ABLE and each baseline; and (3) include a brief discussion of the limitations of evaluation at this scale. We note that the small-sample limitation is partly inherent to the problem: documented derivative relationships among open-source LLMs are relatively scarce, and we chose to restrict the dataset to pairs with verifiable metadata rather than inflating it with uncertain labels. However, we agree that the paper must be transparent about this constraint and provide the statistical tests the referee requests.","revision_made":"yes","referee_comment":"The relation prediction dataset contains only 135 model pairs, split 8:2 into train and test, yielding approximately 27 test pairs. With 5 random seeds, the standard deviations on Accuracy (±0.042) and AUC (±0.057) are substantial relative to the margins between ABLE (0.867) and the best baseline (0.721). The paper should report the exact number of test pairs and discuss whether the performance gap is statistically significant."},{"response":"The referee is correct that Step 3 of Proposition A.1 is incomplete as written. The product of L layer Jacobians can indeed lead to exponential dependence on L, and the current proof sketch does not make this explicit. In the revision, we will expand Step 3 to derive the explicit form of M. Specifically, for an L-layer network where each layer Jacobian J_l has spectral norm bounded by ρ_l (with ρ_l depending on B_W, L_σ, and the softmax Jacobian bound), the product bound gives M = O(∏_{l=1}^{L} ρ_l · max_l C_l), where C_l is the per-layer Lipschitz constant from Steps 1–2. Under the bounded weight assumption ∥W^l∥ ≤ B_W and activation derivative bound L_σ, each ρ_l ≤ B_W · L_σ, yielding M = O((B_W · L_σ)^L · (L_σ + B_W · L_{σ'} · B_x)). We will state this explicitly and add a remark that this bound grows exponentially with L, making it primarily a qualitative existence result for deep models rather than a tight quantitative guarantee. This is consistent with the paper's framing of the theory as providing a principled stability guarantee rather than a practically tight bound.","revision_made":"yes","referee_comment":"The proof sketch for multi-layer composition in Proposition A.1, Step 3, states that 'perturbation analysis shows that the product ∇_x f remains Lipschitz in θ' but does not provide the actual perturbation bound. Steps 1 and 2 give explicit per-layer Lipschitz constants, but Step 3 only asserts that aggregating across L layers yields a global constant M without deriving how M depends on L, B_W, L_σ, etc. For an L-layer network, the product of Jacobians can lead to exponential dependence on L. The proposition should either state the explicit form of M or clarify that the bound is qualitative and the constant may be vacuously large for deep models."}],"tokens_in":23700,"tokens_out":1127,"duration_ms":117207,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Here's my read on the ABLE paper. The core idea is genuinely new: aggregate gradient-based feature attributions (Gradient×Input) across a fixed probing set, align them at the word level to handle tokenizer differences, and use JL projection to get a compact model-level embedding. Nobody has done this particular combination before, and the motivation is sound — two models can produce identical outputs while relying on very different input features, and capturing that difference gives you a richer signal than output-only methods. The empirical work is solid for what it is. They run on 239 models, which is a real collection. The relation prediction results (Table 1) show ABLE beating output-based baselines, with a clear precision advantage that makes sense given the method's design. The benchmark score prediction (Spearman ρ 0.81–0.89) and routing results (competitive with EmbedLLM without retraining) are practically useful. The interpolation experiment in Appendix C is a nice check — ABLE space preserves the topology of weight-space interpolation on same-architecture models. The Mantel test (Appendix F) showing 0.722 correlation across disjoint probe subsets addresses sample stability reasonably. Now the soft spots. The stress-test concern about the theory-experiment gap is real but I'd calibrate it as moderate, not fatal. The paper is transparent about this — Appendix A explicitly states the theory covers fixed-parameterization models and cross-architecture comparisons remain empirical. The abstract oversells slightly by calling it 'a principled stability guarantee for ABLE' without scoping it, but the body is honest. The theory gives you Lipschitz continuity within an architecture family, which is useful but not what the headline experiments need. The relation prediction dataset is small (135 pairs, 8:2 split) — that's a real limitation for the classification results, though the effect sizes are large enough that I'm not deeply worried. No code or data is shipped, which hurts reproducibility. The JL dimension K=256 is tuned on downstream task performance (Appendix B), which is a minor empirical fitting step but not circular. The character-level alignment is crude — uniform distribution of token attributions across characters — and the authors acknowledge this. It works for English MCQ probes but would likely struggle on code or multilingual text. Overall: this is a useful methods paper with a clean new idea and decent empirical validation. The theory is correctly stated but scoped more narrowly than the experiments. A reader working on model comparison, routing, or provenance auditing gets value from this. It deserves a serious referee who can push on the dataset size, the alignment scheme, and whether the theory can be extended to cover cross-architecture settings.","headline":"New method for LLM comparison via aggregated gradient attributions; theory covers only same-architecture case while main experiments are cross-architecture","tokens_in":24684,"tokens_out":620,"would_cite":true,"duration_ms":57088,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Gradient fingerprints map 239 LLMs without retraining","keywords":["feature attribution","model embedding","LLM representation","gradient-based interpretability","model routing","provenance auditing","tokenizer alignment","Johnson-Lindenstrauss projection"],"falsifier":"If two models with genuinely different internal computation produced identical attribution patterns on the probe set, ABLE would conflate them — the representation's discriminative power depends on the probe set exercising distinct input-sensitivity profiles across models.","tokens_in":23858,"feed_emoji":"🧭","tokens_out":896,"duration_ms":62930,"temperature":0.7,"pith_summary":"ABLE represents each language model by how it responds to input perturbations, not by its outputs or weights. It computes Gradient×Input attributions on a fixed probe set, aligns them across tokenizers at the word level, and projects to a compact embedding. This produces a reusable model fingerprint that distinguishes models with identical outputs but different internal computation. On 239 open-source LLMs, ABLE predicts model family relationships, routes queries, and estimates benchmark scores — all without training. The paper proves that, under bounded weights and smooth activations, the parameter-to-embedding map is Lipschitz continuous with finite-sample convergence.","feed_headline":"Gradient fingerprints map 239 LLMs without retraining","feed_subtitle":"ABLE turns input-sensitivity patterns into reusable model embeddings, predicting family ties, routing, and benchmark scores across diverse L","key_machinery":"Gradient×Input (GI) attribution; tokenizer-agnostic word-level alignment via character-span redistribution; Johnson–Lindenstrauss random projection; Lipschitz continuity of the gradient field in θ under bounded Transformer weights.","core_discovery":"The central mechanism is that gradient-based feature attributions, aggregated across a fixed probe set and aligned at the word level, encode model-specific input-sensitivity patterns that are more discriminative than output behavior alone and remain comparable across heterogeneous architectures. The paper shows this representation is Lipschitz continuous in model parameters (Theorem A.1), preserves pairwise distances under random projection (Theorem A.2), and concentrates around its population value with finite samples (Theorem A.3). Empirically, on 239 LLMs, ABLE achieves 0.867 accuracy in relation prediction (vs. 0.721 for the best output-based baseline), 0.676 routing accuracy with frozen","pith_inferences":["The theoretical stability guarantee covers only homogeneous models under a shared parameterization, while the empirical results span heterogeneous architectures. Whether the Lipschitz bound transfers across architecture boundaries remains an open question; the cross-architecture results may rest on the word-level alignment acting as an implicit normalization rather than on the proven continuity.","If higher-fidelity attribution methods (Integrated Gradients, SmoothGrad) were substituted for Gradient×Input, the embedding might capture finer-grained sensitivity patterns, but the single-pass efficiency that makes ABLE scalable would be lost.","The character-level alignment uniformly distributes token attributions across characters, which may lose subword-level semantic information — this could limit ABLE's discriminative power on code or multilingual inputs where subword boundaries carry meaning."],"forward_implications":["Practitioners can pre-compute ABLE embeddings once per model and reuse them for routing, provenance auditing, and candidate shortlisting without retraining when new models are added to a pool.","ABLE's high-precision, moderate-recall profile on relation prediction makes it suitable as a first-pass screening filter for suspected model derivation, reducing manual investigation load.","The attribution-based paradigm could extend to detecting backdoor or adversarial fine-tuning: if a backdoor changes input-sensitivity patterns without changing outputs, ABLE should surface it.","Benchmark score prediction with Spearman ρ up to 0.89 suggests ABLE could reduce the cost of model evaluation by predicting rankings before running full benchmarks."],"fun_headline_variants":["Tokenizer-agnostic attributions map 239 LLMs","Gradient attributions fingerprint 239 LLMs without retraining","Training-free embeddings map 239 LLMs via input gradients","Input-sensitivity maps 239 LLMs across architectures","Probe-set gradients map 239 LLMs without retraining"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The stability proof assumes the model's gradient field is Lipschitz continuous in its parameters, which holds for Transformers with bounded weights and smooth activations under a fixed architecture. The paper does not prove this condition across different architectures, yet the main empirical contribution relies on cross-architecture comparison.","fun_headline_variants_meta":{"raw":{"variants":["Tokenizer-agnostic attributions map 239 LLMs","Gradient attributions fingerprint 239 LLMs without retraining","Training-free embeddings map 239 LLMs via input gradients","Input-sensitivity maps 239 LLMs across architectures","Probe-set gradients map 239 LLMs without retraining","Gradient attributions map 239 LLMs for routing and auditing","Tokenizer-agnostic gradients map 239 heterogeneous LLMs","Mapping 239 LLMs via tokenizer-agnostic input gradients"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1106,"prompt_tokens":545,"completion_tokens":561,"prompt_tokens_details":null},"tokens_in":545,"tokens_out":561,"duration_ms":15392,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T15:50:07.612821+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If two models with genuinely different internal computation produced identical attribution patterns on the probe set, ABLE would conflate them — the representation's discriminative power depends on the probe set exercising distinct input-sensitivity profiles across models.","supporting_citations":[],"review_version":1}