ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding

Bowen Tian; Shaofeng Liang; Wenshuo Chen; Yanlin Zhang; Yusen Hou; Yutao Yue; Zirui Wang

arxiv: 2606.07524 · v1 · pith:FOOXXCW3new · submitted 2026-04-20 · 💻 cs.CL · cs.AI

ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding

Zirui Wang , Yusen Hou , Shaofeng Liang , Bowen Tian , Yanlin Zhang , Wenshuo Chen , Yutao Yue This is my paper

Pith reviewed 2026-07-05 15:50 UTC · model glm-5.2

classification 💻 cs.CL cs.AI

keywords feature attributionmodel embeddingLLM representationgradient-based interpretabilitymodel routingprovenance auditingtokenizer alignmentJohnson-Lindenstrauss projection

0 comments

The pith

Gradient fingerprints map 239 LLMs without retraining

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ABLE represents each language model by how it responds to input perturbations, not by its outputs or weights. It computes Gradient×Input attributions on a fixed probe set, aligns them across tokenizers at the word level, and projects to a compact embedding. This produces a reusable model fingerprint that distinguishes models with identical outputs but different internal computation. On 239 open-source LLMs, ABLE predicts model family relationships, routes queries, and estimates benchmark scores — all without training. The paper proves that, under bounded weights and smooth activations, the parameter-to-embedding map is Lipschitz continuous with finite-sample convergence.

Core claim

The central mechanism is that gradient-based feature attributions, aggregated across a fixed probe set and aligned at the word level, encode model-specific input-sensitivity patterns that are more discriminative than output behavior alone and remain comparable across heterogeneous architectures. The paper shows this representation is Lipschitz continuous in model parameters (Theorem A.1), preserves pairwise distances under random projection (Theorem A.2), and concentrates around its population value with finite samples (Theorem A.3). Empirically, on 239 LLMs, ABLE achieves 0.867 accuracy in relation prediction (vs. 0.721 for the best output-based baseline), 0.676 routing accuracy with frozen

What carries the argument

Gradient×Input (GI) attribution; tokenizer-agnostic word-level alignment via character-span redistribution; Johnson–Lindenstrauss random projection; Lipschitz continuity of the gradient field in θ under bounded Transformer weights.

If this is right

Practitioners can pre-compute ABLE embeddings once per model and reuse them for routing, provenance auditing, and candidate shortlisting without retraining when new models are added to a pool.
ABLE's high-precision, moderate-recall profile on relation prediction makes it suitable as a first-pass screening filter for suspected model derivation, reducing manual investigation load.
The attribution-based paradigm could extend to detecting backdoor or adversarial fine-tuning: if a backdoor changes input-sensitivity patterns without changing outputs, ABLE should surface it.
Benchmark score prediction with Spearman ρ up to 0.89 suggests ABLE could reduce the cost of model evaluation by predicting rankings before running full benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The theoretical stability guarantee covers only homogeneous models under a shared parameterization, while the empirical results span heterogeneous architectures. Whether the Lipschitz bound transfers across architecture boundaries remains an open question; the cross-architecture results may rest on the word-level alignment acting as an implicit normalization rather than on the proven continuity.
If higher-fidelity attribution methods (Integrated Gradients, SmoothGrad) were substituted for Gradient×Input, the embedding might capture finer-grained sensitivity patterns, but the single-pass efficiency that makes ABLE scalable would be lost.
The character-level alignment uniformly distributes token attributions across characters, which may lose subword-level semantic information — this could limit ABLE's discriminative power on code or multilingual inputs where subword boundaries carry meaning.

Load-bearing premise

The stability proof assumes the model's gradient field is Lipschitz continuous in its parameters, which holds for Transformers with bounded weights and smooth activations under a fixed architecture. The paper does not prove this condition across different architectures, yet the main empirical contribution relies on cross-architecture comparison.

What would settle it

If two models with genuinely different internal computation produced identical attribution patterns on the probe set, ABLE would conflate them — the representation's discriminative power depends on the probe set exercising distinct input-sensitivity profiles across models.

Figures

Figures reproduced from arXiv: 2606.07524 by Bowen Tian, Shaofeng Liang, Wenshuo Chen, Yanlin Zhang, Yusen Hou, Yutao Yue, Zirui Wang.

**Figure 2.** Figure 2: Overview of the ABLE pipeline. Given a model, we compute Gradient [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Documented relation verification experiments. Left: Mistral family similarity tree reconstructed from [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: Model Map visualization using t-SNE projection of ABLE embeddings. Models are colored by family, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Hierarchical similarity tree constructed from ABLE embeddings using hierarchical clustering with cosine [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Scatter plots showing predicted benchmark [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: Effect of ABLE embedding dimension on relation prediction performance. The blue curve corresponds [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 8.** Figure 8: Comparison of weight space and ABLE fea [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: Robustness test for ABLE embeddings. Each [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

read the original abstract

The explosive growth of large language models (LLMs) has created a heterogeneous and poorly documented ecosystem, making systematic model comparison increasingly important for provenance auditing, security analysis, and model selection. Existing representation methods struggle to address this setting efficiently. Approaches analyzing internal parameters are powerful when architectures are compatible, but face scalability barriers under structural heterogeneity, while methods relying on external outputs may conflate models with similar behaviors and are difficult to align in richer output spaces across different tokenizers. To bridge this gap, we propose ABLE (Attribution-Based Large-model Embedding), a framework that leverages the interpretability space to construct model representations. By aggregating gradient-based feature attributions via a tokenizer-agnostic word-level alignment, ABLE captures model-specific input-sensitivity patterns rather than only surface-level outputs. Beyond empirical utility, we provide a stability analysis showing that, under standard regularity assumptions for differentiable Transformer-style models, ABLE induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees. Extensive experiments on 239 open-source LLMs demonstrate that our training-free approach achieves competitive or superior performance in relation prediction, model routing, and benchmark score prediction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

New method for LLM comparison via aggregated gradient attributions; theory covers only same-architecture case while main experiments are cross-architecture

read the letter

Here's my read on the ABLE paper. The core idea is genuinely new: aggregate gradient-based feature attributions (Gradient×Input) across a fixed probing set, align them at the word level to handle tokenizer differences, and use JL projection to get a compact model-level embedding. Nobody has done this particular combination before, and the motivation is sound — two models can produce identical outputs while relying on very different input features, and capturing that difference gives you a richer signal than output-only methods. The empirical work is solid for what it is. They run on 239 models, which is a real collection. The relation prediction results (Table 1) show ABLE beating output-based baselines, with a clear precision advantage that makes sense given the method's design. The benchmark score prediction (Spearman ρ 0.81–0.89) and routing results (competitive with EmbedLLM without retraining) are practically useful. The interpolation experiment in Appendix C is a nice check — ABLE space preserves the topology of weight-space interpolation on same-architecture models. The Mantel test (Appendix F) showing 0.722 correlation across disjoint probe subsets addresses sample stability reasonably. Now the soft spots. The stress-test concern about the theory-experiment gap is real but I'd calibrate it as moderate, not fatal. The paper is transparent about this — Appendix A explicitly states the theory covers fixed-parameterization models and cross-architecture comparisons remain empirical. The abstract oversells slightly by calling it 'a principled stability guarantee for ABLE' without scoping it, but the body is honest. The theory gives you Lipschitz continuity within an architecture family, which is useful but not what the headline experiments need. The relation prediction dataset is small (135 pairs, 8:2 split) — that's a real limitation for the classification results, though the effect sizes are large enough that I'm not deeply worried. No code or data is shipped, which hurts reproducibility. The JL dimension K=256 is tuned on downstream task performance (Appendix B), which is a minor empirical fitting step but not circular. The character-level alignment is crude — uniform distribution of token attributions across characters — and the authors acknowledge this. It works for English MCQ probes but would likely struggle on code or multilingual text. Overall: this is a useful methods paper with a clean new idea and decent empirical validation. The theory is correctly stated but scoped more narrowly than the experiments. A reader working on model comparison, routing, or provenance auditing gets value from this. It deserves a serious referee who can push on the dataset size, the alignment scheme, and whether the theory can be extended to cover cross-architecture settings.

Referee Report

3 major / 7 minor

Summary. ABLE proposes a model representation framework for heterogeneous LLM ecosystems based on gradient-based feature attributions. The method computes Gradient×Input attributions on a fixed probing set, aligns token-level scores to a unified word vocabulary, and projects the result to a compact embedding via Johnson-Lindenstrauss random projection. The paper validates the approach on 239 open-source LLMs across three tasks: relation prediction, model routing, and benchmark score prediction. A theoretical appendix establishes Lipschitz continuity of the parameter-to-embedding map and finite-sample concentration bounds under a fixed parameterization.

Significance. The paper addresses a genuine gap: existing parameter-space methods require architectural compatibility, while output-based methods may conflate models with similar behavior. The attribution-based approach is a reasonable middle ground. Strengths include the training-free design, the scale of evaluation (239 models), and the inclusion of a self-contained theoretical analysis (Appendix A) with an empirical interpolation check (Appendix C). The tokenizer-agnostic word-level alignment is a practical contribution. The benchmark ranking comparison against output-based baselines (Table 5) provides useful evidence that attribution signals carry information beyond surface outputs.

major comments (3)

§Abstract and §1: The abstract states that ABLE 'induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees' without scoping this to a fixed parameterization. The main text (§1, contribution bullet 2) and Appendix A do scope the claim to 'differentiable Transformer-style models with a fixed parameterization,' and the paper explicitly acknowledges that 'the heterogeneous cross-architecture comparisons in the main paper remain empirical' (§A). However, the abstract's unqualified framing creates a mismatch between the stated guarantee and the primary empirical contributions (Tables 1–3), which involve heterogeneous architectures with different parameter dimensions. The Lipschitz constant L = B·M·L_φ from Theorem A.1 has no defined value when θ and θ' live in different-dimensional spaces. This is a presentation issue that affects how readers interpret
§4.3, Table 1: The relation prediction dataset contains only 135 model pairs, split 8:2 into train and test. This yields approximately 27 test pairs. With 5 random seeds, the standard deviations on Accuracy (±0.042) and AUC (±0.057) are substantial relative to the margins between ABLE (0.867) and the best baseline (0.721). The paper should report the exact number of test pairs and discuss whether the performance gap is statistically significant (e.g., via a paired test or bootstrap confidence interval). The current presentation does not allow the reader to assess whether the improvement over Log-Likelihood is robust or an artifact of small-sample variance.
§A.2, Proposition A.1, Step 3: The proof sketch for multi-layer composition states that 'perturbation analysis shows that the product ∇_x f remains Lipschitz in θ' but does not provide the actual perturbation bound. Steps 1 and 2 give explicit per-layer Lipschitz constants, but Step 3 only asserts that aggregating across L layers yields a global constant M without deriving how M depends on L, B_W, L_σ, etc. For an L-layer network, the product of Jacobians can lead to exponential dependence on L. The proposition should either state the explicit form of M (even if loose) or clarify that the bound is qualitative and the constant may be vacuously large for deep models.

minor comments (7)

§3.4: The JL lemma requires K = O(ε^{-2} log M), but the paper states K = O(ε^{-2} log M) in the main text while Theorem A.2 derives K ≥ (4/ε²) ln(2m²/δ). These should be consistent; the main text omits the δ dependence.
Table 1: The 'Greedy' baseline description ('predicts that all within-organization pairs are related and cross-organization pairs are unrelated') is ambiguous — does 'organization' refer to the model creator's organization? This should be clarified.
§4.4, Table 2: The ABLE router accuracy (0.676) is described as 'slightly higher than the fully trained EmbedLLM router' (0.665). The difference is 0.011 with standard deviations of ±0.001 and ±0.003. The claim of superiority should be softened or supported by a significance test.
Figure 4: The t-SNE visualization is informative but the figure is difficult to read at normal zoom. Consider increasing font size for model names or providing an interactive version.
§A.1: The notation s_θ^(y)(x) is used in the appendix while the main text uses S_c. Aligning notation would improve readability.
Appendix B, Figure 7: The axis label 'ABLE Dimension' uses a non-standard character (uni00A0, a non-breaking space). This should be cleaned up.
§4.5: The ridge regression for benchmark score prediction uses leave-one-out cross-validation. It would be useful to report the regularization hyperparameter selection procedure.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The referee identifies three major issues: (1) an over-qualified theoretical claim in the abstract, (2) insufficient statistical reporting for the relation prediction experiment given the small test set, and (3) a missing explicit perturbation bound in the multi-layer composition step of Proposition A.1. We agree with all three points and will revise the manuscript accordingly. Below we address each comment in detail.

read point-by-point responses

Referee: The abstract states that ABLE 'induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees' without scoping this to a fixed parameterization. The main text and Appendix A do scope the claim, but the abstract's unqualified framing creates a mismatch with the primary empirical contributions involving heterogeneous architectures. The Lipschitz constant L = B·M·L_φ has no defined value when θ and θ' live in different-dimensional spaces.

Authors: The referee is correct. The abstract should scope the theoretical claim to fixed-parameterization models, consistent with the main text (§1, contribution bullet 2) and Appendix A, which explicitly state that the guarantee applies to 'differentiable Transformer-style models with a fixed parameterization' and that 'the heterogeneous cross-architecture comparisons in the main paper remain empirical.' We will revise the abstract to include this scoping qualifier. Specifically, we will change the relevant sentence to read approximately: 'under standard regularity assumptions for differentiable Transformer-style models with a fixed parameterization, ABLE induces a Lipschitz-continuous parameter-to-embedding map with finite-sample convergence guarantees.' This ensures the abstract does not overstate the scope of the theoretical result relative to the heterogeneous empirical setting of Tables 1–3. revision: yes
Referee: The relation prediction dataset contains only 135 model pairs, split 8:2 into train and test, yielding approximately 27 test pairs. With 5 random seeds, the standard deviations on Accuracy (±0.042) and AUC (±0.057) are substantial relative to the margins between ABLE (0.867) and the best baseline (0.721). The paper should report the exact number of test pairs and discuss whether the performance gap is statistically significant.

Authors: The referee's concern is well-taken. The 8:2 split of 135 pairs yields exactly 27 test pairs (18 positive, 9 negative, given the 2:1 negative-to-positive sampling ratio). We agree that this is a small test set and that the current presentation does not allow readers to assess statistical significance. In the revision, we will: (1) report the exact number of test pairs (27) explicitly in §4.3; (2) add a paired bootstrap 95% confidence interval for the accuracy and AUC differences between ABLE and each baseline; and (3) include a brief discussion of the limitations of evaluation at this scale. We note that the small-sample limitation is partly inherent to the problem: documented derivative relationships among open-source LLMs are relatively scarce, and we chose to restrict the dataset to pairs with verifiable metadata rather than inflating it with uncertain labels. However, we agree that the paper must be transparent about this constraint and provide the statistical tests the referee requests. revision: yes
Referee: The proof sketch for multi-layer composition in Proposition A.1, Step 3, states that 'perturbation analysis shows that the product ∇_x f remains Lipschitz in θ' but does not provide the actual perturbation bound. Steps 1 and 2 give explicit per-layer Lipschitz constants, but Step 3 only asserts that aggregating across L layers yields a global constant M without deriving how M depends on L, B_W, L_σ, etc. For an L-layer network, the product of Jacobians can lead to exponential dependence on L. The proposition should either state the explicit form of M or clarify that the bound is qualitative and the constant may be vacuously large for deep models.

Authors: The referee is correct that Step 3 of Proposition A.1 is incomplete as written. The product of L layer Jacobians can indeed lead to exponential dependence on L, and the current proof sketch does not make this explicit. In the revision, we will expand Step 3 to derive the explicit form of M. Specifically, for an L-layer network where each layer Jacobian J_l has spectral norm bounded by ρ_l (with ρ_l depending on B_W, L_σ, and the softmax Jacobian bound), the product bound gives M = O(∏_{l=1}^{L} ρ_l · max_l C_l), where C_l is the per-layer Lipschitz constant from Steps 1–2. Under the bounded weight assumption ∥W^l∥ ≤ B_W and activation derivative bound L_σ, each ρ_l ≤ B_W · L_σ, yielding M = O((B_W · L_σ)^L · (L_σ + B_W · L_{σ'} · B_x)). We will state this explicitly and add a remark that this bound grows exponentially with L, making it primarily a qualitative existence result for deep models rather than a tight quantitative guarantee. This is consistent with the paper's framing of the theory as providing a principled stability guarantee rather than a practically tight bound. revision: yes

Circularity Check

0 steps flagged

No significant circularity: theory is self-contained; one minor empirical hyperparameter tuning does not undermine the central claim.

full rationale

The paper's theoretical derivation (Appendix A) is self-contained. Theorem A.1 establishes Lipschitz continuity of the ABLE map under stated regularity assumptions, Proposition A.1 argues these hold for Transformers with bounded weights, and Theorem A.2 applies the standard Johnson-Lindenstrauss lemma. None of these steps reduce to their inputs by construction, and no load-bearing self-citation chain is present. The only empirical fitting is the choice of projection dimension K=256, determined via downstream task performance (Appendix B). This is a standard hyperparameter selection step, not a circular derivation: the stability guarantee (Theorem A.1) and distance preservation (Theorem A.2) are independent of the specific value of K. The theory-experiment gap (theory covers fixed-parameterization, experiments cover heterogeneous architectures) is a correctness/scope concern, not a circularity issue. The paper explicitly acknowledges this scope limitation in Appendix A. No step in the derivation chain is equivalent to its inputs by definition or by self-citation. Score 2 reflects the minor empirical tuning of K, which does not constitute circular reasoning but is the only point where a fitted value touches the theoretical framework.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions). It uses existing techniques (Gradient×Input, JL projection) in a new combination. The free parameters are empirically tuned, not derived from first principles.

free parameters (3)

JL projection dimension K = 256
Determined via downstream task performance (Appendix B), not purely from theoretical JL bounds.
L2 normalization = true
Chosen based on ablation results showing it outperforms unnormalized variant (Appendix B).
Probing dataset size N = 1200
200 samples from each of 6 benchmarks, chosen empirically.

axioms (3)

domain assumption Gradient field of log-probability function is M-Lipschitz continuous in θ
Theorem A.1 relies on this assumption. Proposition A.1 argues it holds for Transformers with bounded weights and smooth activations, but this is limited to fixed parameterization.
domain assumption Differentiability of s_θ(x) with respect to x almost everywhere
Stated in Theorem A.1. Standard for Transformer models with continuous activations.
domain assumption Bounded input norms: E[||x||_2] ≤ B
Stated in Theorem A.1. Reasonable for token embeddings from a finite vocabulary.

pith-pipeline@v1.1.0-glm · 25976 in / 1932 out tokens · 184850 ms · 2026-07-05T15:50:07.612821+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 7 internal anchors

[1]

William B Johnson, Joram Lindenstrauss, and 1 others

Springer. William B Johnson, Joram Lindenstrauss, and 1 others

work page
[2]

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Extensions of lipschitz mappings into a hilbert space.Contemporary mathematics, 26(189-206):1. Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Boluk- basi. 2021. Guided integrated gradients: An adap- tive path method for removing noise. InProceed- ings of the IEEE/CVF conference on computer vi- sion and pattern ...

work page internal anchor Pith review Pith/arXiv arXiv 2021
[3]

Qwen2 Technical Report

PMLR. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158,...

work page internal anchor Pith review Pith/arXiv arXiv 2019
[4]

Protect Your Prompts: Protocols for IP Protection in LLM Applications

Protect your prompts: Protocols for ip protection in llm applications.arXiv preprint arXiv:2306.06297. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information process- ing systems, 30. Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen

work page internal anchor Pith review Pith/arXiv arXiv 2017
[5]

Gradient based Feature Attribution in Explainable AI: A Technical Review

Gradient based feature attribution in ex- plainable ai: A technical review.arXiv preprint arXiv:2403.10415. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Re- solving interference when merging models.Ad- vances in Neural Information Processing Systems, 36:7093–7115. An Yang, Anfeng Li, Baosong Yang, Beichen...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[6]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhou Yang, Jieke Shi, Prem Devanbu, and David Lo

work page internal anchor Pith review Pith/arXiv arXiv
[7]

Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri

Ecosystem of large language models for code.ACM Transactions on Software Engineering and Methodology. Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri. 2024. Phylolm: Inferring the phy- logeny of large language models and predicting their performances in benchmarks.arXiv preprint arXiv:2404.04671. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Far...

work page arXiv 2024
[8]

A Survey of Large Language Models

A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2). Zhixue Zhao and Boxuan Shan. 2024. Reagent: A model-agnostic feature attribution method for generative language models.arXiv preprint arXiv:2402.00794. Xinyu Zhou, Delong Chen, Samuel Cahyawijaya, Xufeng Duan, and Zhenguang G Cai. 2024. Linguis- tic minimal pairs elicit linguistic si...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[9]

EmbedLLM: Learning Compact Representations of Large Language Models

Embedllm: Learning compact representa- tions of large language models.arXiv preprint arXiv:2410.02223. A Theoretical Analysis In this section, we provide a theoretical justifica- tion that ABLE constitutes a valid and mechanism- aware embedding of large language models. Our analysis abstracts away implementation details and focuses on the relationship bet...

work page internal anchor Pith review Pith/arXiv arXiv
[10]

The gradient field of the log-probability func- tion,∇ xsθ(x), isM-Lipschitz continuous in θ:∥∇ xsθ(x)− ∇ xsθ′(x)∥2 ≤M∥θ−θ ′∥2

work page
[11]

Then the ABLE mappingΦ :θ7→Φ(θ)is Lips- chitz continuous: ∥Φ(θ)−Φ(θ ′)∥2 ≤L∥θ−θ ′∥2, whereL=BM L ϕ

For allxin the support ofD,s θ(x)is differ- entiable with respect toxalmost everywhere, ϕisL ϕ-Lipschitz, and inputs are bounded on average:E x∼D∥x∥2 ≤B. Then the ABLE mappingΦ :θ7→Φ(θ)is Lips- chitz continuous: ∥Φ(θ)−Φ(θ ′)∥2 ≤L∥θ−θ ′∥2, whereL=BM L ϕ. Proof.Using the definition ofa θ(x)and the as- sumption on∇ xsθ(x): we first expand ∥aθ(x)−aθ′(x)∥2 =∥x...

work page
[12]

Activation functions (e.g., GELU, Swish) have bounded first and second derivatives: |σ′(z)| ≤L σ and|σ ′′(z)| ≤L σ′ for allz

work page
[13]

Token embeddings are drawn from a finite vo- cabulary with bounded norms:∥x∥ 2 ≤B x

work page
[14]

Then the gradient field∇ xsθ(x)isM-Lipschitz continuous inθfor some constantM >0

The network has finite depthLand bounded weight matrices:∥W l∥2 ≤B W for all layers l. Then the gradient field∇ xsθ(x)isM-Lipschitz continuous inθfor some constantM >0. Proof.We proceed by analyzing the Lipschitz de- pendence layer by layer. Step 1: Linear layer.Fory=W x, the input gradient is∇ xy=W T . For two parameter config- urationsW, W ′: ∥∇xyW − ∇x...

work page
[15]

Conversely, dimensions beyond 256 offer only marginal gains while incur- ring higher computational and storage costs

Very low dimensions (e.g.,d= 4) yield poor performance because excessive compression discards structural information from the original high-dimensional space. Conversely, dimensions beyond 256 offer only marginal gains while incur- ring higher computational and storage costs. This trade-off between quality and efficiency motivates our choice ofd= 256for a...

work page

[1] [1]

William B Johnson, Joram Lindenstrauss, and 1 others

Springer. William B Johnson, Joram Lindenstrauss, and 1 others

work page

[2] [2]

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Extensions of lipschitz mappings into a hilbert space.Contemporary mathematics, 26(189-206):1. Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Boluk- basi. 2021. Guided integrated gradients: An adap- tive path method for removing noise. InProceed- ings of the IEEE/CVF conference on computer vi- sion and pattern ...

work page internal anchor Pith review Pith/arXiv arXiv 2021

[3] [3]

Qwen2 Technical Report

PMLR. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158,...

work page internal anchor Pith review Pith/arXiv arXiv 2019

[4] [4]

Protect Your Prompts: Protocols for IP Protection in LLM Applications

Protect your prompts: Protocols for ip protection in llm applications.arXiv preprint arXiv:2306.06297. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information process- ing systems, 30. Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen

work page internal anchor Pith review Pith/arXiv arXiv 2017

[5] [5]

Gradient based Feature Attribution in Explainable AI: A Technical Review

Gradient based feature attribution in ex- plainable ai: A technical review.arXiv preprint arXiv:2403.10415. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Re- solving interference when merging models.Ad- vances in Neural Information Processing Systems, 36:7093–7115. An Yang, Anfeng Li, Baosong Yang, Beichen...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[6] [6]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhou Yang, Jieke Shi, Prem Devanbu, and David Lo

work page internal anchor Pith review Pith/arXiv arXiv

[7] [7]

Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri

Ecosystem of large language models for code.ACM Transactions on Software Engineering and Methodology. Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri. 2024. Phylolm: Inferring the phy- logeny of large language models and predicting their performances in benchmarks.arXiv preprint arXiv:2404.04671. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Far...

work page arXiv 2024

[8] [8]

A Survey of Large Language Models

A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2). Zhixue Zhao and Boxuan Shan. 2024. Reagent: A model-agnostic feature attribution method for generative language models.arXiv preprint arXiv:2402.00794. Xinyu Zhou, Delong Chen, Samuel Cahyawijaya, Xufeng Duan, and Zhenguang G Cai. 2024. Linguis- tic minimal pairs elicit linguistic si...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[9] [9]

EmbedLLM: Learning Compact Representations of Large Language Models

Embedllm: Learning compact representa- tions of large language models.arXiv preprint arXiv:2410.02223. A Theoretical Analysis In this section, we provide a theoretical justifica- tion that ABLE constitutes a valid and mechanism- aware embedding of large language models. Our analysis abstracts away implementation details and focuses on the relationship bet...

work page internal anchor Pith review Pith/arXiv arXiv

[10] [10]

The gradient field of the log-probability func- tion,∇ xsθ(x), isM-Lipschitz continuous in θ:∥∇ xsθ(x)− ∇ xsθ′(x)∥2 ≤M∥θ−θ ′∥2

work page

[11] [11]

Then the ABLE mappingΦ :θ7→Φ(θ)is Lips- chitz continuous: ∥Φ(θ)−Φ(θ ′)∥2 ≤L∥θ−θ ′∥2, whereL=BM L ϕ

For allxin the support ofD,s θ(x)is differ- entiable with respect toxalmost everywhere, ϕisL ϕ-Lipschitz, and inputs are bounded on average:E x∼D∥x∥2 ≤B. Then the ABLE mappingΦ :θ7→Φ(θ)is Lips- chitz continuous: ∥Φ(θ)−Φ(θ ′)∥2 ≤L∥θ−θ ′∥2, whereL=BM L ϕ. Proof.Using the definition ofa θ(x)and the as- sumption on∇ xsθ(x): we first expand ∥aθ(x)−aθ′(x)∥2 =∥x...

work page

[12] [12]

Activation functions (e.g., GELU, Swish) have bounded first and second derivatives: |σ′(z)| ≤L σ and|σ ′′(z)| ≤L σ′ for allz

work page

[13] [13]

Token embeddings are drawn from a finite vo- cabulary with bounded norms:∥x∥ 2 ≤B x

work page

[14] [14]

Then the gradient field∇ xsθ(x)isM-Lipschitz continuous inθfor some constantM >0

The network has finite depthLand bounded weight matrices:∥W l∥2 ≤B W for all layers l. Then the gradient field∇ xsθ(x)isM-Lipschitz continuous inθfor some constantM >0. Proof.We proceed by analyzing the Lipschitz de- pendence layer by layer. Step 1: Linear layer.Fory=W x, the input gradient is∇ xy=W T . For two parameter config- urationsW, W ′: ∥∇xyW − ∇x...

work page

[15] [15]

Conversely, dimensions beyond 256 offer only marginal gains while incur- ring higher computational and storage costs

Very low dimensions (e.g.,d= 4) yield poor performance because excessive compression discards structural information from the original high-dimensional space. Conversely, dimensions beyond 256 offer only marginal gains while incur- ring higher computational and storage costs. This trade-off between quality and efficiency motivates our choice ofd= 256for a...

work page