{"id":"fa394875-5746-48de-864a-8607506edbef","arxiv_id":"2509.04442","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive composition.","lead":"This paper introduces Delta Activations, a way to represent each finetuned large language model as a vector by measuring how its internal signals shift from the base model on a few generic prompts. The resulting vectors cluster models by domain and task, which could help people search, select, and merge models in the growing public model ecosystem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline clustering result may reflect training-data identity rather than domain specialization: Section 3.1 uses one dataset per domain, so 'domain clustering' is not separated from dataset fingerprinting.","rationale":"Reading the paper in good faith, Delta Activations is a simple, cheap, reproducible model embedding: one forward pass per model on five shared prompts, with code released, and it visibly separates finetuned models in several controlled pools. The method may well work for model-hub discovery. My concern is not internal inconsistency but an identifiability gap in the main evidence. In Section 3.1, each 'domain' is a single dataset, and the three replicates per domain are disjoint splits of that dataset; the reported silhouette therefore measures same-dataset closeness. The paper's own auxiliary experiments reinforce this reading: Table 8 clusters Tulu-v2 subsets by instruction format, and Table 14 clusters medical sub-expertise by dataset, which are dataset-level separations. Nothing in the paper shows that a model finetuned on MedQA lies near a model finetuned on PubMedQA, both far from a model finetuned on Codeforces. Unless that cross-dataset-within-domain separation is shown, the central claim that the embedding reflects finetuning domain (or task) rather than dataset identity is not established. Additivity and few-shot task embedding inherit the same confound because they use the same five datasets. The proposed test—multiple datasets per domain, domain-label silhouette vs dataset-label silhouette—would settle the issue directly. I would keep the reader's CONDITIONAL verdict but make cross-dataset evaluation a required condition for the domain-level interpretation.","tokens_in":16120,"tokens_out":4346,"duration_ms":42635,"concrete_test":"Build a new pool with at least three independent datasets per domain (e.g., medical: PubMedQA, MedQA, MedMCQA; coding: Codeforces, Evol-Instruct, HumanEval; math: GSM8K, MATH, DeepMind-Math). Finetune each dataset with the same LoRA recipe and compute Delta Activations with the same 5-prompt probe. Measure silhouette against domain labels (all datasets within a domain share one label) and against dataset labels. If domain-label silhouette drops to near zero or clearly below dataset-label silhouette, the embedding is a dataset fingerprint, not a domain representation. Repeat with three seeds per model and report error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 builds each domain cluster from three disjoint 3000-example splits of a single dataset (LegalBench, GSM8K, PubMedQA, HellaSwag, OPC-SFT), trained with identical LoRA settings. The reported silhouette scores (~0.61) therefore certify that models finetuned on different splits of the same dataset are mutually close in Delta-Activation space, not that models finetuned on different datasets within the same domain cluster together. The method could in principle be encoding dataset-specific distributional cues (answer formats, vocabulary, prompt templates used during training) rather than domain specialization. The Tulu-v2 experiment (Table 8) and intra-domain sub-expertise experiment (Table 14) both group models by dataset/subset identity, so they do not break the confound. The few-shot task embeddings and additivity experiments reuse the same five datasets, and the BBH model-selection experiment matches tasks to LoRAs trained on the same benchmark suite; all inherit the confound. Because the paper's central claim is an effective 'domain and task' embedding, the missing cross-dataset-within-domain test is the load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Delta Activations, defined in Section 2.4 as the average over a fixed probe set of the difference between a finetuned model's last-token, last-layer hidden state and that of its base model. The authors evaluate this embedding by constructing pools of LoRA-finetuned LLaMA-3.1-8B, Gemma-2-9B, and Qwen-2.5-7B models across five domains, reporting silhouette scores around 0.61 and favorable comparisons to flattened weights, salient masks, and output sentence embeddings. They also report robustness to training hyperparameters, an additive property for mixed datasets, few-shot task embeddings, cross-base transfer with Delta Meaning, and a proof-of-concept BBH model-selection improvement.","tokens_in":16450,"tokens_out":4744,"duration_ms":45453,"significance":"The proposed representation is simple, inexpensive (one forward pass per model), and does not require metadata or joint fitting, which makes it genuinely attractive for model-hub applications. The paper contains useful ablations of probe-prompt number, length, and content, token and layer position, and checks across three backbones and several training settings, and the code release supports reproducibility. If the domain-and-task clustering claim survives the dataset-identity confound identified below, this would be a practical contribution to model discovery and reuse. At present, however, the headline claim is not yet established at the level claimed.","major_comments":[{"comment":"The 'domain' clusters are built from three disjoint splits of a single dataset per domain (LegalBench, GSM8K, PubMedQA, HellaSwag, OPC-SFT). Consequently, the silhouette score of ~0.61 demonstrates only that models finetuned on different splits of the same dataset are close in Delta-Activation space; it does not demonstrate that models finetuned on different datasets within the same domain cluster together. Dataset-specific cues such as answer formats, vocabulary, and the training prompt templates in Appendix A.2 could drive the separation. This confound propagates to the additive property (Tables 4 and 13), the few-shot task embeddings (Section 3.3), and the BBH experiment, all of which use the same five datasets. The intra-domain experiment in Table 14 does not resolve the issue because it clusters by dataset/sub-expertise rather than showing invariance across datasets within a domain. Please add a cross-dataset-within-domain experiment (e.g., finetune on a second legal reasoning dataset and a second math dataset and test whether the two legal models land in one cluster) or soften the 'domain' claim to 'dataset identity'.","section":"Section 3.1, Table 2"},{"comment":"The probe prompts and extraction location are selected via ablations on the same pools used to report the main result. The reported silhouette score is therefore conditional on those choices, and all scores are single numbers with no variance estimate or significance test. Since the pools contain only 15 models each (five clusters of three), the difference between 0.61 and, for example, the 0.51 of Delta Logits may not be meaningful. Please report bootstrap confidence intervals or cluster-level score distributions. In the robustness table, the Qwen row with different learning rates reaches only 0.23, which weakens the statement that varied training settings 'generally did not break domain-specific clustering.'","section":"Section 3.2, Tables 5-7"},{"comment":"The additive property is central to the paper's claims, but the similarity metric used for 'Mixed vs. D1', 'Mixed vs. D2', and 'Mixed vs. Sum' is never defined. Please specify whether cosine similarity is used, report per-pair values with a measure of dispersion, and compare the additive match to a baseline such as similarity to a random vector or to a model finetuned on an unrelated dataset. This would make clear whether the additive effect is specific to the two constituent datasets rather than a generic property of any combination.","section":"Section 3.2, Table 4 and Appendix B.2"}],"minor_comments":[{"comment":"The dimension for flattened adapter weights is reported as ~2e7 while the salient mask is ~8e9; please clarify whether these are computed over different parameter sets and explain the discrepancy.","section":"Table 2"},{"comment":"The caption states that the 2/3-depth layer performs best, yet the default remains the final layer. Since the difference is small (0.64 vs. 0.61), please provide a justification or a significance analysis for this choice.","section":"Section 3.2, Table 6"},{"comment":"Please proofread for typographical errors: for example, 'LL AMA' in Section 2.3 and 'layerees' in the Table 6 caption. The captions of Figures 4 and 5 are missing closing periods.","section":"Throughout"},{"comment":"The motivating observation is supported only by a few selected output examples. Since the method itself uses activations rather than outputs, this is not a fatal issue, but it would be helpful to state explicitly that this is an exploratory observation rather than evidence for the effectiveness of the final method.","section":"Section 2.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The dataset-identity confound is the main obstacle. The current intra-domain experiment (Table 14) actually clusters by dataset identity and therefore cannot serve as evidence for cross-dataset domain invariance. If the authors add a true cross-dataset-within-domain experiment and report uncertainty intervals for the silhouette scores, I would be willing to reconsider. The paper is within scope and the method is promising; the gap is in the experimental design rather than in the internal logic of the derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a real methodology gap. The method—averaging activation differences over a handful of generic prompts—is simple, cheap, and works. But the headline claim is that Delta Activations cluster by domain and task, and the main experiment never actually tests that. Section 3.1 builds each 'domain' from three disjoint splits of a single dataset. So the silhouette scores (~0.61) certify that models finetuned on different splits of the same dataset are close together. That is dataset fingerprinting, not domain specialization. The Tulu-v2 experiment (Table 8) and the intra-domain experiment (Table 14) also group by dataset or subset identity, so they don't break the confound. The additivity and few-shot experiments inherit the same five datasets. This is a load-bearing gap, not a minor quibble.\n\nWhat is genuinely good: the representation itself is new in this specific form, and the paper does the work to compare against parameter-space and output-space baselines, ablates prompt count/length/content and extraction layer/token, checks robustness to training settings, full finetuning, and DPO, and ships a cross-architecture variant via Delta Meaning. The code is available. That is more than many papers do.\n\nSoft spots beyond the confound: no error bars on any silhouette score, so we don't know how stable the numbers are across seeds; the probe set is tiny (N=5) and was chosen through ablations on the same pools, which is fine but not a universal lens proof; and the model-selection experiment is a proof-of-concept with a 2% gain on BBH. All are minor compared to the confound.\n\nI think the right fix is straightforward: rerun the core clustering experiment with multiple datasets per domain and show that same-domain, different-dataset models cluster together, or soften the claim to 'dataset and training-similarity embedding.' The method would still be useful for model-hub discovery even with that narrowed claim.\n\nWho is this for? People building model-hub tooling, LoRA retrieval, or model merging systems. It deserves a serious referee—send it out. My recommendation: condition acceptance on a cross-dataset-within-domain experiment and repeated-seed error bars.","headline":"A useful, cheap model-embedding method that is overclaimed as 'domain clustering' when the experiments only show dataset clustering; worth a serious referee, but the central claim needs a cross-dataset-within-domain test.","tokens_in":16853,"tokens_out":1702,"would_cite":false,"duration_ms":17635,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuned LLMs become clusterable vectors when their internal activation shifts from the base model are averaged over generic prompts.","keywords":["Delta Activations","model embedding","finetuned LLMs","activation shifts","clustering","model selection","model merging","task embedding"],"falsifier":"Take one of the paper's model pools, compute $\\Delta$ Activations with the published five-prompt probe set, then recompute with a second set of five paraphrased generic prompts of the same length; if the two embeddings disagree about domain membership for more than one model per backbone, or if average silhouette across both probe sets falls below, say, 0.3, the claim that the probe set is a universal lens fails.","tokens_in":15921,"feed_emoji":"🧭","tokens_out":7763,"duration_ms":48989,"temperature":0.7,"pith_summary":"This paper proposes treating a finetuned large language model as a vector: average, over a few generic instruction prompts, the difference between the finetuned model's internal activation and the base model's activation on the same prompt. The authors claim that this vector, called $\\Delta$ Activations, is an effective model embedding: across LLaMA-3.1-8B, Gemma-2-9B, and Qwen-2.5-7B pools it separates models into clean domain clusters, with average silhouette scores around 0.61. They also claim that the embedding is robust to learning rate, epoch count, and dataset size, extends to DPO-trained models, and is additive when finetuning datasets are mixed. If these claims hold, public model hubs could be organized, searched, and compared without trusting metadata or accessing training data, and the same vector could guide model selection and merging.","feed_headline":"Activation shifts turn finetuned LLMs into clusterable vectors","feed_subtitle":"One generic five-prompt probe maps each model to a domain vector that also adds when datasets are mixed.","key_machinery":"The central object is the $\\Delta$ Activations vector, $\\Delta_f(x) = h_f(x) - h_{\\mathrm{base}}(x)$, averaged over a fixed probe dataset $D_{\\mathrm{probe}}$ of five generic instruction templates. The probe prompts are deliberately task-free, so any consistent activation shift they evoke is attributed to the finetuning itself rather than to a particular task, and averaging over paraphrased templates dampens prompt-specific noise. The default extraction point is the last token of the final layer, though the paper finds 2/3 depth and weighted token averaging slightly better, and the same differencing operation applied to logits or inverse-perplexity meaning vectors defines the $\\Delta$-X family. The model-agnostic $\\Delta$ Meaning variant is what allows clustering across different base architectures.","core_discovery":"The central claim is that the mean difference in last-token, final-layer hidden states between a finetuned model and its base model, computed on a small fixed set of generic prompt templates, is a compact behavioral fingerprint of the finetuned model. Concretely, for $v_f = \\frac{1}{N}\\sum_{i=1}^N \\left(h_f(x_i) - h_{\\mathrm{base}}(x_i)\\right)$, the paper argues that $v_f$ clusters models by finetuning domain across three open backbones, outperforming flattened LoRA weights, salient masks, and output sentence embeddings. The same vector is shown to stay informative under varied training settings and under preference optimization, and to satisfy an approximate additive property: the delta vector of a model finetuned on $D_1 \\cup D_2$ is closer to the sum of the separately trained deltas than to either component alone. A five-prompt probe set of paraphrased Alpaca-style templates with no task content is sufficient, and replacing activations with a model-agnostic meaning vector ($\\Delta$ Meaning) extends the representation to models finetuned from different base architectures.","pith_inferences":["A testable extension is to mine probe prompts per base model family and check whether cross-backbone domain clusters survive; if the generic lens is itself architecture-sensitive, the method's universality claim needs qualification.","The additive property is shown with equal-proportion data mixing, so an obvious next test is whether mixture weights map to weighted vector sums; if so, delta arithmetic could predict the embedding of arbitrary data blends before training.","The model-selection experiment found that nearest-neighbor selection hurt merging, so the practical route may be to use delta vectors to choose diverse-but-relevant subsets rather than the single most similar model.","Because delta vectors separate models trained on the same data under different settings, they may serve as a reproducibility diagnostic that flags finetuning runs whose internal behavior drifted despite identical data."],"forward_implications":["A hub that stores delta vectors can embed a newly uploaded finetuned model in one forward pass and place it in the same space as all existing models, with no retraining and no metadata.","Domain clusters that survive learning-rate, epoch, and data-size variation make nearest-neighbor search in delta space a viable substitute for parsing model names or trusting repository labels.","Because finetuning on 20 examples yields a task vector that lands near the cluster of fully finetuned same-domain models, task similarity can be estimated before committing to a large training run.","The additive property implies that the delta vector of a mixed-dataset model can be approximated as the sum of component deltas, which links model composition in data space to vector arithmetic in embedding space.","Cross-architecture clustering via Delta Meaning means model pools do not have to be restricted to a single backbone, broadening the scope of model discovery."],"supporting_citations":[{"why":"Supplies the Alpaca instruction template that seeds the generic probe dataset.","marker":"[61]"},{"why":"Cited as the explanation for why generic prompts can elicit a finetuned model's specialization.","marker":"[53]"},{"why":"Defines the silhouette score used to quantify clustering quality.","marker":"[54]"},{"why":"Describes LoRA, the finetuning method used to construct all model pools.","marker":"[20]"},{"why":"Defines the salient-mask baseline that Delta Activations is compared against.","marker":"[17]"},{"why":"Provides the sentence embedder used for the output-embedding baseline.","marker":"[66]"},{"why":"Defines meaning representations, which the Delta Meaning variant adapts for cross-architecture clustering.","marker":"[37]"},{"why":"Introduces the LoraHub model hub and merging setting used for the model-selection experiment.","marker":"[21]"},{"why":"Supplies the Big-Bench Hard tasks used to evaluate model selection.","marker":"[59]"},{"why":"Provides the model-interference explanation used to interpret why nearest-neighbor selection underperforms in merging.","marker":"[47]"}],"fun_headline_variants":["Activation deltas fingerprint finetuned LLMs for clustering","Finetuned LLMs become vectors via activation shifts","Delta activations: a compact fingerprint for finetuned models","Probe activation shifts to map and merge finetuned LLMs","Vectorize finetuned LLMs with a five-prompt activation probe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that five short, task-free prompts reliably switch on a finetuned model's specialization in its internal activations no matter the domain, task, or backbone; if some specializations stay silent under generic prompts, the resulting clusters and additive sums will not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Activation deltas fingerprint finetuned LLMs for clustering","Finetuned LLMs become vectors via activation shifts","Delta activations: a compact fingerprint for finetuned models","Probe activation shifts to map and merge finetuned LLMs","Vectorize finetuned LLMs with a five-prompt activation probe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2843,"prompt_tokens":945,"completion_tokens":1898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1811}},"tokens_in":561,"tokens_out":1898,"duration_ms":12575,"temperature":1.0,"reasoning_tokens":1811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:28:58.059473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the paper's model pools, compute $\\Delta$ Activations with the published five-prompt probe set, then recompute with a second set of five paraphrased generic prompts of the same length; if the two embeddings disagree about domain membership for more than one model per backbone, or if average silhouette across both probe sets falls below, say, 0.3, the claim that the probe set is a universal lens fails.","supporting_citations":[{"cited_title":"Alpaca: A strong, replicable instruction-following model","cited_arxiv_id":null,"evidence_quote":"Supplies the Alpaca instruction template that seeds the generic probe dataset."},{"cited_title":"Learning dynamics of llm finetuning","cited_arxiv_id":null,"evidence_quote":"Cited as the explanation for why generic prompts can elicit a finetuned model's specialization."},{"cited_title":"Rousseeuw","cited_arxiv_id":null,"evidence_quote":"Defines the silhouette score used to quantify clustering quality."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Describes LoRA, the finetuning method used to construct all model pools."},{"cited_title":"Localize-and-stitch: Efficient model merging via sparse task arithmetic","cited_arxiv_id":null,"evidence_quote":"Defines the salient-mask baseline that Delta Activations is compared against."},{"cited_title":"MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers","cited_arxiv_id":null,"evidence_quote":"Provides the sentence embedder used for the output-embedding baseline."},{"cited_title":"Meaning representations from trajectories in autoregressive models","cited_arxiv_id":null,"evidence_quote":"Defines meaning representations, which the Delta Meaning variant adapts for cross-architecture clustering."},{"cited_title":"Lorahub: Efficient cross-task generalization via dynamic lora composition","cited_arxiv_id":null,"evidence_quote":"Introduces the LoraHub model hub and merging setting used for the model-selection experiment."},{"cited_title":"Task arithmetic in the tangent space: Improved editing of pre-trained models","cited_arxiv_id":null,"evidence_quote":"Provides the model-interference explanation used to interpret why nearest-neighbor selection underperforms in merging."}],"review_version":2}