{"id":"567ce1e7-94c0-46f5-b8a3-cb8061e25ac2","arxiv_id":"2502.09619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ProbeLog represents each classifier output by its responses to fixed probe images and uses CLIP to answer text queries, achieving 43.8% top-1 accuracy when searching 1,500 ImageNet-trained models for a concept.","lead":"ProbeLog finds machine-learning models that can recognize concepts like 'dog' by looking only at each model's outputs on a fixed set of images, with no training data or documentation. It also lets users search by text, which could make large model repositories far more usable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot text retrieval depends entirely on the unproven claim that z-score normalization puts CLIP text descriptors and raw logit descriptors in a common metric space; Table 3 shows normalization is necessary but no evidence shows it is sufficient.","rationale":"The paper's central contribution is zero-shot weight-based model search. Logit-based search is well-supported by Fig. 4 and Table 1, and collaborative probing has a clear efficiency story. The weakest point is the bridge from text to logits. After reading the full text, I could not find any argument beyond normalization that the dot-product similarities from CLIP live in the same metric space as raw pre-softmax logits. Table 3's ablation proves normalization is necessary but is silent on sufficiency; no calibration data are provided. I agree with the reader's weakest_assumption. I also considered the unsubstantiated 'scalable to full-size repositories' claim and the unstated k hyperparameter, but those are secondary: the top-k parameter can be reported, and scalability is a forward-looking statement; the text-to-logit alignment is the one step without which the zero-shot headline number cannot be interpreted. A conditional verdict is appropriate: release of code/data plus the proposed alignment check would resolve the concern. If the check fails, the text-search results should be withdrawn or heavily qualified.","tokens_in":15621,"tokens_out":8386,"duration_ms":100031,"concrete_test":"Recompute the text-to-INet-Hub evaluation with an explicit alignment check: for each of the 1,000 query concepts, compute the z-scored text descriptor and the mean z-scored descriptor of all matching gallery logits, and measure (i) Jaccard overlap of their top-k probe index sets and (ii) the distribution of per-probe L2 distances on those indices, against the same quantities for non-matching logits. If matching concepts do not show substantially higher top-k overlap and lower distances than non-matching concepts, then Eq. 3 is not selecting shared semantic probes and the reported 43.8% is not evidence of a common metric space.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ProbeLog's headline result, 43.8% top-1 for text-to-INet-Hub, is produced by applying the logit-to-logit discrepancy measure (Eq. 3) to a CLIP text descriptor (Eq. 4) after per-vector z-scoring (Eq. 5). This only makes sense if the normalized text similarities and normalized logit responses have matching units and comparable shapes on the probes that Eq. 3 selects. The paper's only support is the ablation in Table 3: 'Top-k + No Norm.' drops text-to-INet to 0%, showing normalization is necessary, not that it is sufficient. There is no calibration plot, no distributional comparison, and no evidence that the top-k probes of the text descriptor are the same probes on which a matching logit is confident. If the alignment is imperfect, the 43.8% could be an artifact of the particular CLIP checkpoint and probe set, rather than a general property of classifier logits and text embeddings. This is the load-bearing assumption for the zero-shot contribution; if it fails, the text-search claim collapses even though logit-search may still stand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ProbeLog, a method for retrieving classification models by semantic concept using only model weights. For each logit, it records responses to a fixed ordered set of probe images, normalizes the resulting vector (Eq. 5), and compares descriptors with an asymmetric discrepancy that focuses on the query's highest-response probes (Eq. 3). A zero-shot variant computes a probe-level text descriptor from CLIP similarities between probe images and the query concept (Eq. 4), normalizes it like logit descriptors, and applies the same discrepancy measure. Collaborative probing (Eq. 6) uses truncated-SVD imputation to reduce the number of forward passes needed to encode a repository. The method is evaluated on two new hubs, INet-Hub (1,500 models trained on ImageNet subsets) and HF-Hub (71 real Hugging Face models), in both logit-to-logit and text-to-logit retrieval settings, with reported top-1 accuracies up to 72.8% and 43.8%, respectively.","tokens_in":15923,"tokens_out":5433,"duration_ms":58909,"significance":"The problem is timely and important: model hubs are growing quickly, and searching by weights rather than documentation could be practically valuable. The logit-level descriptor is a sensible functional representation that addresses class-order permutation, and collaborative probing is an interesting efficiency idea. If the zero-shot text-to-logit alignment is robust, the paper would make a useful contribution to model retrieval. However, the current evidence for the central text-search claim is incomplete, and several implementation details needed for reproduction are missing. The strengths are the clean problem formulation and the new evaluation hubs, but the quantitative claims need additional support before the paper can be accepted.","major_comments":[{"comment":"This is the load-bearing assumption for the zero-shot contribution; if the revision does not address it, the text-search claim is not supported.","section":"Sec. 4.3, Eq. (5), Table 3"},{"comment":"I also ask for a sensitivity curve for k, since the ablation in Table 3 only compares top-k against other selection rules, not against different k values.","section":"Sec. 4.2, Eq. (3); Sec. 4.4, Eq. (6)"},{"comment":"The authors should also state how many random seeds or probe-set draws the reported standard deviations correspond to.","section":"App. A; App. B; Sec. 5.2"},{"comment":"The model-level and full-query baselines are useful, but they do not address the paper's stated alternative of metadata-based search.","section":"Sec. 5.2, Tables 1 and 4"},{"comment":"Please also clarify whether the 'same accuracy' comparison in Fig. 7 uses the same k and normalization as the full-probe experiments.","section":"Sec. 4.4, Fig. 7"}],"minor_comments":[{"comment":"The sentence defining the mask matrix says 'all ones expect for zeros'; it should read 'all ones except for zeros'.","section":"Sec. 4.4, Eq. (6)"},{"comment":"It should be stated explicitly that Eq. (5) is applied to both the logit descriptor and the zero-shot text descriptor; the current notation only shows the logit descriptor, which is a source of confusion.","section":"Sec. 4.3"},{"comment":"There are typographical issues: 'over40%' should be 'over 40%' and 'IN ettask' should be 'INet task'.","section":"Sec. 5.2"},{"comment":"The conclusion that 'even out-of-distribution probes sampled from COCO retrieve relevant logits with high accuracy' would be easier to evaluate if the table also reported the mean number of relevant logits per query for each task, so the reader can interpret top-k accuracy against chance.","section":"Sec. 5.4, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core and the proposed method is likely publishable after a substantive revision. The main risk is that the zero-shot text-search result is tuned to the particular benchmarks: the paper reports no validation split for selecting the normalization and k, and the cross-modal alignment is supported only by an ablation showing that normalization is necessary. The small HF-Hub size and the short training of INet-Hub models also limit the strength of the generalization claims. I recommend major revision rather than rejection, provided the authors supply the requested alignment evidence, hyperparameters, and additional baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nYou should know about this paper: it proposes ProbeLog, a simple and effective way to retrieve classifiers by what they recognize, using only their weights. The idea is to represent each output logit by its responses to a fixed set of probe images, then match those descriptors between models and even against text via CLIP. The core contribution is real, and the paper validates the main design choices with careful ablations.\n\nWhat is actually new: per-logit probing for model search, the top-k confident-probe discrepancy metric, the z-score normalization that makes text and logit descriptors comparable, and the collaborative filtering trick that reduces probing cost by 3x. The paper is clear about what it does, and the limitations section is honest. The synthetic INet-Hub (1,500 models) and the real HF-Hub (71 models) give a reasonable first look at the problem.\n\nNow the soft spots. The zero-shot text search is the headline result (43.8% top-1 on INet-Hub), and it rests on the assumption that after z-scoring, CLIP text-descriptor values and raw logit responses live in a common metric space. The paper only shows that normalization is necessary (Table 3), not that it is sufficient. That is a real gap: no calibration plot, no distributional comparison, and no analysis of whether the top-k probes selected by the text descriptor align with the probes where the matching logit is confident. It does not sink the paper, because the results across several probe distributions and two model hubs suggest the alignment works empirically, but a referee should ask for direct evidence.\n\nOther gaps: the value of k and the SVD rank are not reported; there is no metadata-based baseline (a text-search baseline that uses class names would be a useful point of reference); the INet-Hub models are trained for only 2-5 epochs, so the main benchmark may be easier than real model hubs; and the abstract's \"scalable to full-size repositories\" claim is not backed by an experiment—the discussion acknowledges this. These are all addressable.\n\nWho is this for? Anyone working on model search, weight-space learning, or model hubs. It would be a loss to desk-reject this; it deserves a serious referee. With code and data release, plus the requested analyses, this could be a solid contribution.\n\nMy recommendation: send it to peer review, and ask the authors for the missing hyperparameters, a metadata baseline, and direct evidence about the normalization assumption.\n\nBest,\n[Name]","headline":"ProbeLog is a fresh, useful take on model search with some evaluation gaps; the zero-shot text alignment needs harder evidence, but the paper deserves a real review.","tokens_in":16387,"tokens_out":4730,"would_cite":true,"duration_ms":45984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68P20","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces ProbeLog, which represents each output dimension of a classifier by its responses to fixed probe images, enabling both logit-based and zero-shot text-based model search using only weights.","keywords":["model search","model zoos","probing","logit descriptors","zero-shot retrieval","collaborative filtering","weight-space learning","CLIP"],"falsifier":"Run the exact text-retrieval pipeline on a held-out repository of classifiers trained on concept sets not overlapping the probe distribution (for instance medical image classes) while keeping COCO probes; if top-1 accuracy does not stay far above the random baseline after normalization, the claimed shared metric space for zero-shot search is refuted.","tokens_in":15450,"feed_emoji":"🐶","tokens_out":5851,"duration_ms":54642,"temperature":0.7,"pith_summary":"The paper introduces ProbeLog, a method for searching large repositories of classification models by what the models actually do, using only their weights. The claim is that every output dimension (logit) of a classifier can be described by its pattern of responses to a fixed set of probe images, and that these logit descriptors can be matched both to other logits and to a text prompt such as \"Dog\" without any metadata or training data. A user could therefore find every publicly available model that recognizes a target concept even when the models are undocumented. On the paper's curated hubs, ProbeLog reaches 72.8% top-1 retrieval accuracy when searching in-distribution, and 43.8% top-1 accuracy when searching a 1,000-class hub from a text prompt, against a random baseline of 0.1%. The authors also show that probing cost can be cut by about three times using collaborative filtering to impute unprobed entries.","feed_headline":"ProbeLog finds models that recognize 'dog' from weights alone","feed_subtitle":"Zero-shot retrieval over 1,000 classes reaches 43.8% top-1 accuracy with no metadata or training data.","key_machinery":"The central objects are ProbeLog descriptors defined per logit (Eq. 2), mean-standard-deviation normalization (Eq. 5), a top-$k$ asymmetric discrepancy (Eq. 3), and zero-shot text descriptors built from CLIP dot products (Eq. 4). Together they turn model search into nearest-neighbor search over logit-level functional signatures, with no access to weights beyond forward passes. Collaborative probing (Eq. 6) completes missing probe entries by low-rank matrix factorization, making the gallery construction tractable.","core_discovery":"ProbeLog represents each logit $i$ of a model $f$ by the vector $\\varphi(f,i) = [f(x_1)[i], f(x_2)[i], \\ldots, f(x_n)[i]]$ of its outputs on $n$ fixed ordered probe images. The paper shows that after subtracting each descriptor's mean and dividing by its standard deviation, these vectors form a usable semantic index that is invariant to output-class order and to the other classes the model sees. Retrieval uses an asymmetric discrepancy: for a query descriptor, only the $k$ probe positions where the query logit responds most strongly are compared against gallery descriptors in $L_2$. For zero-shot text search, the probe images and the target text are embedded with a joint text-image model such as CLIP; the text descriptor is the vector of dot products between the text embedding and each probe embedding, and the same normalization makes it comparable to real logit descriptors. A final component completes partially probed galleries via truncated-SVD matrix factorization, so repositories can be encoded with a small fraction of probes per model.","pith_inferences":["Because the method indexes functions rather than architectures, it could plausibly extend to matching adapters, LoRA modules, or generative model concepts through activation or attention probes, though the paper only claims classification logits.","The dependence of text retrieval on z-score normalization suggests an empirical correspondence between CLIP's semantic geometry and logit confidence geometry; testing this correspondence across more text models could either broaden or constrain the zero-shot claim.","A testable design improvement the paper leaves implicit: choosing probes adaptively per concept (coreset selection) rather than uniformly at random could reduce the 4,000-probe requirement while keeping accuracy."],"forward_implications":["Model repositories can be searched by capability even when model cards are empty or templates.","Users can issue a text query for a concept never seen during indexing and retrieve models that recognize it.","The per-logit representation is naturally invariant to class ordering, so identical models with permuted outputs do not confuse retrieval.","Gallery construction cost scales down materially: roughly one third of the probes per model suffices when combined with matrix-factorization imputation.","The representation is lightweight relative to model weights, so a large repository's descriptors can be stored and searched cheaply."],"supporting_citations":[{"why":"Provides the CLIP text-image embeddings used to build zero-shot ProbeLog descriptors.","marker":"Radford et al., 2021"},{"why":"Supplies the COCO image pool used as the default probe set.","marker":"Lin et al., 2014"},{"why":"Contributes the truncated-SVD matrix-factorization procedure used for Collaborative Probing.","marker":"Koren et al., 2009"},{"why":"Defines the ImageNet class set and evaluation concepts in the INet-Hub.","marker":"Deng et al., 2009"},{"why":"Motivates functional representations of models, the view ProbeLog applies at logit level.","marker":"Herrmann et al., 2024"},{"why":"Provides the pretrained checkpoints used to initialize the trained models in INet-Hub.","marker":"Wightman, 2019"}],"fun_headline_variants":["ProbeLog: zero-shot model search from weights alone","Find models that know 'dog' without any metadata","ProbeLog retrieves models by output, not documentation","Zero-shot model retrieval: ProbeLog reads weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after subtracting the mean and dividing by the standard deviation, CLIP text-probe similarities and raw logit responses occupy the same metric space; the paper's own ablation shows that removing this normalization drops text-to-INet top-1 accuracy from 43.8% to 0%.","fun_headline_variants_meta":{"raw":{"variants":["ProbeLog: zero-shot model search from weights alone","Find models that know 'dog' without any metadata","ProbeLog retrieves models by output, not documentation","Zero-shot model retrieval: ProbeLog reads weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1331,"prompt_tokens":960,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":307}},"tokens_in":576,"tokens_out":371,"duration_ms":4354,"temperature":1.0,"reasoning_tokens":307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:47:55.040696+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact text-retrieval pipeline on a held-out repository of classifiers trained on concept sets not overlapping the probe distribution (for instance medical image classes) while keeping COCO probes; if top-1 accuracy does not stay far above the random baseline after normalization, the claimed shared metric space for zero-shot search is refuted.","supporting_citations":[{"cited_title":"Learning Useful Representations of Recurrent Neural Network Weight Matrices","cited_arxiv_id":"2403.11998","evidence_quote":"Motivates functional representations of models, the view ProbeLog applies at logit level."}],"review_version":1}