{"id":"655d7a56-7c62-42fb-8aab-8e89157d15a4","arxiv_id":"2506.13496","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hierarchical multi-positive contrastive loss using Locarno taxonomy improves patent image retrieval at subclass and main class levels, with mixed results at patent level.","lead":"This paper proposes a contrastive training loss that treats patent images as multiple positive pairs with similarity weights based on the Locarno classification hierarchy, and tests it on the DeepPatent2 dataset. It reports improved retrieval at subclass and main class levels, especially for small models, but not consistently at the patent level.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central premise that LIC co-class images are useful positives is untested; a label-shuffle control is needed to show gains come from the taxonomy, especially since patent-ID mAP drops on five model variants.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing point: the LIC taxonomy's usefulness as a visual-similarity proxy enters Equation 2 through the hand-set scores (sp=1, ss=0.35, sm=0.2) and is never independently validated. The label-shuffle control I propose would settle this concern directly, because it isolates the semantic content of the taxonomy from the statistical effect of having multiple positive pairs per anchor. I also share the reader's reading of Tables 1 and 2: the claim that HMCL 'enhances retrieval results' is contradicted at the patent-ID level for five of the nine model configurations, so the abstract needs qualification even if the coarse-level gains are real. I do not think this warrants rejection: the method is simple, the reported coarse-level improvements are consistent across architectures, and a conditional acceptance with the label-shuffle check plus a revised claim about the patent-ID trade-off is proportionate. The concern is not about author conduct or external consensus; it is an internal, testable gap between the mechanism claimed and the evidence provided.","tokens_in":8424,"tokens_out":7455,"duration_ms":80382,"concrete_test":"Run a label-shuffle control: train HMCL on the same DeepPatent2 2007 split with LIC labels randomly permuted across training images, preserving class sizes, batch construction, seeds, and hyperparameters; evaluate on the true LIC labels. If shuffled-label HMCL retains most of the subclass/main-class mAP gains over CL, the improvements are driven by label co-occurrence rather than the LIC taxonomy, and the central claim fails. If the gains disappear, the taxonomy premise is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central premise, that LIC co-class images are semantically useful positives, is not tested independently of the loss objective. Equation 2 assigns nonzero h_ij to every image in the same subclass or main class regardless of visual content, so HMCL optimizes embeddings to pull together images that may depict different objects (e.g., seats vs. beds under 'furnishing'). The observed patent-ID degradation in Tables 1 and 2 (ViT-Small -0.002, ViT-Base -0.009, ViT-Large -0.003, CLIP-B/16 -0.017, CLIP-L/14 -0.002 mAP relative to CL) is consistent with this premise being wrong: coarse positives dilute exact-pair similarity. Section 4 attributes the coarse-level gains to 'hierarchical information' without a control that separates the LIC taxonomy's semantic content from mere label co-occurrence. Because the same taxonomy supplies both the training labels and the evaluation labels, the subclass/main-class mAP improvements could reflect overfitting to administrative classes rather than a useful visual hierarchy. The abstract's unqualified claim that the method 'enhances the retrieval results' is therefore not yet established; at minimum it requires a stated trade-off and a mechanism check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hierarchical multi-positive contrastive learning (HMCL) loss for patent image retrieval. Instead of treating only same-patent images as positives, Eq. (4) weights every pair in the batch by a score h_ij from Eq. (2), which assigns decreasing relevance to images sharing the same patent ID, subclass, or main class under the Locarno International Classification (LIC). The authors fine-tune ViT, ResNet, and CLIP models on DeepPatent2 and report mAP/nDCG at three hierarchical levels, concluding that the method improves retrieval, especially for low-parameter models.","tokens_in":8689,"tokens_out":3241,"duration_ms":29985,"significance":"The loss formulation is simple, coherent, and a natural extension of standard contrastive learning, and the paper's strengths include multi-seed experiments across several architectures, use of the public DeepPatent2 benchmark, and a clear hierarchical evaluation protocol. If the claims were fully supported, the method would be a useful, low-cost contribution to patent image retrieval. However, the evidence is currently mixed: patent-ID mAP drops on several models, and the mechanism is not isolated from label co-occurrence, so the central claim is not yet established as stated.","major_comments":[{"comment":"The abstract and Section 5 claim that HMCL 'enhances retrieval performance at all hierarchical levels,' but the patent-ID mAP in Table 1 decreases for ViT-Small (0.349 to 0.347), ViT-Base (0.333 to 0.324), and ViT-Large (0.348 to 0.345) relative to CL, and in Table 2 for CLIP-B/16* (0.373 to 0.356), CLIP-L/14* (0.454 to 0.452), CLIP-B/16 (0.401 to 0.386), and CLIP-L/14 (0.458 to 0.439). This directly contradicts the unqualified claim that the method enhances retrieval results. The paper must either restrict the claim to the subclass and main-class levels or characterize the trade-off explicitly, and the explanation in Section 4 that this deterioration is 'expected' should be supported by a mechanism check rather than asserted.","section":"Table 1 and Table 2"},{"comment":"The scores sp=1, ss=0.35, sm=0.2 and the weight λ=0.2 are fixed without any sensitivity analysis, although the entire method rests on these hand-set hyperparameters. Please provide an ablation over these values (at least over ss and sm, and ideally over λ and the temperature τ) to show that the reported subclass/main-class gains are not an artifact of a particular weighting. Without such an ablation, it is unclear whether the method is robust or tuned to one setting.","section":"Section 3, Eq. (2) and Eq. (5)"},{"comment":"The LIC taxonomy is used both to define the training positives (Eq. 2) and to define the evaluation relevance sets, so the observed subclass/main-class improvements could reflect fitting to administrative label structure rather than visual-semantic similarity. The paper needs a control experiment that breaks the label-signal link—for example, randomly permuting subclass and main-class labels during training, or training with non-hierarchical co-class positives—to establish that the gains come from the LIC hierarchy and not from label co-occurrence alone. This is load-bearing because the entire motivation of the method is the semantic usefulness of the taxonomy.","section":"Section 3 and Section 4"}],"minor_comments":[{"comment":"The keyword 'Hierarchical Multipositive Contrastive Learning' and the phrase 'hierarchical multipositive contrastive learning' in Section 5 use 'Multipositive' as one word, while the abstract and title use 'multi-positive.' Please use hyphenated spelling consistently throughout.","section":"Keywords and Section 5"},{"comment":"State explicitly whether the sums over j include the anchor itself. The standard contrastive loss excludes the self-pair, and the notation should be unambiguous about this; otherwise the normalization factor H_i in Eq. (3) cannot be verified.","section":"Eq. (4) and Eq. (5)"},{"comment":"The paragraph reporting standard deviations gives approximate aggregate values but does not report them per model or per method. Please include per-method standard deviations in the tables or a supplementary table, since the paper claims multi-seed experiments and the reader cannot assess the significance of the differences.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a tidy little workshop paper with a honest core idea and unusually broad experiments for six pages, but the abstract says 'enhances the retrieval results' without qualification, and the tables don't support that at the patent level for five model variants.\n\nWhat's genuinely new: the weighted multi-positive contrastive loss over LIC levels, Eq. 4 with h_ij defined in Eq. 2. It's a straightforward extension of soft-label contrastive learning, but I haven't seen it applied to patent retrieval, and the graded hierarchy (patent > subclass > main class) is a reasonable inductive bias. The authors also correctly note that Eq. 1 is a special case, and they run ResNet, ViT, CLIP vision-only, and CLIP vision+language with multiple seeds. The standard deviations are small and roughly consistent across methods, and the paper is honest in Section 4 that higher-level gains come with a patent-level cost on larger models.\n\nThe soft spots are real but not disqualifying. First, the abstract and conclusion claim gains at 'all hierarchical levels,' which contradicts the patent-ID mAP drops for ViT-Small, ViT-Base, ViT-Large, CLIP-B/16*, CLIP-L/14*, CLIP-B/16, and CLIP-L/14 (Tables 1 and 2). The body does acknowledge this trade-off in one sentence, so the mismatch is mostly an abstract problem, but it's still a misleading headline. Second, the weights sp=1, ss=0.35, sm=0.2 and lambda=0.2 are set by hand with no sensitivity analysis or ablation. The authors say these values 'offer a balanced performance,' but that's an empirical claim without support. Third, there's no comparison with other hierarchical or soft-label contrastive methods, so it's hard to know if the specific weighting matters or just the multi-positive extension. Fourth, no code is released; the training details are sufficient for replication, but only barely.\n\nOn the stress-test note: yes, the abstract overclaims, and a label-shuffle control would have separated the LIC taxonomy's semantic content from mere label co-occurrence. That said, the paper's actual argument is about retrieval performance, not about the perceptual reality of LIC classes, and the body already concedes the patent-level cost. So the stress-test lands mostly on the abstract, not on the method's internal logic.\n\nWho is this for? Anyone working on patent image retrieval or on hierarchical contrastive losses with cheap taxonomy labels. It's a modest incremental contribution, not a breakthrough, but the formulation is clean and the experimental breadth is useful. I'd send it to a serious referee. The fixes are easy: correct the abstract, add a sensitivity analysis for the weights, and ideally a shuffled-label control. Without those, the result is still a reasonable baseline, just with an overstated headline.","headline":"Clean, honest core idea with a misleading abstract: patent-level mAP drops in the paper's own tables contradict the 'enhances retrieval' headline, but the method is a reasonable baseline and worth a referee.","tokens_in":9158,"tokens_out":4838,"would_cite":false,"duration_ms":43326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes replacing the standard single-positive contrastive loss with a hierarchy-weighted multi-positive loss, using the Locarno International Classification to assign patent images multiple positive pairs with decreasing…","keywords":["patent image retrieval","hierarchical contrastive learning","multi-positive contrastive loss","Locarno International Classification","technical drawing retrieval","low-parameter models","vision transformer","multimodal CLIP fine-tuning"],"falsifier":"Retrain the same encoders with the same batches but shuffled or random LIC labels; if subclass- and main-class mAP still rises above conventional contrastive learning, the gains are not caused by the taxonomy's semantic structure. A second check is to count, for a fixed query, how often the top retrieved same-subclass images are visually unrelated drawings: enough such cases would indicate the loss is aligning administrative labels rather than visual content.","tokens_in":8284,"feed_emoji":"🔍","tokens_out":5110,"duration_ms":47385,"temperature":0.7,"pith_summary":"Patent images are technical drawings whose retrieval rankings often ignore how patents are categorized. The paper argues that the Locarno International Classification (LIC) hierarchy encodes useful visual-semantic closeness: two images from the same patent should be pulled together most strongly, images from the same LIC subclass less so, and images from the same main class least but still positively. The proposed hierarchical multi-positive contrastive loss replaces the single-positive contrastive loss with a weighted sum over many positives, and the reported experiments on DeepPatent2 show that this raises retrieval accuracy at subclass and main class levels across ResNet, ViT, and CLIP encoders. A reader should care because the method is simple enough for low-parameter models and plugs into existing fine-tuning pipelines without a new architecture.","feed_headline":"Hierarchy-aware loss lifts patent image retrieval","feed_subtitle":"Same-subclass and same-class positives improve ranking on DeepPatent2, most for small models.","key_machinery":"The load-bearing object is the hierarchical multi-positive contrastive loss of Equation 4, built from the relevance function $h_{ij}$ of Equation 2 and its normalization $H_i$ of Equation 3. $h_{ij}$ converts the LIC taxonomy into training weights, so the loss maximizes the cosine similarity of an anchor to several positives, weighted by how close they are in the patent hierarchy; the standard contrastive loss of Equation 1 is the special case with a single positive pair scored 1. This mechanism is what injects the hierarchy into the embedding space, and it generalizes to any taxonomy by redefining $h_{ij}$.","core_discovery":"The central claim is that replacing the standard contrastive loss with a hierarchy-weighted multi-positive version improves patent image retrieval. Concretely, for an anchor image $x_i$, every other image $\\tilde{x}_j$ in the batch receives a relevance score $h_{ij}$: $s_p=1$ for same patent ID, $s_s=0.35$ for same LIC subclass, $s_m=0.2$ for same main class, and $0$ otherwise, and the loss in Equation 4 uses $h_{ij}/H_i$ as weights over the positive log-probabilities. The authors report that this formulation outperforms conventional contrastive fine-tuning at the subclass and main class levels on almost every model tested, with the largest gains on small models such as ResNet-18 and ViT-Tiny, while acknowledging a small patent-ID level drop for larger ViT and CLIP models. The hierarchy signal also produces more separable subclass clusters in the embedding space, as shown by PCA projections.","pith_inferences":["A natural experiment the paper does not run is to permute the LIC labels during training; if subclass-level mAP gains persist under shuffled taxonomy labels, the effect would come from extra positives in general rather than from hierarchy semantics.","The hand-set weights $s_p=1$, $s_s=0.35$, $s_m=0.2$ are arbitrary; optimizing them per level or per subclass may widen or narrow the reported trade-off, and the paper's own claim would be strengthened by a sensitivity analysis.","The larger gains for small models suggest the hierarchy acts partly as a regularizer; one could test this by measuring whether HMCL also improves linear probing on held-out subclasses not seen during fine-tuning."],"forward_implications":["At the subclass and main class levels, retrieval quality improves on the DeepPatent2 test set for ResNet, ViT, and CLIP encoders when the hierarchical loss replaces standard contrastive fine-tuning.","Small models, including ResNet-18 and ViT-Tiny, receive the largest relative gains, which suggests useful patent retrieval can run on hardware with limited compute.","The method trades a small amount of exact patent-ID precision on larger ViT and CLIP models for broader-level ranking gains, an explicit design trade-off.","Because Equation 4 only depends on the scoring function $h_{ij}$, the same recipe transfers to other classification systems, such as the Cooperative Patent Classification.","Adding the text-supervision term of Equation 5 further improves multimodal CLIP retrieval, so the hierarchy signal composes with language supervision."],"supporting_citations":[{"why":"Supplies the DeepPatent dataset and the retrieval evaluation setup that the experiments build on.","marker":"[1]"},{"why":"Supplies the Vision Transformer architecture used in the small and large model experiments.","marker":"[4]"},{"why":"Supplies the ResNet architecture used for the lower-parameter experiments.","marker":"[5]"},{"why":"Provides the contrastive loss formulation and the pretrained CLIP models used as encoders in the multimodal experiments.","marker":"[11]"},{"why":"Supplies DeepPatent2, the dataset with LIC codes and object descriptions used for training and evaluation.","marker":"[12]"},{"why":"Defines the Locarno International Classification hierarchy that the scoring function $h_{ij}$ converts into positive weights.","marker":"[15]"},{"why":"Establishes the contrastive predictive coding loss that Equation 1 adapts as the baseline contrastive objective.","marker":"[20]"}],"fun_headline_variants":["Multi-positive contrastive loss uses patent taxonomy","Hierarchy-weighted positives boost patent image search","Taxonomy-informed loss improves retrieval for small models","Patent class hierarchy fine-tunes image retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the LIC taxonomy tracks visual relevance: images sharing a subclass or main class are treated as usefully similar even when they depict different objects, so the hierarchy can serve as a training signal.","fun_headline_variants_meta":{"raw":{"variants":["Multi-positive contrastive loss uses patent taxonomy","Hierarchy-weighted positives boost patent image search","Taxonomy-informed loss improves retrieval for small models","Patent class hierarchy fine-tunes image retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1161,"prompt_tokens":940,"completion_tokens":221,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":165}},"tokens_in":556,"tokens_out":221,"duration_ms":2607,"temperature":1.0,"reasoning_tokens":165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:59:32.511709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same encoders with the same batches but shuffled or random LIC labels; if subclass- and main-class mAP still rises above conventional contrastive learning, the gains are not caused by the taxonomy's semantic structure. A second check is to count, for a fixed query, how often the top retrieved same-subclass images are visually unrelated drawings: enough such cases would indicate the loss is aligning administrative labels rather than visual content.","supporting_citations":[{"cited_title":"Kucer, D","cited_arxiv_id":null,"evidence_quote":"Supplies the DeepPatent dataset and the retrieval evaluation setup that the experiments build on."},{"cited_title":"Dosovitskiy, L","cited_arxiv_id":null,"evidence_quote":"Supplies the Vision Transformer architecture used in the small and large model experiments."},{"cited_title":"Ajayi, X","cited_arxiv_id":null,"evidence_quote":"Supplies DeepPatent2, the dataset with LIC codes and object descriptions used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Locarno International Classification hierarchy that the scoring function $h_{ij}$ converts into positive weights."}],"review_version":2}