pith. sign in

arxiv: 2604.01619 · v3 · submitted 2026-04-02 · 💻 cs.CV · cs.AI

Automatic Image-Level Morphological Trait Annotation for Organismal Images

Vardaan Pahuja , Samuel Stevens , Alyson East , Sydne Record , Yu Su This is my paper

Pith reviewed 2026-05-13 21:44 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords morphological trait annotationsparse autoencodersfoundation modelsinsect imagesbioscan datasetvision-language promptingbiological image analysistrait localization
0
0 comments X

The pith

Sparse autoencoders on foundation-model features produce monosemantic neurons that activate on morphological parts of insects.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that sparse autoencoders trained on features from large vision models isolate individual neurons each tied to specific biological structures in insect images. These neurons support a pipeline that locates key regions and then uses vision-language prompting to produce natural-language descriptions of morphological traits. The process generates a dataset called Bioscan-Traits containing 80,000 annotations across 19,000 images. This replaces slow expert labeling with an automated method, making large-scale studies of organism-environment interactions more feasible. Human reviewers confirm the resulting descriptions align with visible biological features.

Core claim

Training sparse autoencoders on foundation-model features yields monosemantic, spatially grounded neurons that consistently activate on meaningful morphological parts. This property enables a trait annotation pipeline that localizes salient regions and generates interpretable trait descriptions through vision-language prompting, producing the Bioscan-Traits dataset of 80K annotations from 19K insect images.

What carries the argument

sparse autoencoders trained on foundation-model features that yield monosemantic, spatially grounded neurons activating on morphological parts

If this is right

  • Constructs Bioscan-Traits, a dataset of 80K trait annotations spanning 19K insect images.
  • Supports large-scale morphological analyses without manual expert effort.
  • Provides a modular pipeline for injecting biologically meaningful supervision into foundation models.
  • Bridges ecological relevance and machine-learning practicality for organismal studies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same neuron-isolation step could be applied to images of plants or other non-insect organisms.
  • The resulting trait annotations might serve as weak supervision signals to improve accuracy on related vision tasks such as species classification.
  • Automated trait extraction could enable systematic comparisons of morphological variation across geographic regions or environmental gradients.
  • The pipeline might surface previously overlooked fine-scale traits that human annotators tend to miss.

Load-bearing premise

The monosemantic neurons reliably match biologically meaningful traits and the vision-language prompting produces accurate non-hallucinated descriptions.

What would settle it

Expert biologists examining the generated trait descriptions and finding consistent mismatches with the visible morphological features shown in the source images.

Figures

Figures reproduced from arXiv: 2604.01619 by Alyson East, Samuel Stevens, Sydne Record, Vardaan Pahuja, Yu Su.

Figure 1
Figure 1. Figure 1: Given an input specimen image, we first compute dense visual representations using [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of trait localization for Thymoites guanicae. BIOSCAN-TRAITS (left) generates interpretable trait descriptions that are tied to clear, specific anatomical structures. In contrast, Grad-CAM (center) produces diffuse heatmaps that highlight broad body areas without species-level disentanglement. 4 EXPERIMENTS 4.1 SPARSE AUTOENCODER TRAINING We use the BIOSCAN-5M dataset (Gharaee et al., 2024) for … view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of salient morphological trait description generation using a just MLLM vs. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of salient morphological trait description generation using a single image [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Neurons 4852 and 13860 in SAE get activated at the wings and antennae of insects, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Variation of rating with different lev￾els of SAE sparsity. A lower level of sparsity performs better for both values of frequency threshold tfreq. SAE Filtering. We analyze the effect of the normalized frequency threshold tfreq on the trait through￾put using 1,000 input images and sparsity coefficient (α) = 4e − 4. We observe that increasing tfreq leads to a progressive reduction in the number of retained… view at source ↗
read the original abstract

Morphological traits are physical characteristics of biological organisms that provide vital clues on how organisms interact with their environment. Yet extracting these traits remains a slow, expert-driven process, limiting their use in large-scale ecological studies. A major bottleneck is the absence of high-quality datasets linking biological images to trait-level annotations. In this work, we demonstrate that sparse autoencoders trained on foundation-model features yield monosemantic, spatially grounded neurons that consistently activate on meaningful morphological parts. Leveraging this property, we introduce a trait annotation pipeline that localizes salient regions and uses vision-language prompting to generate interpretable trait descriptions. Using this approach, we construct Bioscan-Traits, a dataset of 80K trait annotations spanning 19K insect images from BIOSCAN-5M. Human evaluation confirms the biological plausibility of the generated morphological descriptions. We assess design sensitivity through a comprehensive ablation study, systematically varying key design choices and measuring their impact on the quality of the resulting trait descriptions. By annotating traits with a modular pipeline rather than prohibitively expensive manual efforts, we offer a scalable way to inject biologically meaningful supervision into foundation models, enable large-scale morphological analyses, and bridge the gap between ecological relevance and machine-learning practicality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that sparse autoencoders trained on foundation-model features produce monosemantic, spatially grounded neurons that activate on meaningful morphological parts of insects. It leverages this to build a pipeline that localizes regions and uses vision-language prompting to generate trait descriptions, yielding the Bioscan-Traits dataset (80K annotations on 19K images from BIOSCAN-5M). Validation consists of human plausibility ratings and a design ablation study.

Significance. If the central claim holds, the work provides a scalable, modular route to large-scale morphological trait annotation without exhaustive expert labeling. This could inject biologically relevant supervision into vision models and support ecological analyses at the scale of BIOSCAN-5M. The explicit construction of a public dataset and the ablation on pipeline components are concrete strengths.

major comments (2)
  1. [Evaluation and Results] The assertion that SAE neurons are monosemantic and reliably map to biologically meaningful morphological traits rests on qualitative activation visualizations, VLM prompting, and human plausibility ratings. No quantitative validation against expert part annotations (e.g., neuron-to-part alignment scores, consistency metrics across images, or comparison to ground-truth segmentations) is reported, leaving open whether activations reflect true structures or dataset/foundation-model artifacts.
  2. [Trait Annotation Pipeline] The weakest link in the pipeline—the assumption that vision-language prompting yields accurate, non-hallucinated trait descriptions—is tested only via human ratings rather than direct comparison to expert trait labels or inter-annotator agreement on a held-out set. This directly affects the reliability of the 80K-annotation Bioscan-Traits dataset.
minor comments (2)
  1. [Methods] The abstract and methods should explicitly state the foundation model backbone, SAE hyperparameters (including the sparsity penalty value), and exact prompting templates used for trait generation.
  2. [Figures] Figure captions for activation maps should include the number of images shown, the source model layer, and any thresholding applied to neuron activations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback on our work. We address the major comments point-by-point below, indicating where revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: [Evaluation and Results] The assertion that SAE neurons are monosemantic and reliably map to biologically meaningful morphological traits rests on qualitative activation visualizations, VLM prompting, and human plausibility ratings. No quantitative validation against expert part annotations (e.g., neuron-to-part alignment scores, consistency metrics across images, or comparison to ground-truth segmentations) is reported, leaving open whether activations reflect true structures or dataset/foundation-model artifacts.

    Authors: We acknowledge that the current validation is largely qualitative. To address this, we will incorporate quantitative consistency metrics in the revised manuscript, such as the average pairwise overlap of localized activation regions for neurons corresponding to the same trait across different images. We will also include a comparison to non-SAE baselines to rule out artifacts. These additions will provide stronger evidence for the biological meaningfulness of the neurons. revision: yes

  2. Referee: [Trait Annotation Pipeline] The weakest link in the pipeline—the assumption that vision-language prompting yields accurate, non-hallucinated trait descriptions—is tested only via human ratings rather than direct comparison to expert trait labels or inter-annotator agreement on a held-out set. This directly affects the reliability of the 80K-annotation Bioscan-Traits dataset.

    Authors: We agree that direct expert validation is desirable. Given the scale, we will add inter-annotator agreement scores from our human raters and a small-scale comparison against expert annotations on a held-out subset of 100 images. This will be reported in the revised version to better support the reliability of the Bioscan-Traits dataset. We note that full expert annotation remains infeasible, which underscores the need for our automated approach. revision: partial

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper proposes an empirical pipeline that trains sparse autoencoders on foundation-model features, localizes regions via neuron activations, and generates trait descriptions via vision-language prompting. All central claims about monosemanticity, spatial grounding, and biological plausibility are supported by human evaluation ratings and ablation studies rather than any closed mathematical derivation. No equations, fitted parameters, or self-citations are presented as load-bearing derivations that reduce the output to the input by construction. The methodology remains self-contained with external validation steps that do not rely on renaming prior results or smuggling ansatzes through citations.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The approach depends on the assumption that foundation-model features contain decomposable semantic information about biological morphology and that language models can reliably translate localized activations into valid trait descriptions.

free parameters (1)
  • sparsity penalty in autoencoder training
    Standard hyperparameter controlling monosemanticity whose specific value is not stated in the abstract.
axioms (1)
  • domain assumption Pre-trained vision foundation models produce features that can be decomposed into biologically meaningful parts via sparsity
    Invoked when stating that SAEs yield monosemantic neurons for morphological parts.

pith-pipeline@v0.9.0 · 5519 in / 1123 out tokens · 63414 ms · 2026-05-13T21:44:22.521839+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 1 internal anchor

  1. [1]

    Zahra Gharaee, Scott C

    URLhttps://openreview.net/forum?id=tcsZt9ZNKD. Zahra Gharaee, Scott C. Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Eyriay, Lila Kari, Dirk Steinke, Graham W. Taylor, Paul W. Fieguth, and Angel X. Chang. Bioscan-5m: A multimodal dataset for insect biodiversity. In Proceedings of NeurIPS, 2024. ...

    work page doi:10.1162/dint_a_00134 2024

  2. [2]

    Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Y

    URLhttps://openreview.net/forum?id=DaNnkQJSQf. Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Y . Berger-Wolf, Charles V . Stewart, Song Gao, Wei-Lun Chao, and Yu Su. Reviving the context: Camera trap species classification as link prediction on multimodal knowledge graphs. InProceedings of CIKM, 2024. URLhttps://doi.org/10.1145/3627...

    work page doi:10.1145/3627673.3679545 2024

  3. [3]

    L., Jordano, P., & Bascompte, J

    URL https://nsojournals.onlinelibrary.wiley.com/doi/10.1111/ j.0030-1299.2007.15559.x. Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. 2011. URL https://authors.library.caltech.edu/ records/cvm3y-5hh21. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen...

    work page arXiv 2007

  4. [4]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    URLhttps://arxiv.org/abs/2409.12191. Di Wu, Siyuan Li, Zelin Zang, and Stan Z Li. Exploring localization for self-supervised fine-grained contrastive learning. InProceedings of BMVC, 2022. URL https://bmvc2022.mpi-inf. mpg.de/0268.pdf. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christo- pher D Manning, and Christoph...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  5. [5]

    background

    For every highlighted region, determine whether it contains a visible insect body part or just background. If it is mostly background, respond with "background"

    work page

  6. [6]

    Use only the visual information present in the image

    If it contains a visible body part, identify which part it is (e.g., leg, wing, antenna), and describe its visible morphological traits: shape, size, color, texture, and any distinct markings. Use only the visual information present in the image. After analyzing all three highlighted regions in images:

    work page

  7. [7]

    Important Instructions: - Do not infer or assume information that is not directly observable

    Identify and list the morphological traits that are common across all three regions, solely based on what is visible in all images. Important Instructions: - Do not infer or assume information that is not directly observable. Avoid adding external knowledge. - Use only what is clearly visible. - Be concise. Limit the total response to under 200 tokens. Ou...

    work page

  8. [8]

    background

    Determine whether the highlighted region contains a visible body part of the insect or only the background. If it appears to be background, respond with “background”

    work page

  9. [9]

    Then, briefly describe the observable morphological features - such as shape, size, color, texture, or distinct markings - based solely on what is visible in the image

    If it contains a visible body part, identify which part it is. Then, briefly describe the observable morphological features - such as shape, size, color, texture, or distinct markings - based solely on what is visible in the image. IMPORTANT: Do not infer or assume information that is not directly observable. Avoid adding external knowledge. Figure C.3: P...

    work page 2026

  10. [10]

    Identify the visible body parts of the insect (e.g., head, thorax, abdomen, legs, wings, antennae), common in all three images

    work page

  11. [11]

    For each part, identify its morphological features - such as shape, size, color, texture, or distinct markings

    work page

  12. [12]

    Only output traits that are visibly consistent across all images

    After analyzing all three images individually, list the morphological traits that are common across all three insects. Only output traits that are visibly consistent across all images. IMPORTANT: Do not infer or assume information that is not directly observable. Avoid adding external knowledge. Figure C.4: Prompt for MLLM-only baseline (multiple images) ...

    work page

  13. [13]

    Identify the visible body parts of the insect (e.g., head, thorax, abdomen, legs, wings, antennae)

    work page

  14. [14]

    IMPORTANT:

    For each part, briefly describe the observable morphological features - such as shape, size, color, texture, or distinct markings - based solely on what is visible in the image. IMPORTANT:

    work page

  15. [15]

    Avoid adding external knowledge

    Do not infer or assume information that is not directly observable. Avoid adding external knowledge

    work page

  16. [16]

    A photo of <species-name> with <trait-description>

    Keep your response concise and under 200 tokens. Figure C.5: Prompt for MLLM-only baseline (single image) D EXPERIMENTALSETUP D.1 HYPERPARAMETERCONFIGURATION Table D.4 summarizes all hyperparameters used for SAE training and dataset generation. We experiment with different learning rate values and choose 1e−3 based on qualitative inspection of learned tra...

    work page 2022