Pith. sign in

REVIEW 4 major objections 4 minor 9 references

SAM$^{*}$: Task-Adaptive SAM with Physics-Guided Rewards

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAM's hidden tuning parameters can be set automatically by optimizing physics-derived reward functions over candidate masks, yielding SAM*.

desk verdict Useful framework idea, but the empirical claim is unproven: the reward is both optimizer and evaluator, and there is no ground truth. read the letter →

arxiv 2509.07047 v1 pith:J2YJEXJZ submitted 2025-09-08 cs.CV cond-mat.mtrl-scics.LG

Kamyar Barakati , Utkarsh Pratiush , Sheryl L. Sanchez , Aditya Raghavan , Delia J. Milliron , Mahshid Ahmadi , Philip D. Rack , Sergei V. Kalinin This is my paper
classification cs.CVcond-mat.mtrl-scics.LG
keywords imagesegmentationSegmentAnythingModelreward-guidedoptimizationhyperparametertuningNSGA-IImicroscopynanoparticleanalysismulti-objective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the many tuning parameters of the Segment Anything Model (SAM), normally set by hand, can be optimized automatically by defining rewards that encode what the imaged material should look like. Across three nanoparticle datasets, the authors replace pixel-wise accuracy with quantitative targets—overlap counts, circularity, aspect ratio, mask area—and run a genetic algorithm to search SAM's hyperparameter space. The result, SAM*, detects overlapping indium-tin-oxide nanocrystals that vanilla SAM merges (415 vs 349 masks) and can balance small-particle detection against large-aggregate coverage in gold-cobalt nanoparticles. The paper also reports a limit: hyperparameter tuning alone cannot make SAM distinguish circular from rectangular particles, because the underlying model has no shape concept.

What carries the argument

The reward layer and NSGA-II search loop: mask sets produced by SAM's automatic pipeline are scored by equations 1–8 that count distinct overlapping pairs, penalize near-duplicate masks and merged blobs, and measure circularity, aspect ratio, inverse/mean area; NSGA-II then proposes new hyperparameter vectors (points_per_side, IoU thresholds, NMS thresholds, crop layers) and iterates until the reward landscape converges. This carries the argument because it is what turns 'physically meaningful' from a qualitative idea into a scalar or multi-objective target.

What would settle it

Manually annotate the three micrographs and compare SAM* masks against those labels: if the extra masks (e.g., 415 versus 349 for ITO) do not correspond to real particles, or if reward-optimal hyperparameters do not improve agreement with human annotations, the central improvement claim fails.

Watch

Extended reading notes

Core claim

The central claim is that segmentation quality for a specific scientific task can be treated as an optimization problem over the hyperparameters of a frozen foundation model, using reward functions derived from the physics of the sample rather than from annotated labels. Under this view, 'good segmentation' means recovering overlapping particles, preserving morphology classes, or covering both small and large features, and each objective is written as a closed-form function of the candidate mask set. The authors show that optimizing these rewards produces better particle recovery than SAM's default settings, and the Pareto front of conflicting objectives gives the user a menu of operating po

Load-bearing premise

The reward functions are assumed to capture what correct segmentation means, but the tuned masks are never checked against human annotations or physical ground truth.

Editorial extensions

If this is right

  • If correct, any foundation model with opaque knobs can be specialized to a domain without labels, as long as a reward can be written down.
  • Overlapping nanocrystals that vanilla SAM merges can be recovered by tuning NMS and crop settings found by search (415 vs 349 masks for ITO).
  • In conflicting-objective settings, the Pareto front gives physically interpretable operating points, e.g., 117 masks favoring small particles versus 11 masks favoring large aggregates.
  • The framework suits real-time streaming microscopy because optimization happens offline and inference uses the tuned SAM.
  • The method does not claim to add new semantic capabilities: morphology discrimination remains out of reach without changing SAM's architecture or training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The numerical gains are internally consistent but not yet tied to true particle counts; a labeled benchmark would tell whether more masks equals more correct masks.
  • The same reward loop could be applied to other promptable segmentation backbones or other imaging modalities, since the rewards only need mask statistics.
  • A natural next step is to make the rewards differentiable and fine-tune SAM's decoder directly, rather than searching only the AutoMaskGenerator hyperparameters.
  • For real-time use, the search cost could be amortized by learning a mapping from image statistics to optimal hyperparameters, eliminating per-image search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SAM*, a framework that tunes the hyperparameters of SAM's automatic mask generator using physics-guided reward functions rather than manual expert tuning. Reward functions are defined for three nanoparticle microscopy tasks: an overlap reward for ITO nanocrystals (Eq. 4), morphology rewards for CsPbBr3 perovskite nanocrystals (Eqs. 5-6), and an antagonistic small-vs-large-area reward pair for AuCo nanoparticles (Eqs. 7-8). Hyperparameters are optimized with NSGA-II (Supplementary Information) and the resulting masks are compared visually and by mask counts against vanilla SAM. The claimed contribution is that task-adaptive reward optimization improves SAM's segmentation relevance and enables real-time, label-efficient adaptation.

Significance. The proposed idea—using hand-designed, physics-informed reward functions to steer a foundation model's hyperparameters without labels—is timely and potentially useful for microscopy and other scientific imaging domains where labeled data are scarce. The paper is also commendably honest in Section V, where it reports that hyperparameter tuning alone cannot impart morphology awareness. However, the central claim that SAM* 'improves performance' is not established because the reported success metrics are the very same reward functions used as optimization objectives, and no independent ground-truth validation or comparison to existing segmentation tools is provided. As a result, the paper's main conclusion rests on a circular evaluation. The manuscript also provides no code, no error bars, and only a single image per domain, making the quantitative claims of improvement unsubstantiated.

major comments (4)
  1. [§IV, Eq. (4) and §VI, Eqs. (7)–(8)] The evaluation is circular. In Section IV, the reward F(M) in Eq. (4) is the objective maximized by NSGA-II, and the reported improvement from 349 to 415 masks is exactly the quantity this reward encourages: more partially overlapping mask pairs O and fewer duplicates D or merges B. Likewise, in Section VI, the Pareto endpoints of 117, 22, and 11 masks are points along the trade-off defined by Eqs. (7)–(8); the optimizer is designed to produce such counts. No independent metric (e.g., precision/recall or IoU against human or synthetic ground truth) is reported, so the improvements are consequences of the optimization setup rather than evidence of correct segmentation.
  2. [§II, Figs. 3–8] Each of the three demonstrations uses a single image, with no ground-truth annotations, no comparison against the cited alternatives (μSAM, SAM-I-Am, AtomSegNet, classical segmentation), and no statistical analysis across images or runs. Without such validation, the claim that SAM* 'better aligns with requirements of diverse segmentation tasks' is not supported. A single favorable visual example and a mask count do not establish segmentation accuracy, especially in microscopy where particle identity and physical connectivity are the quantities of interest.
  3. [§VI, Eqs. (7)–(8) and surrounding text] There is an internal inconsistency between the equations and the text. Eq. (7) rewards small mask areas (via 1/A + ε) and Eq. (8) rewards large mean mask area, but the paragraph after Eq. (8) states that F1 'rewards large, contiguous masks' and F2 'rewards the detection of nanoscale particles by emphasizing smaller mask areas.' If the text is intended to describe F1 and F2, it swaps their definitions; if it is not, the naming is confusing. This ambiguity also affects Figure 8's 'Objective-1/Objective-2' labels and must be corrected. More importantly, the section presents the Pareto endpoints as 'improved detection' without any check that the small masks correspond to real sub-100-nm particles rather than fragments of larger aggregates.
  4. [§V, 'Morphology-Aware Framework'] The paper explicitly states that SAM* 'failed to discriminate between circular and rectangular nanocrystals' and that 'hyperparameter tuning alone cannot impart morphology awareness.' This is an admitted limitation, and I appreciate the honesty. However, it substantially narrows the paper's scope: the 'diverse segmentation tasks' claimed in the abstract and conclusion reduce to overlap recovery and size-selectivity trade-offs. The central claim should be revised accordingly, and the morphology scenario should be presented as a negative result rather than a demonstration of the method's effectiveness.
minor comments (4)
  1. [§I, first paragraph] Grammatical error: 'Here we introduce a physics-aware, reward-guided framework is introduced' should read 'Here we introduce a physics-aware, reward-guided framework.'
  2. [Table 1] The column 'Operational status' mixes scenario-dependent, fixed, and effectively-fixed categories; consider renaming to 'Sensitivity' and using a consistent legend (e.g., 'fixed in this work' vs. 'tuned'). Also, 'points_per_batch' is listed as affecting only speed/memory, but Section VI names it as one of the tuned hyperparameters; clarify whether it affects masks.
  3. [Figure 8 and Eq. (2)] Figure 8 labels 'Objective-1/Objective-2' but the text and equations use F1 and F2; align the labels. In Eq. (2), τ_dup is defined but no example threshold is given as in Eqs. (1) and (3); supply one or remove the implicit dependence.
  4. [References and data] The reference list has inconsistent formatting and some incomplete entries (e.g., Ref. 2 lacks venue; Ref. 29 formatting). Data availability states that data are available from corresponding authors upon request, but no code or hyperparameter search details beyond the SI description are provided; making the NSGA-II configuration and datasets public would improve reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Optimization objective doubles as the only success metric; reported gains are the fitted reward components, not validated segmentation accuracy.

  1. fitted input called prediction [Section IV, Eq. 4 and the '415 masks' paragraph]
    "With these definitions, the reward function is: ℱ(ℳ)= (1+𝑂(ℳ))/(1+𝛼.𝐷(ℳ)+ β.𝐵(ℳ)) [4] ... With physics-aware tuning (SAM*), segmentation performance improves substantially compared to the vanilla configuration. A total of 415 masks were detected, exceeding the 349 obtained previously, indicating enhanced sensitivity in crowded regions."

    NSGA-II is used to maximize ℱ, whose numerator O counts overlapping mask pairs and whose denominator penalizes duplicates D and merges B. The reported success—more masks and separated overlaps—is therefore the optimizer reporting its own objective. The paper supplies no human annotations, synthetic ground truth, or independent segmentation benchmark that would show the reward-maximizing mask set corresponds to real overlapping nanocrystals. The 'improvement' is forced by construction: a better-optimized reward score is called segmentation performance.

  2. self definitional [Section VI, Eqs. 7–8 and Fig. 8 Pareto-front discussion]
    "For small-particle sensitivity, the reward is designed to favor detection of minimal regions by penalizing mask area: ℱ!(ℳ)= 1/|ℳ|`1/(Α(𝑚)+𝜀) ... Conversely, for large-particle coverage, we maximize the mean detected area: ℱ"(ℳ)= 1/|ℳ|`Α(𝑚) ... The resulting optimization space forms a Pareto front in the ℱ!–ℱ" plane ... A configuration biased toward ℱ!–ℱ" as shown in Fig. 8b yields 117 detected instances ... At the opposite extreme ... produces only 11 masks."

    The three reported outcomes (117, 22, 11 masks) are exactly the Pareto-optimal settings of the two hand-designed area rewards. The trade-off between small-mask and large-mask objectives is built into the reward definitions, and the paper explicitly notes the small-particle endpoint over-fragments large islands and the large-particle endpoint drops nanoparticles. Without independent labels, saying SAM* 'better reconciles' the objectives is just restating that a Pareto point balances the two user-defined reward functions. The evaluation metric and the optimization target are the same quantities.

full rationale

The paper's derivation chain is an optimization loop, not an independent predictive test. The reward functions in Eqs. 1–8 are the optimization objectives; the same quantities are then reported as evidence that SAM* 'improves' segmentation. Because NSGA-II is explicitly run to maximize F (Eq. 4) or the Pareto objectives (Eqs. 7–8), any increase in those reward components relative to vanilla is guaranteed by the search (up to optimizer failure), not by comparison to physical ground truth. No human annotations, synthetic ground truth, or independent segmentation benchmark are used. The honest failure in Section V (morphology-awareness cannot be achieved by hyperparameter tuning) is a useful negative control but does not validate the positive claims. The central quantitative claims (415 vs 349 masks; Pareto endpoints 117/22/11) are therefore partial circularity: the objective and metric coincide. The paper is not relying on self-citation or imported uniqueness theorems; the circularity is of the fitted-input-called-prediction / self-definitional kind. Score 6.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The method rests on several hand-set thresholds and weights (tau_l, tau_h, tau_dup, gamma, alpha, beta, epsilon) and on the assumption that simple geometric rewards encode physical correctness. The only 'invented entity' is the SAM* configuration, which is a hyperparameter vector, not a new physical or model component. No new particles, forces, or measurements are postulated.

free parameters (7)
  • tau_l (lower overlap IoU threshold, Eq. 1) = 0.1 (stated in text)
    Hand-set to define valid partial overlaps; no sensitivity analysis or data-driven fitting.
  • tau_h (upper overlap IoU threshold, Eq. 1) = 0.60 (stated in text)
    Hand-set to separate overlapping particles from duplicates; arbitrary.
  • tau_dup (duplicate IoU threshold, Eq. 2) = 0.9 (stated in text)
    Hand-set threshold for near-complete overlap; no independent justification.
  • gamma (merged-mask area ratio, Eq. 3) = 3 (stated in text)
    Defines what counts as a merged candidate; chosen by hand.
  • alpha, beta (penalty weights in Eq. 4) = not given
    Weights for duplicates and merged masks; values are not reported, so reward landscape is underspecified.
  • epsilon (small constant in Eq. 7) = not given
    Introduced to avoid division by zero; value not reported.
  • NSGA-II settings (population size, generations, mutation rate) = not given
    Optimization hyperparameters deferred to missing Supplementary; required to reproduce.
assumptions (3)
  • domain assumption SAM's AutoMaskGenerator output is a valid starting point for segmentation and the 13 hyperparameters in Table 1 are the only controls needed.
    The entire framework operates on these hyperparameters; Section III asserts this without comparing to prompt-based SAM variants or alternative decoders.
  • standard math NSGA-II converges to a useful Pareto front for the multi-objective reward problems.
    Invoked implicitly in Sections V and VI; no convergence diagnostics or repeated-run variance are shown.
  • ad hoc to paper Maximizing the hand-designed rewards (overlap counts, inverse area, mean area) corresponds to physics-relevant segmentation.
    This is the load-bearing assumption of the method; it is never validated against labeled ground truth. The morphology failure in Section V illustrates that reward design is not guaranteed to capture the desired semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAM$^{*}$: Task-Adaptive SAM with Physics-Guided Rewards." pith.science (2026). https://pith.science/paper/J2YJEXJZ

@misc{pith2026250907047,
  author       = {Pith},
  title        = {Pith review of: SAM$^*$: Task-Adaptive SAM with Physics-Guided Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2YJEXJZ}},
  note         = {Machine review of arXiv:2509.07047}
}
abstract

Image segmentation is a critical task in microscopy, essential for accurately analyzing and interpreting complex visual data. This task can be performed using custom models trained on domain-specific datasets, transfer learning from pre-trained models, or foundational models that offer broad applicability. However, foundational models often present a considerable number of non-transparent tuning parameters that require extensive manual optimization, limiting their usability for real-time streaming data analysis. Here, we introduce a reward function-based optimization to fine-tune foundational models and illustrate this approach for SAM (Segment Anything Model) framework by Meta. The reward functions can be constructed to represent the physics of the imaged system, including particle size distributions, geometries, and other criteria. By integrating a reward-driven optimization framework, we enhance SAM's adaptability and performance, leading to an optimized variant, SAM$^{*}$, that better aligns with the requirements of diverse segmentation tasks and particularly allows for real-time streaming data segmentation. We demonstrate the effectiveness of this approach in microscopy imaging, where precise segmentation is crucial for analyzing cellular structures, material interfaces, and nanoscale features.

Figures

Figures reproduced from arXiv: 2509.07047 by the authors.

Figure 3
Figure 3. Vanilla SAM segmentation. (a) Input image. (b) Automatic mask proposals from SAM (no input of prior knowledge or per-post-processing); (c), regions are uniquely color-coded and indexed. 349 masks were detected, overlapping particles is missed segmented. These hyperparameters jointly orchestrate a delicate balance between sensitivity and specificity. When tuned appropriately, they enable SAM to recover overlapping na… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages

  1. [8]

    segment anything

    which promotes masks that cover extended structures. By jointly optimizing ℱ! and ℱ", we construct a multi-objective reward landscape in which hyperparameters such as points_per_side (pps), points_per_batch (ppb), and pred_iou_thresh (pit) are tuned to reconcile two competing aims: (i) fine-grained discovery of nanoscale particles, and (ii) robust coverag...

    work page 2023

  2. [9]

    Learning a universal template for few-shot dataset generalization

    (33) Triantafillou, E.; Larochelle, H.; Zemel, R.; Dumoulin, V. Learning a universal template for few-shot dataset generalization. In International conference on machine learning, 2021; PMLR: pp 10424-10433. (34) Archit, A.; Freckmann, L.; Nair, S.; Khalid, N.; Hilt, P.; Rajashekar, V.; Freitag, M.; Teuber, C.; Spitzner, M.; Tapia Contreras, C. Segment an...

    work page 2021

  3. [25]

    When does self-supervision improve few-shot learning? In European conference on computer vision, 2020; Springer: pp 645-666

    (31) Su, J.-C.; Maji, S.; Hariharan, B. When does self-supervision improve few-shot learning? In European conference on computer vision, 2020; Springer: pp 645-666. (32) Parnami, A.; Lee, M. Learning from few examples: A summary of approaches to few-shot learning. arXiv preprint arXiv:2203.04291

    arXiv 2020

  4. [30]

    A threshold selection method from gray-level histograms

    (10) Otsu, N. A threshold selection method from gray-level histograms. Automatica 1975, 11 (285-296), 23-27. (11) Roerdink, J. B.; Meijster, A. The watershed transform: Definitions, algorithms and parallelization strategies. Fundamenta informaticae 2000, 41, 187-228. (12) Sinaga, K. P.; Yang, M.-S. Unsupervised K-means clustering algorithm. IEEE access 20...

    arXiv 1975

  5. [187]

    A comprehensive survey on transfer learning

    (27) Zhuang, F.; Qi, Z.; Duan, K.; Xi, D.; Zhu, Y.; Zhu, H.; Xiong, H.; He, Q. A comprehensive survey on transfer learning. Proceedings of the IEEE 2020, 109 (1), 43-76. (28) Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y. Segment anything. In Proceedings of the IEEE/CVF internationa...

    work page 2020

  6. [1590]

    Unsupervised template learning for fine-grained object recognition

    (30) Yang, S.; Bo, L.; Wang, J.; Shapiro, L. Unsupervised template learning for fine-grained object recognition. Advances in neural information processing systems 2012,

    work page 2012

  7. [2016]

    V.; Balke, N.; Kalinin, S

    (3) Borodinov, N.; Tsai, W.-Y.; Korolkov, V. V.; Balke, N.; Kalinin, S. V.; Ovchinnikova, O. S. Machine learning-based multidomain processing for texture-based image segmentation and analysis. Applied Physics Letters 2020, 116 (4). (4) Minaee, S.; Boykov, Y.; Porikli, F.; Plaza, A.; Kehtarnavaz, N.; Terzopoulos, D. Image segmentation using deep learning: ...

    work page 2020

  8. [2022]

    D.; Hagopian, N.; Jacobs, R.; Morgan, D.; Voyles, P

    (20) Wei, J.; Moore, K.; Bammes, B.; Levin, B. D.; Hagopian, N.; Jacobs, R.; Morgan, D.; Voyles, P. M. Deep Learning Approach for High-accuracy Electron Counting of Monolithic Active Pixel Sensor-type Direct Electron Detectors at Increased Electron Dose. Microscopy and Microanalysis 2023, 29 (6), 2026-2036. (21) Ziatdinov, M.; Ghosh, A.; Wong, C. Y.; Kali...

    work page 2023

Show all 9 references
  1. [5386]

    E.; Wang, L.; Du, Y.; Spurgeon, S

    (26) Akers, S.; Kautz, E.; Trevino-Gavito, A.; Olszta, M.; Matthews, B. E.; Wang, L.; Du, Y.; Spurgeon, S. R. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Computational Materials 2021, 7 (1),

    2021

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.