Pith. sign in

REVIEW 1 major objections 14 references

SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation

T0 review · 1 major / 0 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SUPREME distributes image unlearning evaluation across multiple GPUs to support more seeds.

desk verdict SUPREME gives a practical registry and multi-GPU setup for scaling unlearning seed runs, but the abstract shows no equivalence checks between multi-GPU and single-GPU outputs on the metrics. read the letter →

arxiv 2606.00380 v1 pith:ZNTH7B64 submitted 2026-05-29 cs.CV cs.AI

Petros Andreou , Jamie Lanyon , Axel Finke , Georgina Cosma This is my paper
classification cs.CVcs.AI
keywords machineunlearningmulti-GPUimageclassificationreproducibilityevaluationframeworkfacerecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents SUPREME as an open-source framework that spreads the computationally heavy stages of machine unlearning evaluation over several GPUs. Unlearning methods must be tested by repeating full training, removal, and assessment cycles across many random seeds, a process that single-GPU tools make impractically slow. SUPREME supplies a registry to register new methods, metrics, models, and scenarios, plus a multi-GPU design that works with different accelerators and numeric precisions. The authors demonstrate the system on the Pins Face Recognition task with ResNet18 and ViT models, running both full-class and random-sample unlearning across ten seeds.

What carries the argument

SUPREME's registry-based design and multi-GPU architecture that register extensible components and distribute the training-unlearning-evaluation pipeline.

What would settle it

A controlled comparison showing statistically significant differences in final unlearning metrics or model accuracy between otherwise identical single-GPU and multi-GPU runs.

Watch

Extended reading notes

Core claim

SUPREME is a framework whose registry-based design lets users add new unlearning components and whose multi-GPU architecture distributes training, unlearning, and evaluation steps, enabling reproducible runs on multiple accelerators and precision modes as illustrated by experiments on Pins Face Recognition using ResNet18 and ViT under full-class and random-sample unlearning across ten seeds.

Load-bearing premise

That splitting the pipeline across multiple GPUs and precision settings yields the same numerical outcomes and reproducibility as single-GPU execution.

Editorial extensions

If this is right

  • More seeds can be evaluated in the same wall-clock time for any given unlearning method.
  • New unlearning methods, metrics, models, and scenarios can be added through the registry without altering core code.
  • Experiments can run on combinations of accelerators and mixed precision modes while preserving the same workflow.
  • Reproducible benchmarks become feasible for both full-class and random-sample unlearning on image tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same distribution pattern could be applied to unlearning evaluation in domains beyond image classification.
  • Widespread use might create de-facto standard test suites once many methods are registered in one place.
  • Researchers could test whether the multi-GPU speed-up changes which unlearning methods appear strongest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The manuscript introduces SUPREME, an open-source multi-GPU framework for evaluating image unlearning methods. It contributes (1) a registry-based design for adding methods, metrics, models, and scenarios, (2) a multi-GPU architecture supporting multiple accelerators and precision modes, and (3) a demonstration on the Pins Face Recognition dataset using ResNet18 and ViT under full-class and random-sample unlearning across ten seeds. The framework is available at the provided GitHub link.

Significance. If the multi-GPU implementation is validated to produce numerically equivalent results to single-GPU runs, the framework would address a practical bottleneck in unlearning research by enabling faster, reproducible multi-seed evaluations. The registry design and open-source release are explicit strengths that support extensibility and community use.

major comments (1)
  1. [Abstract] Abstract: The central claim that the multi-GPU architecture enables reproducible evaluation rests on the unstated assumption that distributed training, unlearning, and evaluation across accelerators and precision modes yields results equivalent to single-GPU runs. No quantitative checks, ablations, or comparisons (e.g., on membership-inference scores or post-unlearning accuracy) are reported to confirm the absence of synchronization artifacts or precision-induced shifts.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the importance of validating numerical equivalence between single-GPU and multi-GPU executions. We address this point directly below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that the multi-GPU architecture enables reproducible evaluation rests on the unstated assumption that distributed training, unlearning, and evaluation across accelerators and precision modes yields results equivalent to single-GPU runs. No quantitative checks, ablations, or comparisons (e.g., on membership-inference scores or post-unlearning accuracy) are reported to confirm the absence of synchronization artifacts or precision-induced shifts.

    Authors: We agree that the manuscript would be strengthened by explicit quantitative validation of equivalence. In the revised version we will add a dedicated subsection (under Experiments) that reports side-by-side results for the same ten seeds on the Pins Face Recognition dataset: post-unlearning accuracy and membership-inference scores obtained with (i) single-GPU runs and (ii) the multi-GPU configuration used in the paper. We will also state the floating-point tolerance thresholds and confirm that PyTorch DistributedDataParallel with fixed seeds was employed to minimize non-determinism. These additions will directly address the concern about synchronization artifacts and precision-induced shifts. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: software framework paper with no derivations or predictions

full rationale

The paper introduces an open-source multi-GPU framework for unlearning evaluation. It describes architecture (registry design, multi-GPU support) and reports a demonstration run, but contains no equations, fitted parameters, predictions, or derivation chains. No self-citations are used to justify load-bearing claims about uniqueness or results. The central contribution is the software itself; reproducibility claims rest on the described design and seed-wise runs rather than any self-referential reduction. This matches the default expectation of no circularity for non-derivational papers.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is a software framework contribution rather than a theoretical derivation. The abstract introduces no free parameters, mathematical axioms, or new postulated entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation." pith.science (2026). https://pith.science/paper/ZNTH7B64

@misc{pith2026260600380,
  author       = {Pith},
  title        = {Pith review of: SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNTH7B64}},
  note         = {Machine review of arXiv:2606.00380}
}
read the original abstract

Machine unlearning removes the influence of specific training data from a trained model without retraining it from scratch. Evaluating an unlearning method requires repeating training, unlearning, and evaluation across multiple seeds, which is computationally expensive. To our knowledge, existing image classification unlearning frameworks run on a single GPU, which limits how many seeds can be evaluated in reasonable time. We introduce SUPREME, an open-source framework that distributes these stages across multiple GPUs. SUPREME makes three contributions: a registry-based design for adding new methods, metrics, models, and scenarios; a multi-GPU architecture supporting multiple accelerators and precision modes; and a demonstration on Pins Face Recognition using ResNet18 and ViT under full-class and random-sample unlearning across ten seeds. The framework is available at https://github.com/pedroandreou/supreme-unlearning.

Figures

Figures reproduced from arXiv: 2606.00380 by the authors.

Figure 1
Figure 1. SUPREME’s seeded multi-stage pipeline. All three stages run across P devices. Gradient Synchronisation (Stages 1–2) averages gradients across devices after each backward pass so all devices apply the same parameter update. Result Aggregation (Stage 3), implemented as an all-gather, collects per-batch metric values from all devices and averages them into the final metric value. The training forget/retain split is com… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages

  1. [1]

    Kaggle (2020),https://www.kaggle.com/datasets/ hereisburak/pins-face-recognition

    Burak: Pins Face Recognition. Kaggle (2020),https://www.kaggle.com/datasets/ hereisburak/pins-face-recognition

    2020

  2. [2]

    In: Proceedings of the 10th IEEE European Symposium on Security and Privacy (EuroS&P)

    Cadet, X.F., Borovykh, A., Malekzadeh, M., Ahmadi-Abhari, S., Haddadi, H.: Deep unlearn: Benchmarking machine unlearning for image classification. In: Proceedings of the 10th IEEE European Symposium on Security and Privacy (EuroS&P). pp. 939–962 (2025)

    2025

  3. [3]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Chundawat, V.S., Tarun, A.K., Mandal, M., Kankanhalli, M.: Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 7210–7217 (2023)

    2023

  4. [4]

    In: Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI- 25)

    D’Angelo, A., Savelli, C., Tagliente, G., Giobergia, F., Baralis, E., Stilo, G.: How to make reproducible research in machine unlearning with ERASURE. In: Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI- 25). pp. 11025–11029 (2025)

    2025

  5. [5]

    In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    Dorna, V., Mekala, A., Zhao, W., McCallum, A., Lipton, Z.C., Kolter, J.Z., Maini, P.: OpenUnlearning: Accelerating LLM unlearning via unified benchmarking of methods and metrics. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    2025

  6. [6]

    In: International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)

    2021

  7. [7]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Foster, J., Schoepf, S., Brintrup, A.: Fast Machine Unlearning without Retraining through Selective Synaptic Dampening. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 12043–12051 (2024)

    2024

  8. [8]

    In: The Second Tiny Papers Track at ICLR 2024 (2024)

    Foster, J., Schoepf, S., Brintrup, A.: Loss-free machine unlearning. In: The Second Tiny Papers Track at ICLR 2024 (2024)

    2024

Show all 14 references
  1. [9]

    In: European Conference on Computer Vision

    Golatkar, A., Achille, A., Soatto, S.: Forgetting Outside the Box: Scrubbing deep networks of information accessible from input-output observations. In: European Conference on Computer Vision. pp. 383–398 (2020)

    2020

  2. [10]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)

    2016

  3. [11]

    arXiv preprint arXiv:2510.26714 (2026)

    Lanyon, J., Finke, A., Andreou, P., Cosma, G.: On the importance of multiple training seeds for evaluating machine unlearning. arXiv preprint arXiv:2510.26714 (2026)

    2026 arXiv

  4. [12]

    In: Proceedings of the 30th ACM Symposium on Access Control Models and Technologies (SACMAT)

    Li, X., Wei, W., Thuraisingham, B.: MUBox: A critical evaluation framework of deep machine unlearning. In: Proceedings of the 30th ACM Symposium on Access Control Models and Technologies (SACMAT). pp. 175–188 (2025)

    2025

  5. [13]

    IEEE Transactions on Neural Networks and Learning Systems 35(9), 13046–13055 (2023) SUPREME: A Framework for Image Unlearning Evaluation 9

    Tarun, A.K., Chundawat, V.S., Mandal, M., Kankanhalli, M.: Fast Yet Effective Machine Unlearning. IEEE Transactions on Neural Networks and Learning Systems 35(9), 13046–13055 (2023) SUPREME: A Framework for Image Unlearning Evaluation 9

    2023

  6. [14]

    IEEE Transactions on Emerging Topics in Computational Intelligence8(3), 2150– 2168 (2024) 10 Andreou et al

    Xu, J., Wu, Z., Wang, C., Jia, X.: Machine Unlearning: Solutions and Challenges. IEEE Transactions on Emerging Topics in Computational Intelligence8(3), 2150– 2168 (2024) 10 Andreou et al. A SUPREME Component Registry T able 2.Components supported by SUPREME. Each item in the ...

    2024

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.