Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification

Haiwei Lin; Hitoshi Kiya; Shoko Imaizumi

REVIEW 3 major objections 4 minor 13 references

Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Keeping the patch embedding trainable during low-rank adaptation lets a vision transformer reach 96.35% accuracy on encrypted CIFAR-10 with 0.71M trainable parameters, slightly above full fine-tuning at 82.56M parameters.

desk verdict A useful empirical observation about unfreezing patch embeddings for encrypted ViTs, but the mechanism claim needs seeds and an ablation to hold up. read the letter →

arxiv 2507.11943 v1 pith:BLAOJ3LP submitted 2025-07-16 cs.CR cs.AIcs.CV

Haiwei Lin , Shoko Imaizumi , Hitoshi Kiya This is my paper · ORCID

classification cs.CRcs.AIcs.CV

keywords privacy-preservingimageclassificationvisiontransformerlow-rankadaptationpatchembeddingblock-wiseencryptionparameter-efficientfine-tuningCIFAR-10MeLo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that low-rank fine-tuning of a vision transformer fails on privacy-preserving encrypted images for a specific reason: the patch embedding is frozen. The proposed method keeps MeLo's low-rank injections into query and value projections but unfreezes the patch embedding, giving the model a small set of parameters that can adapt to the fixed pixel-shuffling pattern of block-wise encryption. On encrypted CIFAR-10, the method reaches 96.35% accuracy with 0.71M trainable parameters, above full fine-tuning's 96.16% with 82.56M parameters and far above MeLo's 90.05% with 0.15M. The payoff, if the claim holds, is that private image classification no longer requires full fine-tuning to maintain accuracy.

What carries the argument

The central object is the patch embedding of a pretrained ViT, plus the MeLo-style low-rank update matrices. The update mechanism is $Q' = Q + \frac{\alpha}{r} W_B^Q W_A^Q X$ and $V' = V + \frac{\alpha}{r} W_B^V W_A^V X$, with $\alpha = 4$ and rank $r = 8$; the patch embedding is the only additional layer left unfrozen. This gives the model a small set of parameters that can adapt to the fixed pixel-permutation pattern introduced by block-wise encryption.

What would settle it

Run the proposed method on encrypted CIFAR-10 with the patch embedding frozen but the positional embedding or the first transformer block unfrozen instead: if accuracy remains near MeLo's 90.05%, the improvement is not attributable to the patch embedding. Repeating the full comparison over at least five random seeds would also show whether 96.35% versus 96.16% is outside run-to-run noise.

Watch

Extended reading notes

Core claim

The central discovery is that unfreezing the patch embedding, and only that layer beyond MeLo's low-rank matrices, restores most of the accuracy lost when training on block-wise encrypted images. The paper reports 96.35% accuracy on encrypted CIFAR-10 with 0.71M trainable parameters, against 96.16% for full fine-tuning and 90.05% for MeLo, and interprets the comparison as evidence that the frozen patch embedding was the bottleneck created by pixel-shuffling encryption.

Load-bearing premise

The paper assumes that the patch embedding is the bottleneck that needs adaptation for encrypted images; because no ablation isolates this layer and no repeated runs are reported, the small gain over full fine-tuning is not separated from random variation.

Editorial extensions

If this is right

A privacy-preserving classifier can be fine-tuned with 0.71M trainable parameters instead of 82.56M while keeping encrypted-image accuracy at or above full fine-tuning.
The 90.05% accuracy of MeLo on encrypted CIFAR-10 is attributed to its frozen patch embedding; unfreezing this layer is what closes most of the gap.
The method uses the same encryption key for training and testing, so a single secret key suffices for the whole pipeline.
On plain images the method is slightly below MeLo (97.98% versus 98.36%), so the patch-embedding update mainly buys robustness to encryption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves untested whether other permutation-based encryptions that respect patch boundaries behave the same way; if the patch-embedding story is right, unfreezing this layer should recover accuracy for block scrambling and channel-shuffling variants too.
The reported 0.19-point edge over full fine-tuning is within the range of run-to-run variation, so a multi-seed replication is the natural next check before treating the edge as real.
Because the model is trained with one fixed secret key, the patch embedding may partially invert the pixel permutation; that raises the question of how much of the key could be inferred from the trained weights, which the paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a short, competent empirical note that makes one useful observation — MeLo's frozen patch embedding hurts on encrypted images, and unfreezing it recovers most of the gap to full fine-tuning. The reported numbers (96.35% vs 90.05% for MeLo, 96.16% for full fine-tuning) are striking, and the idea is simple enough to be actionable.

What's genuinely new: the specific finding that the patch embedding is the bottleneck for block-wise encrypted inputs is not in the prior work it cites, and it is a legitimate, if incremental, extension of MeLo. The experiment is clean: one dataset (CIFAR-10), one model (ViT-B/16 pretrained on ImageNet), standard block-wise encryption with a key-consistent shuffle. The paper is honest in framing; it does not claim a new mechanism, just a change in what gets unfrozen.

Where it goes soft: the evidence is thin in three places. First, there is a single run per configuration, no seeds, no error bars. That makes the 0.19-point edge over full fine-tuning meaningless, and it makes it hard to trust the stability of the 96.35 figure. Second, the method changes two things at once — unfreezing the patch embedding and adding roughly 0.56M trainable parameters (0.71M vs 0.15M). Without an ablation that separates those, the causal story "patch embedding is the key" is not established; it could be just more capacity. Third, the paper skips a direct comparison with [13], the same group's earlier domain-adaptation approach for encrypted ViTs. That reference is close enough that its absence weakens the novelty claim and leaves a baseline question open.

None of this makes the central result wrong. The gap over MeLo is large enough that the finding is probably real, and the paper is right that a parameter-efficient recipe with this accuracy is practically useful. But the mechanism is underdetermined, and the paper needs a round of revision with seeds, an ablation, and a [13] comparison to be convincing.

Verdict: worth a serious referee, but conditional on those additions. I'd bring it to reading group as a short example of how to (and how not to) establish a mechanism claim in a small empirical study.

Referee Report

3 major / 4 minor

Summary. This paper proposes a parameter-efficient fine-tuning method for vision transformers (ViTs) on block-wise encrypted images. The method combines MeLo-style low-rank adaptation with an unfrozen patch embedding layer. In experiments on CIFAR-10 with a ViT pretrained on ImageNet-1K, the method achieves 96.35% accuracy on encrypted images with 0.71M trainable parameters, compared with 96.16% for full fine-tuning (82.56M parameters) and 90.05% for MeLo (0.15M parameters). The authors conclude that unfreezing the patch embedding is key to adapting ViTs to encrypted inputs.

Significance. If the result is robust, the method would be a practically useful compromise between accuracy and parameter efficiency for privacy-preserving image classification, and the observation that the patch embedding is sensitive to pixel-level perturbation is mechanistically plausible. However, the evidence presented is too thin to establish the central causal claim: a single dataset, a single model, one run per condition, no ablations, and no comparison with the closest prior work on encrypted-domain ViT adaptation. The paper also provides no code release or machine-checked derivations; its contribution is purely empirical and needs stronger experimental support.

major comments (3)

[Table I, Section III] Table I reports a single run for each condition, with no standard deviation, confidence interval, or number of seeds. The 0.19 percentage point gap between the proposed method (96.35%) and full fine-tuning (96.16%) is smaller than the run-to-run variation typically observed in ViT fine-tuning on CIFAR-10. Without repeated-seed experiments, the claim that the proposed method 'outperformed full fine-tuning' is not statistically supported.
[Section II-C, Table I] The proposed method differs from MeLo in two ways at once: it unfreezes the patch embedding and increases the trainable parameter count from 0.15M to 0.71M. Because no ablation isolates the patch-embedding unfreezing (e.g., MeLo with only the patch embedding unfrozen, or a variant with the patch embedding frozen and a comparable parameter count), the performance gain cannot be causally attributed to the patch embedding rather than to additional capacity.
[Section III] The experiments do not compare the proposed method with the domain-adaptation approach of [13], which also targets fine-tuning of ViTs on encrypted images. Without this baseline, the paper does not establish that unfreezing the patch embedding is the decisive improvement over existing encrypted-data fine-tuning techniques.

minor comments (4)

[Section II-C] Equations (1) and (2) and the surrounding text contain garbled notation for the low-rank matrices (e.g., 'W Q B and W Q B' should be distinct matrices for Q and V). Please correct the subscripts and superscripts.
[Throughout] The phrase 'full-time tuning' appears repeatedly and should be replaced with 'full-parameter tuning' or 'full fine-tuning'.
[Section III] The paper does not state the training hyperparameters (batch size, optimizer settings beyond the learning rate, learning-rate schedule, number of warmup steps, or data augmentation), which are needed for reproducibility.
[Figure 2] Figure 2 does not clearly indicate that the positional embedding remains frozen; the text should explicitly state which components are frozen and which are trainable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical fine-tuning comparison with externally chosen hyperparameters and no self-referential reduction.

full rationale

The paper's central claim is empirical: unfreezing the patch embedding while applying MeLo low-rank adaptation improves accuracy on block-wise encrypted CIFAR-10 (96.35% vs. 96.16% for full fine-tuning and 90.05% for MeLo) with far fewer trainable parameters. This is not derived from any fitted parameter or self-referential normalization. The hyperparameters (alpha=4, r=8) are stated as MeLo configuration defaults, not chosen to match the reported accuracy. No equation in the paper defines the reported accuracy in terms of a fitted quantity, and no uniqueness theorem or load-bearing self-citation is invoked. The only self-related reference, [13], is used to describe the encryption procedure ('The above procedure is carried out in the same manner as in [13]'), which is a methodological borrowing rather than a justificatory premise for the accuracy claim. Concerns about statistical robustness (single run, no seed variance, no ablation isolating the patch embedding) are about evidence quality, not circularity. Therefore there is no circular step, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method relies on two domain assumptions about the encryption scheme and about the role of the patch embedding; both are taken from prior work or asserted without analysis. The only hand-chosen numbers are the LoRA scaling factor and rank, inherited from MeLo defaults.

free parameters (2)

LoRA scaling factor alpha = 4
Chosen following MeLo defaults; not tuned and no sensitivity analysis is given.
low-rank rank r = 8
Chosen as in MeLo; the trainable parameter count and accuracy depend on this value, but no ablation is performed.

assumptions (3)

domain assumption Block-wise permutation encryption with a fixed key preserves enough semantic structure for a pretrained ViT to adapt with small low-rank updates.
The entire method assumes encrypted images remain classifiable; no analysis of information preservation or permutation properties is given (Section II-A).
ad hoc to paper The patch embedding layer is the main bottleneck for encrypted inputs, so unfreezing only that layer suffices.
The paper claims unfreezing the patch embedding improves accuracy but provides no ablation or gradient analysis establishing this causal link (Section II-C, Table I).
standard math The low-rank decomposition of Q and V projections (Eq. 1 and 2) is a standard linear algebraic identity that preserves the forward pass structure.
The equations are a direct application of LoRA to ViT attention and are not the source of novelty or risk.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification." pith.science (2026). https://pith.science/paper/BLAOJ3LP

@misc{pith2026250711943,
  author       = {Pith},
  title        = {Pith review of: Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLAOJ3LP}},
  note         = {Machine review of arXiv:2507.11943}
}

read the original abstract

We propose a low-rank adaptation method for training privacy-preserving vision transformer (ViT) models that efficiently freezes pre-trained ViT model weights. In the proposed method, trainable rank decomposition matrices are injected into each layer of the ViT architecture, and moreover, the patch embedding layer is not frozen, unlike in the case of the conventional low-rank adaptation methods. The proposed method allows us not only to reduce the number of trainable parameters but to also maintain almost the same accuracy as that of full-time tuning.

Figures

Figures reproduced from arXiv: 2507.11943 by the authors.

**Figure 1.** Pipeline of block-wise image encryption II. PROPOSED METHOD A. Block-wise image encryption We use a block-wise image encryption method to protect sensitive visual information in training and test images. As shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Overview of proposed fine-tuning method. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Samples of plain images and their encrypted images from each class [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[13]

Domain adaptation for efficiently fine-tuning vision transformer with encrypted images,

T. Nagamori, S. Shiota, and H. Kiya, “Domain adaptation for efficiently fine-tuning vision transformer with encrypted images,” in 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , pp. 1677–1683, 2023

work page 2023
[1]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

arXiv 2010
[2]

Vision-language models for vision tasks: A survey,

J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

2024
[3]

Toward transformer-based object detection,

J. Beal, E. Kim, E. Tzeng, D. H. Park, A. Zhai, and D. Kislyuk, “Toward transformer-based object detection,” arXiv preprint arXiv:2012.09958 , 2020

arXiv 2012
[4]

Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xi- ang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6881–6890, June 2021

work page 2021
[5]

Lora: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

arXiv 2021
[6]

Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,

Y . Zhu, Z. Shen, Z. Zhao, S. Wang, X. Wang, X. Zhao, D. Shen, and Q. Wang, “Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,” in 2024 IEEE International Symposium on Biomedical Imaging (ISBI) , pp. 1–5, 2024

work page 2024
[7]

An overview of compressible and learnable image transformation with secret key and its applications,

H. Kiya, A. P. M. Maung, Y . Kinoshita, S. Imaizumi, S. Shiota, et al., “An overview of compressible and learnable image transformation with secret key and its applications,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2022

work page 2022

Show all 13 references

[8]

Grayscale-based block scrambling image encryption for social network- ing services,

W. Sirichotedumrong, T. Chuman, S. Imaizumi, and H. Kiya, “Grayscale-based block scrambling image encryption for social network- ing services,” in 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, 2018

2018
[9]

Privacy-preserving content-based image retrieval using compressible encrypted images,

K. Iida and H. Kiya, “Privacy-preserving content-based image retrieval using compressible encrypted images,” IEEE Access, vol. 8, pp. 200038– 200050, 2020

2020
[10]

A protection method of trained cnn model with a secret key from unauthorized access,

A. MaungMaung and H. Kiya, “A protection method of trained cnn model with a secret key from unauthorized access,” APSIPA Transac- tions on Signal and Information Processing , vol. 10, p. e10, 2021

2021
[11]

Encryption inspired adversarial defense for visual classification,

M. Maung, A. Pyone, and H. Kiya, “Encryption inspired adversarial defense for visual classification,” in 2020 IEEE International Conference on Image Processing (ICIP) , pp. 1681–1685, 2020

2020
[12]

A random ensemble of encrypted vision transformers for adversarially robust defense,

R. Iijima, S. Shiota, and H. Kiya, “A random ensemble of encrypted vision transformers for adversarially robust defense,” IEEE Access , vol. 12, pp. 69206–69216, 2024

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [13]

Domain adaptation for efficiently fine-tuning vision transformer with encrypted images,

T. Nagamori, S. Shiota, and H. Kiya, “Domain adaptation for efficiently fine-tuning vision transformer with encrypted images,” in 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , pp. 1677–1683, 2023

work page 2023

[2] [1]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

arXiv 2010

[3] [2]

Vision-language models for vision tasks: A survey,

J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

2024

[4] [3]

Toward transformer-based object detection,

J. Beal, E. Kim, E. Tzeng, D. H. Park, A. Zhai, and D. Kislyuk, “Toward transformer-based object detection,” arXiv preprint arXiv:2012.09958 , 2020

arXiv 2012

[5] [4]

Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xi- ang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6881–6890, June 2021

work page 2021

[6] [5]

Lora: Low-rank adaptation of large language models,

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

arXiv 2021

[7] [6]

Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,

Y . Zhu, Z. Shen, Z. Zhao, S. Wang, X. Wang, X. Zhao, D. Shen, and Q. Wang, “Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,” in 2024 IEEE International Symposium on Biomedical Imaging (ISBI) , pp. 1–5, 2024

work page 2024

[8] [7]

An overview of compressible and learnable image transformation with secret key and its applications,

H. Kiya, A. P. M. Maung, Y . Kinoshita, S. Imaizumi, S. Shiota, et al., “An overview of compressible and learnable image transformation with secret key and its applications,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, 2022

work page 2022