ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

Anjith George; Francois Poh; S\'ebastien Marcel

REVIEW 3 major objections 3 minor 15 references

ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sitter identification in historical paintings improves when a CLIP foundation model is adapted with LoRA and fused with a conventional face recognition network.

desk verdict A competent, incremental model-adaptation study for historical portrait identification, but the 'state-of-the-art' claim is unverified because it never reports prior published numbers on the same benchmark. read the letter →

arxiv 2508.20626 v1 pith:6JGKWK7Y submitted 2025-08-28 cs.CV cs.AI

Francois Poh , Anjith George , Sébastien Marcel This is my paper

classification cs.CVcs.AI

keywords facerecognitionhistoricalportraitssitteridentificationCLIPLoRAembeddingfusiondomainadaptationIResNet100

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a vision-language foundation model, adapted cheaply, can improve face identification in historical paintings, a domain where conventional face recognition networks degrade because of artistic style, sitter age, and medium shifts. It shows that fine-tuning CLIP with low-rank adapters on a small portrait dataset and then fusing its embeddings with those of a conventional face recognition network—using both the original and a lightly tuned version—cuts equal error rate on the Historical Faces test set from 14.0% (the face network alone) to 9.9%. The same fusion raises the true-accept rate from 55.1% to 65.9% at 1% false-accept rate, and the paper describes this as state-of-the-art sitter identification. The significance is practical: a lightweight recipe for adapting general-purpose models to a data-scarce cultural-heritage task, without retraining a full architecture.

What carries the argument

The load-bearing mechanism is embedding fusion by normalized concatenation: CLIP-LoRA embeddings, tuned IResNet100 embeddings, and base IResNet100 embeddings are each L2-normalized, concatenated, re-normalized, and scored by cosine similarity. The CLIP adaptation inserts LoRA adapters of rank 16 into the query and value matrices of all transformer layers and trains with triplet loss plus hard-negative mining; the face network only tunes its final linear layer. The mechanism works because the two model families produce complementary representations—style and contextual information from CLIP, identity and geometric structure from the face network—and adding the tuned and untuned versions of th

What would settle it

Run the previously published confidence-aware sitter-identification method on the same Historical Faces test split with the same alignment, crop, and scoring protocol. If its equal error rate is below 9.9% or its true-accept rate at 1% false-accept is above 65.9%, the paper's state-of-the-art claim fails. A second check: remove the CLIP-LoRA embeddings from the fusion; if accuracy does not drop, the foundation-model contribution is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the gap between photographic face recognition and painted portraits can be substantially closed by adapting a CLIP ViT-B/16 model with LoRA on portrait images and combining its normalized embeddings with those of an IResNet100 face recognition network in a simple concatenation-and-re-normalization fusion. On the Historical Faces dataset (766 paintings, 210 sitters, identity-disjoint 60/20/20 split), the fused system reaches an equal error rate of 9.9%, a true-accept rate of 39.7% at 0.1% false-accept rate, and 65.9% at 1% false-accept rate. This beats every configuration the paper tests, including a commercial off-the-shelf face recognizer (12.6% EER), the untuned f

Load-bearing premise

That no previously published method performs better on the same Historical Faces test split; the paper never measures a prior method under its own protocol, so the claim of state-of-the-art rests on an untested comparison.

Editorial extensions

If this is right

A parameter-efficient fine-tune (LoRA on attention matrices, a linear head) is sufficient to move a foundation model into a specialized visual domain; no full-network retraining is needed.
Fusing the tuned and untuned versions of the same face network improves low-FAR performance, so representation diversity matters even within one architecture.
Combining vision-language and face-specific embeddings gives consistent gains across both EER and low-false-accept TAR, offering a recipe for other fine-grained recognition tasks under domain shift.
Because the split is identity-disjoint and the dataset is public, the reported numbers provide a direct benchmark for future sitter-identification methods.
At 65.9% TAR at 1% FAR, the method is usable for candidate shortlisting by art historians, even though it is not a final decision tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's 'state-of-the-art' wording is only as strong as the unstated premise that no previously published method beats 9.9% EER on the same test split; the paper does not report prior methods under its own protocol, so a direct re-run of the earlier confidence-aware method would be the deciding check.
The same CLIP-LoRA plus fusion recipe likely transfers to other style-dominated identity tasks such as caricatures, sketches, or engravings; this is an extension the paper does not test.
With only 766 paintings for 210 sitters, per-identity data is very thin; the fusion's advantage may grow if synthetic portraits are used to augment training, as the paper itself suggests in its outlook.
The failure cases where copies of earlier portraits yield false accepts suggest an explicit 'compositional copy' prior could reduce impostor scores; this is a speculative extension beyond the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: the paper is a decent engineering study—LoRA-tune CLIP, lightly adapt IResNet100, concatenate normalized embeddings, and you get better EER/TAR on the Historical Faces dataset than the base models or an unnamed commercial system. The problem is the marketing: the abstract, intro, and conclusion all claim state-of-the-art and outperformance of prior approaches, yet the paper never reports the performance of any published method on this dataset. Huber et al. [8], who introduced the benchmark, is cited but its EER/TAR is never given. So the central comparative claim is an untested premise.

What is actually new: the specific result of combining a LoRA-fine-tuned CLIP with base and tuned IResNet100 embeddings via normalized concatenation, and the observation that this helps at low FAR on paintings. The components are all in the literature (Sony et al. for CLIP+FR fusion, Huber et al. for domain adaptation and confidence-aware recognition), and the paper credits them. That's fine; incremental is not a sin. The internal experiments are methodologically sound: identity-disjoint split, clear training details, hard-negative mining specified, and the comparison across base/tuned/fusion is internally consistent. The visualizations of successful and failed pairs are a nice touch.

Soft spots: (1) The missing comparison to prior published work is load-bearing, not minor. Without it, 'state-of-the-art' is unsupported. We don't know whether Huber et al. gets EER lower than 9.9% or higher TAR at low FAR. The paper needs to add those numbers under a comparable protocol, or drop the claim. (2) No error bars or multiple splits. With 210 sitters and 766 images, the standard error on EER could be several percent. A single train/test split leaves the reported margins fragile. (3) The COTS system is unnamed, which makes the external comparison weak. (4) Minor: Table 3 has a formatting glitch (the '62.15%' row), but that's trivial.

Who gets value: researchers working on facial recognition across artistic domains, and possibly art-historical applications. It's not going to reshape face recognition, but it's a useful data point that foundation models can complement domain-specific networks in this niche.

Recommendation: Send it to peer review, because the internal study is honest and reproducible enough to deserve referee time, but the authors must be pushed to supply the missing prior-art comparisons and ideally confidence intervals. If they can't, the state-of-the-art claim should be removed.

Referee Report

3 major / 3 minor

Summary. The paper addresses sitter identification in historical portrait paintings. It proposes adapting a CLIP ViT-B/16 foundation model via LoRA with triplet loss and hard-negative mining, fine-tuning the final linear layer of an AntelopeV2/IResNet100 face recognition model, and fusing the normalized embeddings by concatenation followed by re-normalization. Experiments on the Historical Faces dataset use an identity-disjoint 60/20/20 split and report EER, TAR@0.1%FAR, and TAR@1%FAR for base/tuned/fused variants, a COTS system, and the proposed three-way fusion. The best result is EER 9.9%, TAR@0.1%FAR 39.7%, and TAR@1%FAR 65.9% for fusing CLIP-LoRA embeddings with both base and tuned IResNet100 embeddings. The authors claim this is state-of-the-art and that it outperforms prior approaches.

Significance. If the comparative claim were established, the paper would make a useful contribution: it demonstrates that lightweight parameter-efficient adaptation of a vision-language foundation model can complement a conventional face recognition network in a low-data art domain, and the proposed fusion is simple and appears to give consistent internal gains. The internal evaluation is methodologically reasonable: the split is identity-disjoint, training details are specified, and results are reported on a held-out test set. However, the central 'state-of-the-art' claim is not verified: no prior published method is evaluated on the same benchmark, no code or trained models are released, and no uncertainty estimates are provided. The practical value therefore hinges on an external comparison that is currently missing.

major comments (3)

[Section 4, Tables 1–3] The central claim that the fusion approach 'achieves state-of-the-art accuracy in sitter identification' and 'outperforms prior approaches' is not supported by any comparison to prior published methods on the Historical Faces benchmark. Huber et al. [8], who introduced the dataset and proposed confidence-aware face recognition, is cited but its EER/TAR are never reported. Gupta et al. [5] and Salavessa et al. [11] are also not evaluated. The only external baseline is an unnamed 'COTS FR system'. Since the evaluation uses verification metrics on the same benchmark, the authors should reproduce prior published methods under an identical protocol or explicitly report the published numbers with a demonstrated protocol match. Without this, the state-of-the-art claim is unsupported and should be withdrawn or substantially qualified.
[Section 3, Tables 1–3] The reported differences are modest (e.g., EER improves from 12.6% for COTS to 9.9% for the proposed fusion), yet no confidence intervals, repeated splits, or multi-seed results are provided. The dataset contains only 766 images of 210 sitters, and a single 60:20:20 identity split is used. It is therefore unclear whether the observed gains are stable or within run-to-run variation. Given that the headline claim is comparative, the authors should provide repeated splits or bootstrap confidence intervals, or temper the quantitative claims accordingly.
[Section 3.1, Table 3] The evaluation protocol is underspecified. The paper does not state how verification pairs are constructed from the test identities, how many genuine and impostor comparisons are used, or whether the same alignment/cropping is applied as in Huber et al. [8]. It also does not explicitly define the final score beyond 'cosine similarity between the concatenated embeddings'. This information is necessary for reproducing the numbers and for any meaningful comparison with prior work. Please provide a precise protocol description.

minor comments (3)

[Section 3.1, Table 2] The text says the hard-negative-mining variant 'does not achieve higher TAR at all FAR levels' and 'shows improved performance above approximately 1% FAR'. The table shows the opposite pattern for the reported points: HN improves TAR@0.1%FAR (17.8 vs 16.8) and degrades TAR@1%FAR (43.5 vs 43.9). Please clarify the intended statement.
[Table 3] The COTS system is never identified. Please name the commercial model/version so that the baseline is meaningful and reproducible.
[Figure 2] It would help to state in the caption that the displayed scores are cosine similarities of the fused concatenated embeddings, and to clarify whether these are from the final model or from an intermediate variant.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all reported results are measured on a held-out test split with no fitted parameters tied to test labels.

full rationale

The paper's derivation chain is an empirical adaptation pipeline: it takes pre-trained CLIP ViT-B/16 and IResNet100 (AntelopeV2), fine-tunes them on the Historical Faces training split using triplet loss with hard negative mining, fuses normalized embeddings, and evaluates on the identity-disjoint test split (Section 3). No equation defines a target quantity in terms of itself; no parameter is fitted to test labels; early stopping and hyperparameters are validated on the validation split. The only potential concern is the unverified 'state-of-the-art' claim in the abstract and Section 4, because the paper does not report prior published methods' EER/TAR on the same benchmark. However, that is a missing comparison / correctness-risk issue, not circularity: the claimed result does not reduce by construction to an input, a fit, or a self-citation. The self-citations [3,4] appear only as suggested future research directions and are not load-bearing. Hence score 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a handful of manually chosen hyperparameters and the reliability of the public dataset. No new theoretical entities are introduced.

free parameters (6)

LoRA rank = 16
Chosen for efficiency, not derived; affects representation capacity of fine-tuned CLIP.
Triplet margin = 0.5
Hyperparameter for triplet loss; chosen by hand without reported sensitivity analysis.
Learning rate (Adam) = 1e-5
Optimization hyperparameter; no schedule or search reported.
Batch size = 48
Training hyperparameter; selected manually.
Hard negative sampling ratio = 30% from top 50, 70% from next 450
Training heuristic; chosen by hand.
Early stopping patience = 10 epochs
Regularization hyperparameter.

assumptions (3)

domain assumption The Historical Faces dataset labels are reliable ground truth for sitter identity.
The paper uses the dataset from Huber et al. [8] without questioning label quality.
domain assumption A single identity-disjoint 60/20/20 split is sufficient to estimate generalization.
All conclusions rest on one split with no repeated evaluation or significance tests.
domain assumption The unnamed COTS system is an appropriate representative of state-of-the-art.
The COTS baseline is treated as a strong comparison, yet its training data, architecture, and identity are unknown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ArtFace: Towards Historical Portrait Face Identification via Model Adaptation." pith.science (2026). https://pith.science/paper/6JGKWK7Y

@misc{pith2026250820626,
  author       = {Pith},
  title        = {Pith review of: ArtFace: Towards Historical Portrait Face Identification via Model Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JGKWK7Y}},
  note         = {Machine review of arXiv:2508.20626}
}

read the original abstract

Identifying sitters in historical paintings is a key task for art historians, offering insight into their lives and how they chose to be seen. However, the process is often subjective and limited by the lack of data and stylistic variations. Automated facial recognition is capable of handling challenging conditions and can assist, but while traditional facial recognition models perform well on photographs, they struggle with paintings due to domain shift and high intra-class variation. Artistic factors such as style, skill, intent, and influence from other works further complicate recognition. In this work, we investigate the potential of foundation models to improve facial recognition in artworks. By fine-tuning foundation models and integrating their embeddings with those from conventional facial recognition networks, we demonstrate notable improvements over current state-of-the-art methods. Our results show that foundation models can bridge the gap where traditional methods are ineffective. Paper page at https://www.idiap.ch/paper/artface/

Figures

Figures reproduced from arXiv: 2508.20626 by the authors.

**Figure 1.** Overview of the proposed method: (a) LoRA-based adaptation of the CLIP model, and (b) head adaptation using triplet loss. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Examples of successful and failed comparisons. Each pair shows the reference and probe images, with the associated cosine [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** ROC curves of tuned and base CLIP, IResNet100, COTS [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[8]

Verification of sitter identity across historical portrait paintings by confidence-aware face recognition

Marco Huber, Philipp Terh ¨orst, Anh Thi Luu, Florian Kirch- buchner, and Naser Damer. Verification of sitter identity across historical portrait paintings by confidence-aware face recognition. In 2022 26th International Conference on Pat- tern Recognition (ICPR), pages 938–944. IEEE, 2022. 1, 2

work page 2022
[5]

Deep learning based identity veri- fication in renaissance portraits

Akash Gupta, Niluthpol C Mithun, Conrad Rudolph, and Amit K Roy-Chowdhury. Deep learning based identity veri- fication in renaissance portraits. In 2018 IEEE International Conference on Multimedia and Expo (ICME) , pages 1–6. IEEE, 2018. 1

work page 2018
[11]

Proto-early renaissance depictions, iconographic analysis and computerised facial similarity as- sessment connections: The 16th century mural paintings of st

Eunice Salavessa, Jos ´e Aranha, Rafael Moreira, and David M Freire-Lista. Proto-early renaissance depictions, iconographic analysis and computerised facial similarity as- sessment connections: The 16th century mural paintings of st. leocadia church (chaves, north of portugal). Heritage, 7 (4):2031–2054, 2024. 1

work page
[1]

Why artists copy: A history of drawing the masters,

Art UK. Why artists copy: A history of drawing the masters,

work page
[2]

Arcface: Additive angular margin loss for deep face recognition

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 4690–4699, 2019. 1

work page 2019
[3]

Modality agnostic het- erogeneous face recognition with switch style modulators

Anjith George and S ´ebastien Marcel. Modality agnostic het- erogeneous face recognition with switch style modulators. In 2024 IEEE International Joint Conference on Biometrics (IJCB), pages 1–10. IEEE, 2024. 3

work page 2024
[4]

Digi2real: Bridging the realism gap in synthetic data face recognition via foun- dation models

Anjith George and S ´ebastien Marcel. Digi2real: Bridging the realism gap in synthetic data face recognition via foun- dation models. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1469–1478, 2025. 3

work page 2025
[6]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 1, 3

work page 2022

Show all 15 references

[7]

Labeled faces in the wild: A database forstudying face recognition in unconstrained environments

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real-Life’Images: detection, align- ment, and recognition, 2008. 1

2008
[9]

PEFT: State-of-the-art parameter-efficient fine-tuning meth- ods

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning meth- ods. https://github.com/huggingface/peft ,
[10]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

2021
[12]

Facenet: A unified embedding for face recognition and clus- tering

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015. 2

2015
[13]

Foundation versus domain-specific models: Perfor- mance comparison, fusion, and explainability in face recog- nition

Redwan Sony, Parisa Farmanifard, Arun Ross, and Anil K Jain. Foundation versus domain-specific models: Perfor- mance comparison, fusion, and explainability in face recog- nition. arXiv preprint arXiv:2507.03541, 2025. 1

2025 arXiv
[14]

Roy- Chowdhury

Ramya Srinivasan, Conrad Rudolph, and Amit K. Roy- Chowdhury. Computerized face recognition in renaissance portrait art: A quantitative measure for identifying uncertain subjects in ancient portraits. IEEE Signal Processing Maga- zine, 32(4):85–94, 2015. 1

2015
[2024]

Accessed: 2025-07-25. 1

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [8]

Verification of sitter identity across historical portrait paintings by confidence-aware face recognition

Marco Huber, Philipp Terh ¨orst, Anh Thi Luu, Florian Kirch- buchner, and Naser Damer. Verification of sitter identity across historical portrait paintings by confidence-aware face recognition. In 2022 26th International Conference on Pat- tern Recognition (ICPR), pages 938–944. IEEE, 2022. 1, 2

work page 2022

[2] [5]

Deep learning based identity veri- fication in renaissance portraits

Akash Gupta, Niluthpol C Mithun, Conrad Rudolph, and Amit K Roy-Chowdhury. Deep learning based identity veri- fication in renaissance portraits. In 2018 IEEE International Conference on Multimedia and Expo (ICME) , pages 1–6. IEEE, 2018. 1

work page 2018

[3] [11]

Proto-early renaissance depictions, iconographic analysis and computerised facial similarity as- sessment connections: The 16th century mural paintings of st

Eunice Salavessa, Jos ´e Aranha, Rafael Moreira, and David M Freire-Lista. Proto-early renaissance depictions, iconographic analysis and computerised facial similarity as- sessment connections: The 16th century mural paintings of st. leocadia church (chaves, north of portugal). Heritage, 7 (4):2031–2054, 2024. 1

work page

[4] [1]

Why artists copy: A history of drawing the masters,

Art UK. Why artists copy: A history of drawing the masters,

work page

[5] [2]

Arcface: Additive angular margin loss for deep face recognition

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 4690–4699, 2019. 1

work page 2019

[6] [3]

Modality agnostic het- erogeneous face recognition with switch style modulators

Anjith George and S ´ebastien Marcel. Modality agnostic het- erogeneous face recognition with switch style modulators. In 2024 IEEE International Joint Conference on Biometrics (IJCB), pages 1–10. IEEE, 2024. 3

work page 2024

[7] [4]

Digi2real: Bridging the realism gap in synthetic data face recognition via foun- dation models

Anjith George and S ´ebastien Marcel. Digi2real: Bridging the realism gap in synthetic data face recognition via foun- dation models. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1469–1478, 2025. 3

work page 2025

[8] [6]

Lora: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 1, 3

work page 2022