Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Human Re-ID Meets LVLMs: What can we expect?

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Specialized person re-identification still clearly outperforms four leading LVLMs, which often give near-identical scores to different candidates.

desk verdict The paper's central d' numbers don't compute from its own table, and the PersonViT-filtered subset biases the comparison; the qualitative material is the real contribution. read the letter →

arxiv 2501.18698 v1 pith:EB4XVF4F submitted 2025-01-30 cs.CV

Kailash Hambarde , Pranita Samale , Hugo Proença This is my paper · ORCID
classification cs.CV
keywords LargeVision-LanguageModelsHumanRe-identificationMarket1501PersonViTChatGPT-4oClaude-3.5Gemini-2.0Qwen-VL-Max
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models have been proposed as general-purpose visual reasoners, and this paper asks whether they can handle human re-identification, a surveillance task where the system must decide if two images show the same person. Using a curated subset of Market1501 with 20 queries and 400 gallery images, the authors compare ChatGPT-4o, Gemini-2.0-Flash, Claude 3.5 Sonnet, and Qwen-VL-Max against PersonViT, a specialized ReID model. The paper's central finding is that PersonViT separates genuine matches from impostors much more cleanly than any LVLM, while the LVLMs often produce nearly identical similarity scores across candidates, especially in batch mode, making standard retrieval metrics uncomputable and yielding what the authors call catastrophic answers. The authors conclude that current LVLMs are not ready to replace specialized ReID models, and suggest future work that fuses the two families so the reasoning and interpretability of LVLMs can be combined with the discriminative power of dedicated models.

What carries the argument

The load-bearing instrument is the decidability index d', defined as the separation between genuine-pair and impostor-pair similarity score distributions measured in pooled standard deviations. Because all four LVLMs produce tied or near-tied scores, the paper replaces rank-1 and mAP with d' and classification metrics (accuracy, precision, recall, F1, AUC). The other key mechanism is the two-phase evaluation protocol, in which each query is first compared with a single gallery image and then with a batch of five, plus the JSON-formatted system prompt that forces the LVLMs to output a similarity score and same/different class; this protocol is what exposes the batch-mode collapse.

What would settle it

Check whether the true identity appears in the top-20 gallery for each of the ten PersonViT-fail queries; if it is missing, re-run the LVLMs with galleries that guarantee the true match and compare d' and accuracy, since a large improvement would overturn the claim that LVLMs cannot handle re-identification.

Watch

Extended reading notes

Core claim

On the curated Market1501 subset, the paper reports that the specialized PersonViT model achieves a decidability index of d'=92.0, cleanly separating genuine from impostor pairs, whereas the best LVLM result is ChatGPT-4o in pairwise mode at d'=75.6, with most LVLMs falling to near-chance decidability in batch mode. The paper also observes that LVLMs frequently assign identical or near-identical similarity scores to different gallery candidates, which prevents the calculation of standard rank-1 accuracy and mean average precision and leads to unreliable answers. The authors present this as evidence that, despite LVLMs' impressive interpretability and verbal reasoning, their discriminative power for human re-identification is still far below state-of-the-art specialized models, particularly when multiple candidates must be compared at once.

Load-bearing premise

The evaluation assumes every query's true match is present in the top-20 gallery built from PersonViT's own ranking, which may not be true for the ten queries PersonViT failed on.

Editorial extensions

If this is right

  • Standard retrieval metrics such as rank-1 and mAP are not reliable for evaluating LVLMs on ReID until their score distributions stop collapsing to identical values.
  • For practical surveillance deployment, specialized ReID models should remain the core decision maker; LVLMs are not yet a substitute.
  • Pairwise and batch evaluations should be reported separately, because batch mode can hide or destroy the discriminative power of LVLMs.
  • Fusing specialized models with LVLMs, with iterative interaction, is the paper's proposed path to combine discriminative power with interpretability and reduce catastrophic answers.
  • LVLMs' initial refusals on surveillance tasks mean that privacy policies can prevent the task entirely, so prompt engineering is a necessary precondition for any such evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the gallery is built from PersonViT's own top-20 ranking, the ten queries that PersonViT failed on may have no correct match in the candidate set; the LVLMs' failure rate could therefore be inflated by impossible trials.
  • A direct test of the paper's fusion proposal would be to have PersonViT generate a shortlist, guarantee the true match is in it, and let an LVLM rerank that shortlist; the fusion conjecture predicts improved end-to-end accuracy.
  • The near-tied LVLM scores suggest that asking for six-decimal similarity scores is an arbitrary format; a rank-based or forced-unique scoring prompt might change the measured decidability.
  • The qualitative examples show different LVLMs rely on different cues (color, gait, accessories), which implies an ensemble or learned combination of LVLMs could reduce catastrophic errors without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates four large vision-language models (ChatGPT-4o, Gemini-2.0-Flash, Claude 3.5 Sonnet, and Qwen-VL-Max) on the human re-identification task using a 20-query subset of the Market1501 dataset, comparing them against the specialized PersonViT model. The evaluation uses pairwise and batch prompting, and reports similarity-score statistics, a decidability index d', classification metrics (accuracy, precision, recall, F1), and ROC/AUC curves. The central claim is that PersonViT provides far better separation of genuine and impostor pairs than any LVLM, and that LVLMs frequently produce nearly identical scores across candidates, leading to what the authors call catastrophic answers.

Significance. If the quantitative results were reliable, this study would provide a timely and useful benchmark for the community, showing the gap between general-purpose LVLMs and specialized ReID models, and offering qualitative insights into how LVLMs reason about identity. The two-phase design (pairwise vs. batch) and the use of structured JSON outputs are sensible ideas, and the interpretability examples are a genuine strength. However, the current quantitative support is compromised by metric inconsistencies and a biased evaluation design, so the headline claims are not yet established.

major comments (4)
  1. [Table 2 and Sections 4.1, 5.1, 6] The column labeled d' is not consistent with the decidability index computed from the means and standard deviations reported in the same table. For PersonViT, d' = (92.0 - 87.6) / sqrt((0.03^2 + 0.05^2)/2) ≈ 106.8, not 92.0; for ChatGPT-4o pairwise, d' ≈ 0.81, not 75.6. The printed values are numerically plausible as AUC percentages (92.0%, 75.6%, 35.3%), a metric the paper also says it computed. Either the column is mislabeled or an unreported formula was used. Since the abstract and conclusion rely on these separation numbers to claim 'catastrophic' LVLM failure, the quantitative evidence is not interpretable as stated.
  2. [Section 3.1, steps 1-4] The 20-query subset is stratified by PersonViT's success/fail status, and for each query the gallery is PersonViT's top-20 ranking. For the 10 'fail' queries, the ground-truth match may be absent from the top-20 gallery, meaning an LVLM cannot possibly return the correct identity regardless of its capability. The paper does not disclose how many queries have their true match in the gallery or how many genuine pairs are used for the d' and classification metrics. This selection bias makes the LVLM-versus-PersonViT comparison unfair and non-representative. The authors should report genuine-pair counts, the rank of the true match in PersonViT's ranking for each query, and either add a random-gallery control or restrict the analysis to queries whose true match is present in the gallery.
  3. [Section 4.1] The evaluation uses only 20 queries, yet no confidence intervals, error bars, or significance tests are reported for accuracy, precision, recall, F1, d', or AUC. With N=20, differences such as ChatGPT-4o pairwise accuracy of 52.6% versus Gemini-2.0-Flash's 41.4% are within sampling variability. The paper should provide bootstrap confidence intervals or a statistical comparison (e.g., McNemar's test) to support any claim of relative performance between models.
  4. [Section 3.3 (system prompt)] The system prompt fixes the response format and includes arbitrary thresholds (e.g., 0.90-1.00 as 'highly confident match', 0.75-0.89 as 'likely match', 0.50-0.74 as 'possible match') that likely steer the LVLMs' score distributions. The paper does not investigate sensitivity to these thresholds or justify their choice. Without a prompt ablation or a clear rationale, it is unclear whether the reported scores reflect intrinsic LVLM discrimination ability or prompt-induced behavior.
minor comments (6)
  1. [Section 4.1] The text says 'to obtain evaluation data (20 queries and 20 galleries)', which is inconsistent with Section 3.1's statement of 20 queries and 400 gallery images.
  2. [Abstract] There are typos in the abstract: 'regraded' should be 'regarded', and 'c+achieve' should likely be 'achieve'.
  3. [Figures 1 and 4 captions] The placeholder text 'Leveraging social media news' appears before the Figure 1 and Figure 4 captions; this appears to be a leftover from a template and should be removed.
  4. [Section 3.4] The text uses 'dĺ' where the decidability index d' is meant; this formatting error appears in Section 3.4.
  5. [Sections 3.4 and 4] The paper does not report the temperature, top-p, or other sampling parameters used for the LVLM API calls. Because LVLM outputs are stochastic, this omission hinders reproducibility.
  6. [Section 3.3] The paper notes that ChatGPT-4o and Claude-3.5-Sonnet initially refused the task for ethical reasons and were then induced to comply via a modified system prompt. The authors should briefly discuss the ethical implications of circumventing model refusals for surveillance-related tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical benchmark whose reported LVLM and PersonViT scores are measured outputs, not quantities derived from or equivalent to the evaluation inputs.

full rationale

The paper is an empirical comparison, not a derivation chain, so the classic circularity patterns do not apply. PersonViT is an externally published baseline model, and the LVLM similarity scores and classifications are API outputs obtained under a fixed prompt; none of these values are fitted parameters that are then renamed as predictions. The only evaluation-design concern is the curation protocol in Section 3.1: queries are split by PersonViT rank-1 success/failure and the gallery for each query is the top-20 list produced by PersonViT itself. That protocol could bias the comparison against LVLMs, because a 'fail' query may not have its correct match inside the gallery at all. However, this is a fairness and soundness issue about how the benchmark was constructed, not a case where a reported result is equivalent to its inputs by construction: the LVLMs are still being tested on images and prompts that do not encode the answer, and no equation reduces the headline d' or accuracy values to the curation rule. The self-citation in the paper (reference [1]) is contextual motivation and is not load-bearing. Separately, Table 2's d' column appears numerically inconsistent with the reported genuine/impostor means and standard deviations, and the printed values are more plausible as AUC percentages; that is a correctness or metric-labeling concern, not circularity. Under the hard rule requiring a specific reduction to inputs, no circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small, PersonViT-filtered test set. The free parameters are author choices in prompt and subset design, not fitted constants.

free parameters (3)
  • Prompt similarity thresholds = 0.90, 0.75, 0.60, 0.50 (specified in system prompt)
    Authored thresholds embedded in the prompt to guide LVLM scoring and classification, which may influence the reported similarity scores and thus all derived metrics.
  • Subset composition (10 success, 10 fail) = 10 each
    Hand-selected number of queries from PersonViT success/fail sets; the central comparison depends on this choice.
  • Batch size in Phase 2 = 5
    Five gallery images per query in batch mode; the paper shows performance varies with batch size, so this is a chosen experimental condition.
assumptions (4)
  • domain assumption The curated 20-query, 400-gallery subset is representative of Market1501 difficulty
    Section 3.1 states this subset ensures 'focused and efficient evaluation', but no statistical argument supports representativeness.
  • domain assumption For each query, the true match is present in its top-20 gallery list
    Section 3.1 builds the gallery via PersonViT ranking without verifying inclusion of the ground-truth match; if absent, the LVLM cannot succeed on that query.
  • domain assumption LVLM similarity scores and PersonViT scores are comparable for d' and AUC
    Section 4.1 computes decidability and AUC directly from these raw scores; the scales and semantics of LVLM scores are model-specific and not calibrated.
  • standard math Tied similarity scores are handled in a well-defined way when computing AUC
    The paper notes frequent identical scores but does not describe tie-breaking in ROC/AUC computation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human Re-ID Meets LVLMs: What can we expect?." pith.science (2026). https://pith.science/paper/EB4XVF4F

@misc{pith2026250118698,
  author       = {Pith},
  title        = {Pith review of: Human Re-ID Meets LVLMs: What can we expect?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EB4XVF4F}},
  note         = {Machine review of arXiv:2501.18698}
}
read the original abstract

Large vision-language models (LVLMs) have been regarded as a breakthrough advance in an astoundingly variety of tasks, from content generation to virtual assistants and multimodal search or retrieval. However, for many of these applications, the performance of these methods has been widely criticized, particularly when compared with state-of-the-art methods and technologies in each specific domain. In this work, we compare the performance of the leading large vision-language models in the human re-identification task, using as baseline the performance attained by state-of-the-art AI models specifically designed for this problem. We compare the results due to ChatGPT-4o, Gemini-2.0-Flash, Claude 3.5 Sonnet, and Qwen-VL-Max to a baseline ReID PersonViT model, using the well-known Market1501 dataset. Our evaluation pipeline includes the dataset curation, prompt engineering, and metric selection to assess the models' performance. Results are analyzed from many different perspectives: similarity scores, classification accuracy, and classification metrics, including precision, recall, F1 score, and area under curve (AUC). Our results confirm the strengths of LVLMs, but also their severe limitations that often lead to catastrophic answers and should be the scope of further research. As a concluding remark, we speculate about some further research that should fuse traditional and LVLMs to combine the strengths from both families of techniques and achieve solid improvements in performance.

Figures

Figures reproduced from arXiv: 2501.18698 by the authors.

Figure 1
Figure 1. Experimental workflow for evaluating LVLM for ReID tasks. recognition, face recognition, age estimation, and iris recognition by leveraging their multimodal nature to analyze facial features, expressions, and contextual cues with impressive accuracy [8, 9, 10, 12]. Despite these achievements the role of LVLMs in ReID remains largely unexplored. To fill this gap we evaluate the performance of leading LVLMs on the Mar… view at source ↗
Figure 2
Figure 2. An illustrative representation of the quality indices for selected LVLM. The chart highlights the comparative strengths of models across key benchmarks [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of similarity score distributions for a single query image (Q) retrieved using PersonViT (left) and the LVLM Qwen-VL-Max (right). PersonViT assigns distinct similarity scores, enabling effective ranking of gallery images as shown by score distribution. Conversely Qwen-VL-Max assigns nearly identical scores across most gallery images as illustrated in the histogram. This lack of differentiation in scores c… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of similarity score distributions for PersonViT (top left) and four LVLMs (columns), shown in both pairwise (top row) and batch (bottom row). Each histogram depicts genuine pairs (green) and impostor pairs (red) with superimposed density curves. The orange m…
Figure 5
Figure 5. Figure 5: ROC curves for pairwise (5a) and batch (5b) evaluations. Each plot shows the true positive rate (TPR) versus the false positive rate (FPR) for each model with the dashed diagonal indicating random chance performance. PersonViT (yellow) demonstrates the largest AUC refl…
Figure 6
Figure 6. Figure 6: Example where all four models unanimously classify the query (Q) and gallery (G) images as the same individual. the same individual. Despite reaching the same conclu￾sion their justifications differ significantly. ChatGPT￾4o emphasizes fine details such as footwear and…
Figure 7
Figure 7. Figure 7: Illustration of how subtle differences in clothing color and activity can affect model judgments. ChatGPT-4o, Claude-3.5 Sonnet, and Gemini-2.0-Flash classify this query gallery pair as the same person based on shared striped patterns and similar build. Qwen-VL-Max how…
Figure 8
Figure 8. Figure 8: Case where all four models agree that Q and G belong to the same person. gender, hair length, and height, assuming the T-shirt is a polo brand. In figure 8, all four models agree that the subject in the red t-shirt and dark shorts is the same individual but their analy…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [1]

    Hambarde, H

    K. Hambarde, H. Proença, Image-based human re- identification: Which covariates are actually (the most) important?, Image and Vision Computing 143 (2024) 104917. Hambarde, K.A. et al.:Preprint submitted to Elsevier Page 9 of 10 Leveraging social media news

    work page 2024

  2. [2]

    B. Hu, X. Wang, W. Liu, Personvit: Large-scale self- supervised vision transformer for person re-identification, arXiv preprint arXiv:2408.05398 (2024)

    work page Pith review arXiv 2024

  3. [3]

    1116–1124

    L.Zheng,L.Shen,L.Tian,S.Wang,J.Wang,Q.Tian, Scal- able person re-identification: A benchmark, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1116–1124

    work page 2015

  4. [4]

    Y. Zang, W. Li, J. Han, K. Zhou, C. C. Loy, Contextual object detection with multimodal large language models, International Journal of Computer Vision (2024) 1–19

    work page 2024

  5. [5]

    Han, S.-N

    G. Han, S.-N. Lim, Few-shot object detection with founda- tion models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28608–28618

    work page 2024

  6. [6]

    S. Ji, X. Zheng, C. Wu, Hargpt: Are llms zero-shot human activity recognizers?, arXiv preprint arXiv:2403.02727 (2024)

    arXiv 2024

  7. [7]

    GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM

    K. Bimbraw, Y. Wang, J. Liu, T. Koike-Akino, Gpt sonograpy: Hand gesture decoding from forearm ultrasound images via vlm, arXiv preprint arXiv:2407.10870 (2024)

    work page Pith review arXiv 2024

  8. [8]

    Z. Zhao, I. Patras, Prompting visual-language models for dynamic facial expression recognition, arXiv preprint arXiv:2308.13382 (2023)

    arXiv 2023

Show all 24 references
  1. [9]

    AlDahoul, M

    N. AlDahoul, M. J. T. Tan, H. R. Kasireddy, Y. Zaki, Exploring vision language models for facial attribute recog- nition: Emotion, race, gender, and age, arXiv preprint arXiv:2410.24148 (2024)

    2024 arXiv

  2. [10]

    Hassanpour, Y

    A. Hassanpour, Y. Kowsari, H. O. Shahreza, B. Yang, S. Marcel, Chatgpt and biometrics: an assessment of face recognition, gender detection, and age estimation capabili- ties, arXiv preprint arXiv:2403.02965 (2024)

    2024 arXiv

  3. [11]

    S.Jia,R.Lyu,K.Zhao,Y.Chen,Z.Yan,Y.Ju,C.Hu,X.Li, B. Wu, S. Lyu, Can chatgpt detect deepfakes? a study of using multimodal large language models for media forensics, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4324–4333

    2024

  4. [12]

    Farmanifard, A

    P. Farmanifard, A. Ross, Chatgpt meets iris biometrics, in: 2024 IEEE International Joint Conference on Biometrics (IJCB), IEEE, 2024, pp. 1–10

    2024

  5. [13]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4-technical-report, arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  6. [14]

    DeepMind, Google gemini ai update - december 2024, https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/ gemini-2-0-flash, 2024

    G. DeepMind, Google gemini ai update - december 2024, https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/ gemini-2-0-flash, 2024. Accessed: 2024-12-13

    2024

  7. [15]

    com/news/claude-3-5-sonnet, 2024

    Anthropic,Claude3.5andsonnet, https://www.anthropic. com/news/claude-3-5-sonnet, 2024. Accessed: 2024-12-13

    2024

  8. [16]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou, Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, arXiv preprint arXiv:2308.12966 (2023)

    2023 arXiv

  9. [17]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    2017

  10. [18]

    Zhang, J

    J. Zhang, J. Huang, S. Jin, S. Lu, Vision-language models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  11. [19]

    S. He, H. Luo, P. Wang, F. Wang, H. Li, W. Jiang, Transreid: Transformer-based object re-identification, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 15013–15022

    2021

  12. [20]

    K. Zhu, H. Guo, T. Yan, Y. Zhu, J. Wang, M. Tang, Pass: Part-aware self-supervised pre-training for person re- identification, in: European conference on computer vision, Springer, 2022, pp. 198–214

    2022

  13. [21]

    W. Chen, X. Xu, J. Jia, H. Luo, Y. Wang, F. Wang, R. Jin, X. Sun, Beyond appearance: a semantic controllable self-supervised learning framework for human-centric visual tasks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15050– 15061

    2023

  14. [22]

    H. Hu, X. Dong, J. Bao, D. Chen, L. Yuan, D. Chen, H. Li, Personmae: Person re-identification pre-training with masked autoencoders, IEEE Transactions on Multimedia (2024)

    2024

  15. [23]

    URL: https://openai.com/ index/hello-gpt-4o/, accessed: 2024-12-13

    OpenAI, Hello gpt-4o, 2024. URL: https://openai.com/ index/hello-gpt-4o/, accessed: 2024-12-13

    2024

  16. [24]

    Analysis, Llm leaderboard - comparison of gpt-4, llama 3, mistral, gemini and over 30 models, 2024

    A. Analysis, Llm leaderboard - comparison of gpt-4, llama 3, mistral, gemini and over 30 models, 2024. URL: https://artificialanalysis.ai/leaderboards/models, accessed: 2024-12-13. Hambarde, K.A. et al.:Preprint submitted to Elsevier Page 10 of 10

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.