Pith. sign in

REVIEW 5 major objections 6 minor 42 references

EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EZ-Sort cuts pairwise-comparison annotation cost by 90.5 percent using a zero-shot CLIP pre-ordering and uncertainty-guided routing.

desk verdict A sensible engineering combination with a real evaluation hole: the auto-resolved comparisons are never checked against human labels, so the reliability claim mostly measures the model agreeing with itself. read the letter →

arxiv 2508.21550 v1 pith:7TZMYB5H submitted 2025-08-29 cs.CV cs.AI

Yujin Park , Haejun Chung , Ikbeom Jang This is my paper · ORCID
classification cs.CVcs.AI
keywords PairwisecomparisonHuman-in-the-loopsortingVLM-basedpre-orderingCLIPzero-shotrankingUncertainty-guidedMergeSortEloratinginitializationAnnotationcost
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes EZ-Sort, an annotation pipeline that tries to reduce the human cost of pairwise ranking by letting a frozen vision-language model absorb most of the comparisons. CLIP first produces a rough zero-shot ordering through hierarchical binary prompts, and each item is seeded with a bucket-aware Elo rating; a standard MergeSort then runs on top, but only pairs whose KL-based uncertainty is high are sent to human annotators, while confident pairs are decided automatically by the sign of the Elo difference. On face-age, historical-photo, and retinal-quality datasets, EZ-Sort reports cutting human annotations by $90.5\%$ against exhaustive pairwise comparison and by $19.8\%$ against the prior human-in-the-loop sorting method at $n=100$, while keeping or improving inter-rater reliability. If true, this matters because pairwise comparison is the trusted protocol for subjective visual annotation, yet its quadratic cost currently prevents use at scale.

What carries the argument

The load-bearing mechanism is the uncertainty-routing rule that turns every MergeSort comparison into a choice between a human query and an automatic answer. For a pair $(i,j)$, EZ-Sort forms a win-probability distribution from their Elo ratings, measures its KL divergence from a uniform distribution, boosts that value for cross-bucket pairs and low-confidence CLIP predictions, and compares the result to an adaptive threshold: if $\mathrm{uncertainty}(i,j) \geq \theta_t$, a human answers; otherwise the outcome is $\mathrm{sign}(r_i - r_j)$. Around this rule sit the hierarchical CLIP pre-ordering, which recursively splits images with binary prompts so that each image's binary decisions encode a group index, and bucket-aware Elo initialization, which merges the fine groups into $k$ buckets and sets an initial rating from the bucket plus a confidence-dependent term. Because every comparison the standard MergeSort would make is still made, the schedule stays $O(n \log n)$; the routing rule only changes who answers.

What would settle it

Run EZ-Sort's automatic-answer rule in isolation on a domain with known ground-truth rankings, extract every pair the rule calls confident, and count how often $\mathrm{sign}(r_i - r_j)$ contradicts the ground truth; if the confident-error rate is non-negligible while the threshold says the pair is safe, the reported annotation savings are being paid for with unvetted inversions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a zero-shot model's rough ordering can replace most human comparisons inside a sorting algorithm. EZ-Sort's hierarchical CLIP prompting assigns each image to a group whose bits seed an Elo rating, and the KL-based routing rule decides, per MergeSort comparison, whether a person or the sign of the rating difference answers. The reported consequence is that at $n=100$ the pipeline needs 467 human comparisons instead of the 4,950 exhaustive comparisons or the 582 required by the earlier sorting approach, and the reliability numbers do not drop: EZ-Sort reaches the highest inter-rater reliability on retinal image quality (ICC 0.94, Spearman 0.85) and comparable reliability on the other two datasets. The paper's claim is therefore not only that annotations can be saved, but that the savings come from automating exactly the comparisons that the model's own uncertainty says are easy.

Load-bearing premise

The load-bearing premise is that whenever the KL-based uncertainty is below the adaptive threshold $\theta_t$, the automatic answer $\mathrm{sign}(r_i - r_j)$ is correct; because the accuracy term inside the threshold update is never defined, the method assumes this correctness rather than measuring it.

Editorial extensions

If this is right

  • At $n=100$ on FGNET, EZ-Sort requires 467 human comparisons versus 582 for the prior sorting method and 4,950 for exhaustive comparison, a direct consequence of replacing confident comparisons with automatic answers.
  • The $O(n \log n)$ comparison bound of MergeSort is preserved, so the savings are not traded against algorithmic complexity; the theoretical lower bound for comparison-based sorting is approached, at about $1.01 \times n\ln n$ for $n=100$.
  • Human annotations are concentrated where the model is unsure: only 23.1%, 18.4%, and 31.2% of comparisons were routed to humans at $n=30$, $50$, and $100$, respectively.
  • Hierarchical prompting outperforms flat prompting on the face-age task, improving Spearman correlation with true age from 0.83 to 0.90 and cutting mean absolute error by up to 2.0, so the pre-ordering itself is part of the measured gain.
  • On ambiguous medical images (EyePACS), EZ-Sort attains ICC 0.94 and Spearman 0.85, the best of the three annotation protocols compared, suggesting the routing rule helps in domains where absolute classification is unreliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The efficiency gain is not uniform in scale: EZ-Sort's relative advantage over the prior method peaks at $n=50$ (a 40.8% savings) and shrinks to 19.8% at $n=100$, which suggests the optimal operating regime is mid-sized batches where CLIP's pre-ordering is still mostly right; extrapolating to much larger $n$ without measuring CLIP accuracy would be risky.
  • A direct calibration experiment, plotting KL-based uncertainty against the empirical probability that $\mathrm{sign}(r_i - r_j)$ matches ground truth, would tell whether the routing rule is a sound decision-theoretic threshold; the paper reports no such curve, so the rule's correctness outside the three tested domains is an open question.
  • The threshold update in Eq. (5) depends on an $\mathrm{accuracy}_t$ variable that the manuscript never defines or measures; supplying that definition would make the automation schedule reproducible and would allow practitioners to set the human-query rate deliberately.
  • One natural extension is online calibration: have the human answers update the Elo scores and the CLIP confidence estimates as the run proceeds, so early human corrections can reduce the model's confident errors instead of only adjudicating uncertain pairs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes EZ-Sort, a three-stage pipeline for reducing the human annotation cost of pairwise ranking. It first produces a zero-shot pre-ordering through hierarchical CLIP prompting, then initializes bucket-aware Elo ratings, and finally runs an uncertainty-guided MergeSort that routes only high-uncertainty comparisons to human annotators while automatically resolving low-uncertainty comparisons from the Elo scores. Experiments on FGNET, DHCI, and EyePACS report inter-rater reliability and annotation counts, and the paper claims a 90.5% reduction in human annotations relative to exhaustive pairwise comparison and a 19.8% reduction relative to a prior sorting-based method at n = 100, while improving or maintaining inter-rater reliability.

Significance. If the efficiency and reliability claims held under proper validation, this would be a practically useful contribution to annotation-efficient ranking: the combination of a VLM prior with uncertainty-aware routing is natural, the code is released, and the reported reduction from O(n^2) to O(n log n) with a constant-factor saving is attractive. However, the current evidence does not yet support the central reliability and efficiency claims as stated. The reliability comparison is confounded because automatically resolved pairs use the same CLIP/Elo prior for all annotators, the adaptive routing rule depends on an undefined quantity, and the headline annotation counts are produced with hyperparameters selected on the same data used for the main efficiency table. The paper is best read as a promising proof-of-concept that needs additional validation before the quantitative claims can be accepted.

major comments (5)
  1. [Section 2.1.5, Eq. (5)] The adaptive threshold theta_t = theta_0 (1 + alpha remaining/total)^beta accuracy_t depends on accuracy_t, but accuracy_t is never defined or measured anywhere in the manuscript. This quantity controls which comparisons are sent to humans, so the exact routing rule cannot be reproduced, audited, or tested for sensitivity. Please define accuracy_t precisely, state how it is estimated during the run, and report its values or, alternatively, remove it from the threshold rule and refit the simpler policy.
  2. [Section 3, Table 1, and Section 2.1.5] The inter-rater reliability evidence is structurally confounded. For every pair that EZ-Sort resolves automatically (23.1%, 18.4%, and 31.2% of comparisons at n = 30, 50, and 100, respectively), the answer is sign(r_i - r_j) computed from the same CLIP/Elo prior for all three expert raters. Identical deterministic inputs produce identical outputs, mechanically inflating ICC and Spearman agreement regardless of whether the automatic answer agrees with human judgment. The paper never measures the accuracy of the automatically resolved comparisons against human labels. Please report auto-answer accuracy on held-out human-judged pairs, and/or recompute Table 1 using only human-resolved comparisons or using a procedure that treats automatic answers as a shared error source.
  3. [Section 3, Table 1] The abstract and Section 1 claim that EZ-Sort improves or maintains inter-rater reliability, but Table 1 shows that on DHCI the ICC drops from 0.78 (Sort comparison [4]) to 0.73 (EZ-Sort), with Spearman unchanged at 0.47. This does not support the 'maintaining reliability' claim for that dataset. Please either temper the claim to state dataset-specific behavior or provide a statistical test and explanation for the DHCI drop.
  4. [Section 3, Table 2, and Implementation details] The headline annotation-count reductions are presented without variance or error bars, yet the text states 'The reported improvements are statistically significant at p < 0.05.' In addition, the implementation details state that parameters (k, theta_0, alpha, beta, delta_b, and bucket counts) were selected via cross-validation, and Table 2 is reported on FGNET, the same domain used for that selection. Some of the reported 90.5% and 19.8% reductions may therefore reflect fitted hyperparameters rather than predicted performance. Please provide a held-out evaluation, report variance across repeated runs or cross-validation folds, and describe the statistical test used.
  5. [Section 2.1.4, Eqs. (2)-(4)] The uncertainty measure defined as uncertainty(i,j) = 1 - Priority(i,j)/log 2 is not a normalized complement of the information gain. Since Priority multiplies InfoGain by gamma (up to 1.2) and phi = 2.0 - avg_conf (up to 2.0), Priority can exceed log 2, making uncertainty negative. The behavior of the query rule in Eq. (4) for negative uncertainty is unspecified. Please redefine uncertainty so that it takes values in a calibrated range, or justify why negative values are acceptable for the routing decision.
minor comments (6)
  1. [Section 2.1.1] There is a typo in the automated prompt template: 'A voidbehavioural or contextual clues' should read 'Avoid behavioural or contextual clues.'
  2. [Section 2.1.1, Eq. (1)] Equation (1) uses d_i as the image-specific depth, but the text elsewhere refers to 'levels' without clearly defining the maximum depth or how image-specific depths are determined. Please make the depth notation and halting criterion explicit.
  3. [Section 2.1.4, Eq. (2)] The summation index in Eq. (2) is k, which conflicts with the symbol k used for the number of buckets elsewhere in the paper. Please use a different index, such as m, to avoid ambiguity.
  4. [Section 2.1.5] The adaptive threshold is described as '(Sec. 2.1.5)' within Section 2.1.5 itself. Please replace this self-reference with a descriptive label or a forward reference to the equation number.
  5. [Section 3, Human annotation cost] In the paragraph discussing the information-theoretic minimum, the text says 'at n = 100, we require 467 queries versus the 520-query lower limit,' but the n ln n bound used in the same paragraph is approximately 460.5 for n = 100. Please clarify which lower bound is meant and reconcile the discrepancy.
  6. [Section 3, Implementation details] Prompt generation uses GPT-4, which makes exact reproduction dependent on an external, non-deterministic model. Please release the exact prompt templates used for each dataset, or otherwise specify the prompts or seeds so the hierarchical pre-ordering can be reproduced.

Circularity Check

2 steps flagged · score 6.0 of 10

Inter-rater reliability is partly a self-comparison of the model: auto-resolved pairs use the same CLIP/Elo prior for every rater, so Table 1's agreement is inflated by construction; efficiency gains also come from FGNET cross-validated parameters reported on FGNET.

  1. other [Section 2.1.5 (Eq. 4) and Section 3 (Table 1, comparison method allocation)]
    "If the condition is false, the outcome is decided automatically by sign(ri − rj), where r denotes the current Elo scores. ... Human annotation was requested for 23.1%, 18.4%, and 31.2% of comparisons at n = 30, 50, and 100, respectively; the rest were resolved automatically using Elo predictions."

    Table 1 is presented as inter-rater reliability of EZ-Sort, but for every automatically resolved comparison all three experts receive the same machine answer computed from the same CLIP/Elo prior. Identical inputs produce identical outputs, so pairwise agreement on those 23-31% of pairs is forced to be perfect regardless of whether the machine judgment matches human judgment. The reliability metric therefore partially measures the model against itself, not human agreement; the paper never measures auto-answer accuracy against independent human labels. The central claim of 'improving or maintaining inter-rater reliability' is thus partly a restatement of the deterministic auto-resolution rule.

  2. fitted input called prediction [Section 3, Implementation details and Table 2]
    "Parameters were selected via cross-validation and prompts were generated with GPT-4. ... Table 2: The number of human annotations (comparisons) required. FGNET dataset is used."

    The headline annotation-cost reductions (90.5% vs exhaustive, 19.8% vs prior work at n=100) are reported on FGNET using hyperparameters (k, theta0, alpha, beta, delta_b) selected by cross-validation on that same FGNET data. No separate held-out split or independent test set is described for the efficiency benchmark. The reported cost figures are therefore in-sample fitted results rather than out-of-sample predictions; the efficiency advantage is partly a consequence of parameter selection rather than an independently verified property of the method.

full rationale

The core algorithm is not circular: hierarchical CLIP pre-ordering, bucket-aware Elo initialization, and KL-based MergeSort are implemented as described, and the human queries that are actually routed are real annotations. The FGNET age correlation (0.90 vs 0.83) is an external check against continuous ground-truth age, and the comparison against prior work [4] is a legitimate baseline rather than a load-bearing self-citation. However, the paper's validation of its two headline claims is partially circular. First, the inter-rater reliability evidence is structurally confounded: automatic comparisons are resolved by sign(ri-rj) from a single shared CLIP/Elo prior, so all raters receive identical machine answers on 23.1%, 18.4%, and 31.2% of comparisons; ICC/Spearman in Table 1 therefore includes machine self-consistency as if it were human agreement. Second, the annotation-efficiency numbers are reported on the same FGNET dataset whose cross-validation was used to select the hyperparameters, with no described held-out split, so the headline cost reductions are partly fitted rather than predicted. Additionally, Eq. (5) depends on accuracy_t, which is never defined or measured anywhere in the paper; this is a reproducibility gap rather than circularity, but it makes the human/auto routing rule impossible to audit. These issues affect the evidence for the strongest claims but do not make the algorithmic derivation itself equivalent to its inputs, so the score is 6: partial circularity in the reliability and efficiency claims.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a large set of hand-set or cross-validated parameters (bucket counts, thresholds, weights, noise scales) and on the unvalidated assumption that CLIP and GPT-4 produce a reliable ordinal prior. There are no new physical or mathematical entities; the method is an engineering combination of existing models and algorithms with several ad hoc tuning decisions.

free parameters (10)
  • k (number of buckets) = 5 (FGNET), 3 (DHCI, EyePACS)
    Chosen empirically per dataset; controls granularity of Elo initialization and cross-bucket weighting.
  • theta_0 (initial uncertainty threshold) = 0.15
    Selected via cross-validation; sets the fraction of comparisons routed to humans.
  • alpha (budget sensitivity) = 0.3
    Adaptive threshold parameter, cross-validated.
  • beta (accuracy adaptation exponent) = 0.9
    Adaptive threshold parameter, cross-validated.
  • gamma (cross-bucket priority weight) = 1.2
    Hand-set constant in Priority(ij); inflates priority for cross-bucket pairs.
  • delta_b (Elo noise bound) = 75
    Uniform noise range for Elo initialization, selected via cross-validation.
  • Elo K = 32
    Rating-update constant used in Elo; standard value, not tuned per task.
  • r_base range = [1200, 1800]
    Linear Elo base scores across buckets; hand-set.
  • tau (CLIP temperature) = 0.1
    Softmax temperature for CLIP confidence; chosen manually.
  • phi constant = 2.0 in 2.0 - avg_conf
    Hand-set multiplier in confidence penalty phi.
assumptions (5)
  • standard math MergeSort runs in O(n log n) comparisons and produces a correct total order when the comparator is consistent.
    Used in Section 2.1.5 to claim EZ-Sort preserves optimal comparison complexity; assumes automatic comparator sign(r_i - r_j) is consistent enough.
  • domain assumption CLIP cosine similarity between image and text prompts is a meaningful zero-shot ordinal signal for the target attribute.
    The entire pre-ordering in Section 2.1.2 relies on this to create buckets and Elo initializations.
  • domain assumption GPT-4-generated binary prompt hierarchies correspond to visually distinguishable attribute levels.
    Section 2.1.1 uses GPT-4 prompts as ground truth for recursive splits; no validation of the prompt quality is given beyond final results.
  • ad hoc to paper The KL-based uncertainty measure and the Priority(ij) formula identify which comparisons a human should resolve.
    Eqs. (2)-(4) are introduced without derivation; gamma=1.2 and phi=2.0-avg_conf are ad hoc weights chosen to make routing work.
  • ad hoc to paper The adaptive threshold formula theta_t = theta_0 (1 + alpha remaining/total)^beta accuracy_t is a valid control policy.
    Section 2.1.5, Eq. (5); accuracy_t is not defined or measured anywhere, so the update rule is not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting." pith.science (2026). https://pith.science/paper/7TZMYB5H

@misc{pith2026250821550,
  author       = {Pith},
  title        = {Pith review of: EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TZMYB5H}},
  note         = {Machine review of arXiv:2508.21550}
}
read the original abstract

Pairwise comparison is often favored over absolute rating or ordinal classification in subjective or difficult annotation tasks due to its improved reliability. However, exhaustive comparisons require a massive number of annotations (O(n^2)). Recent work has greatly reduced the annotation burden (O(n log n)) by actively sampling pairwise comparisons using a sorting algorithm. We further improve annotation efficiency by (1) roughly pre-ordering items using the Contrastive Language-Image Pre-training (CLIP) model hierarchically without training, and (2) replacing easy, obvious human comparisons with automated comparisons. The proposed EZ-Sort first produces a CLIP-based zero-shot pre-ordering, then initializes bucket-aware Elo scores, and finally runs an uncertainty-guided human-in-the-loop MergeSort. Validation was conducted using various datasets: face-age estimation (FGNET), historical image chronology (DHCI), and retinal image quality assessment (EyePACS). It showed that EZ-Sort reduced human annotation cost by 90.5% compared to exhaustive pairwise comparisons and by 19.8% compared to prior work (when n = 100), while improving or maintaining inter-rater reliability. These results demonstrate that combining CLIP-based priors with uncertainty-aware sampling yields an efficient and scalable solution for pairwise ranking.

Figures

Figures reproduced from arXiv: 2508.21550 by the authors.

Figure 1
Figure 1. Overview of the EZ-Sort framework. The framework operates in three stages: (a) CLIP-based zero-shot [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. CLIP-based hierarchical classification at Level 1. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 37 canonical work pages

  1. [4]

    Ikbeom Jang, Garrison Danley, Ken Chang, and Jayashree Kalpathy-Cramer

  2. [1]

    Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al

    Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. 2023. Faster sorting algorithms discovered using deep reinforcement learning. Nature 618, 7964 (2023), 257–263

    work page 2023

  3. [2]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision 130, 9 (2022), 2337–2348

    2022

  4. [3]

    Jing Li, Rafal Mantiuk, Junle Wang, Suiyi Ling, and Patrick Le Callet. 2018. Hybrid- MST: A hybrid active sampling strategy for pairwise preference aggregation. In Advances in Neural Information Processing Systems , Vol. 31

    work page 2018

  5. [5]

    Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345

    work page 1952

  6. [6]

    Joseph L. Hansen. 1978. An Application of the Elo Rating System to Professional Baseball. Ph.D. Dissertation. Kalamazoo College

    work page 1978

  7. [7]

    Robin Swezey, Aditya Grover, Bruno Charron, and Stefano Ermon. 2021. Pi- rank: Scalable learning to rank via differentiable sorting. In Advances in Neural Information Processing Systems, Vol. 34, 21644–21654

    work page 2021

  8. [8]

    Xingjian Bai and Christian Coester. 2023. Sorting with predictions. In Advances in Neural Information Processing Systems , Vol. 36, 26563–26584

    work page 2023

Show all 42 references
  1. [9]

    Lukas Gienapp, Benno Stein, Matthias Hagen, and Martin Potthast. 2020. Efficient pairwise annotation of argument quality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 5772–5781

    2020

  2. [10]

    Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et al. 2022. K-lite: Learning transferable visual models with external knowledge. In Advances in Neural Information Processing Systems, Vol. 35, 15558–15573

    2022

  3. [11]

    Qi Qian and Juhua Hu. 2024. Online zero-shot classification with CLIP. In European Conference on Computer Vision , 462–477. Springer

    2024

  4. [12]

    Jamieson and Robert Nowak

    Kevin G. Jamieson and Robert Nowak. 2011. Active ranking using pairwise comparisons. In Advances in Neural Information Processing Systems , Vol. 24

    2011

  5. [13]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learni...

    2021

  6. [14]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 12888–12900. PMLR

    2022

  7. [15]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International Conference on Machine Learning , 4904–4916. PMLR

    2021

  8. [16]

    Movellan, and Paul Ruvolo

    Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul Ruvolo

  9. [17]

    Raykar, Shipeng Yu, Linda H

    Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo H. Valadez, Charles Florin, Luca Bogoni, and Lauren Moy. 2010. Learning from crowds. Journal of Machine Learning Research 11 (2010), 1297–1322

    2010

  10. [18]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional Prompt Learning for Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 16816–16825

    2022

  11. [19]

    Huazhu Fu, Boyang Wang, Jianbing Shen, Shanshan Cui, Yanwu Xu, Jiang Liu, and Ling Shao. 2019. Evaluation of retinal image quality assessment networks in different color-spaces. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 , 48–56. Springer

    2019

  12. [20]

    Frank Palermo, James Hays, and Alexei A. Efros. 2012. Dating historical color images. In Computer Vision – ECCV 2012 , 499–512. Springer

    2012

  13. [21]

    Lanitis, C.J

    A. Lanitis, C.J. Taylor, and T.F. Cootes. 2002. Toward automatic simulation of aging effects on face images. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 4 (2002), 442–455

    2002

  14. [22]

    Thurstone

    Louis L. Thurstone. 1927. The method of paired comparisons for social values. Journal of Abnormal and Social Psychology 21, 4 (1927), 384

    1927

  15. [23]

    Bennett, Kevyn Collins-Thompson, and Eric Horvitz

    Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining , 193–202

    2013

  16. [24]

    Lucas Maystre and Matthias Grossglauser. 2017. Just sort it! A simple and effective approach to active preference learning. In International Conference on Machine Learning, 2344–2353. PMLR

    2017

  17. [25]

    Hanqing Zhao and Yuehan Luo. 2018. An 𝑂(𝑁) Sorting Algorithm: Machine Learning Sort. arXiv preprint arXiv:1805.04272 (2018)

    2018 arXiv

  18. [26]

    Graham, Donald E

    Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. 1994. Concrete Mathe- matics: A Foundation for Computer Science . Addison-Wesley, 2nd edition

    1994

  19. [27]

    Cormen, Charles E

    Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein

  20. [28]

    Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueSkill ™: A Bayesian skill rating system. In Advances in Neural Information Processing Systems , Vol. 19

    2006

  21. [29]

    Akash Kumar Mohankumar and Mitesh Khapra. 2022. Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8761–8781. Association for Computation...

    2022

  22. [30]

    MIT Press, 3rd edition

    Introduction to Algorithms. MIT Press, 3rd edition

  23. [31]

    Maytal Saar-Tsechansky and Foster Provost. 2004. Active sampling for class probability estimation and ranking. Machine Learning 54 (2004), 153–178

    2004

  24. [32]

    Devichand Budagam, Ashutosh Kumar, Mahsa Khoshnoodi, Sankalp KJ, Vinija Jain, and Aman Chadha. 2024. Hierarchical Prompting Taxonomy: A Univer- sal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles. arXiv preprint arXiv:2406.12644 (2024)

    2024 arXiv

  25. [33]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A. Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, Vol. 35, 1950–1965

    2022

  26. [34]

    Anderson, David R

    Lorin W. Anderson, David R. Krathwohl, Peter W. Airasian, Kathleen A. Cruik- shank, Richard E. Mayer, Paul R. Pintrich, James Raths, and Merlin C. Wittrock

  27. [35]

    Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jonathan Lenchner, Andrea Loreggia, Keerthiram Murgesan, Nicholas Mattei, Francesca Rossi, et al

  28. [36]

    Benjamin Samuel Bloom. 1956. Taxonomy of Educational Objectives: The Classifi- cation of Educational Goals . Longmans, Green

    1956

  29. [37]

    Peter Campbell, Deniz Erdogmus, Peng Tian, Dha- ranish Kedarisetti, Chace Moleta, James D

    Jayashree Kalpathy-Cramer, J. Peter Campbell, Deniz Erdogmus, Peng Tian, Dha- ranish Kedarisetti, Chace Moleta, James D. Reynolds, Kelly Hutcheson, Michael J. Shapiro, Michael X. Repka, et al. 2016. Plus disease in retinopathy of prematurity: improving diagnosis by ranking dis...

    2016

  30. [41]

    Richard Cole. 1988. Parallel merge sort. SIAM Journal on Computing 17, 4 (1988), 770–785

    1988

  31. [2009]

    In Advances in Neural Information Processing Systems , Vol

    Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems , Vol. 22

  32. [2014]

    A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Pearson

  33. [2021]

    In Proceedings of the AAAI Conference on Artificial Intelligence, Vol

    Thinking fast and slow in AI. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 15042–15046

  34. [2022]

    arXiv preprint arXiv:2202.04823 (2022)

    Decreasing annotation burden of pairwise comparisons with human-in- the-loop sorting: Application in medical image artifact rating. arXiv preprint arXiv:2202.04823 (2022)

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.