{"id":"cf894459-d7ea-4c3a-9114-df5b8d76b6af","arxiv_id":"2508.21550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"Zero-shot CLIP pre-ordering plus uncertainty-guided MergeSort reduces human pairwise-comparison annotations by up to 90.5% and by 19.8% over prior active sorting.","lead":"A new annotation pipeline, EZ-Sort, uses a CLIP model to pre-sort images and asks humans only for the comparisons the model is unsure about. On three datasets it cuts human comparison effort by up to 90.5% versus exhaustive pairwise labels and by about 20% versus a prior active-sorting method, with similar inter-rater reliability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reliability claim is confounded: auto-resolved pairs use the same shared CLIP/Elo prior for all raters, so Table 1's ICC/Spearman cannot validate the automatic comparisons. The paper never measures auto-answer accuracy against human labels.","rationale":"The reader's weakest assumption identifies automatic-comparison reliability and the undefined accuracy_t; I agree those are the weak points, but I would sharpen the emphasis: the paper's own reliability metric cannot detect errors in the automatic component because that component is identical across raters. The central claim has two parts: human-annotation cost reduction and preserved reliability. The cost part is straightforwardly measured by counting human queries, and the reported reductions are large. The load-bearing condition is the reliability part: if automated comparisons are wrong, EZ-Sort can achieve low cost by confidently inserting errors into the final ranking. Table 1 cannot bear that weight, since automatically resolved pairs contribute the same fixed answer to every expert's ranking. The missing measurement is a direct accuracy check of Eq. (4)'s automatic decisions against human consensus or ground-truth labels. The proposed test would settle the concern: if auto-rule agreement with human majority is at least as high as human-human agreement, the automation is safe; if it is lower, the reliability advantage is an artifact of the shared prior, and the method needs a human-review fallback for low-uncertainty pairs. The undefined accuracy_t in Eq. (5) is a real reproducibility gap but secondary. The paper provides code, a clear pipeline, and plausible empirical results, so conditional acceptance remains appropriate; it needs this missing validation before the central claim can be accepted as stated.","tokens_in":8508,"tokens_out":6438,"duration_ms":66882,"concrete_test":"For each dataset, take the n = 30 runs, collect all pairs that EZ-Sort resolves automatically (Eq. (4) false), obtain independent majority-vote human labels for those pairs, and compute the agreement between sign(r_i - r_j) and the human majority. If this auto-rule accuracy is not at or above the human inter-rater agreement rate for that dataset, then automatic answers inject errors that the current ICC/Spearman table cannot detect, and the central efficiency claim is undermined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the automatic comparisons chosen by Eq. (4) be correct at roughly human level. The paper never measures this. The only correctness evidence is the inter-rater reliability in Table 1, and that evidence is structurally confounded: for every pair EZ-Sort resolves automatically (23.1%, 18.4%, and 31.2% of comparisons at n = 30, 50, 100, per Section 3), the answer sign(r_i - r_j) is computed from the same fixed CLIP/Elo prior for all three experts. Identical inputs produce identical outputs, mechanically inflating ICC and Spearman agreement regardless of whether the automatic answer matches human judgment. The FGNET age correlation (0.90 vs 0.83 for flat prompting) is an aggregate check on one dataset with cross-validated parameters, not an error rate for the auto-resolved pairs. In addition, the adaptive threshold Eq. (5) depends on accuracy_t, which is never defined or measured anywhere in the paper, so the exact human/auto routing rule cannot be reproduced or audited. A confident CLIP mistake on a low-uncertainty pair therefore enters the final ranking with no human review, and the paper provides no estimate of how often this happens.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EZ-Sort, a three-stage pipeline for reducing the human annotation cost of pairwise ranking. It first produces a zero-shot pre-ordering through hierarchical CLIP prompting, then initializes bucket-aware Elo ratings, and finally runs an uncertainty-guided MergeSort that routes only high-uncertainty comparisons to human annotators while automatically resolving low-uncertainty comparisons from the Elo scores. Experiments on FGNET, DHCI, and EyePACS report inter-rater reliability and annotation counts, and the paper claims a 90.5% reduction in human annotations relative to exhaustive pairwise comparison and a 19.8% reduction relative to a prior sorting-based method at n = 100, while improving or maintaining inter-rater reliability.","tokens_in":8800,"tokens_out":5685,"duration_ms":53221,"significance":"If the efficiency and reliability claims held under proper validation, this would be a practically useful contribution to annotation-efficient ranking: the combination of a VLM prior with uncertainty-aware routing is natural, the code is released, and the reported reduction from O(n^2) to O(n log n) with a constant-factor saving is attractive. However, the current evidence does not yet support the central reliability and efficiency claims as stated. The reliability comparison is confounded because automatically resolved pairs use the same CLIP/Elo prior for all annotators, the adaptive routing rule depends on an undefined quantity, and the headline annotation counts are produced with hyperparameters selected on the same data used for the main efficiency table. The paper is best read as a promising proof-of-concept that needs additional validation before the quantitative claims can be accepted.","major_comments":[{"comment":"The adaptive threshold theta_t = theta_0 (1 + alpha remaining/total)^beta accuracy_t depends on accuracy_t, but accuracy_t is never defined or measured anywhere in the manuscript. This quantity controls which comparisons are sent to humans, so the exact routing rule cannot be reproduced, audited, or tested for sensitivity. Please define accuracy_t precisely, state how it is estimated during the run, and report its values or, alternatively, remove it from the threshold rule and refit the simpler policy.","section":"Section 2.1.5, Eq. (5)"},{"comment":"The inter-rater reliability evidence is structurally confounded. For every pair that EZ-Sort resolves automatically (23.1%, 18.4%, and 31.2% of comparisons at n = 30, 50, and 100, respectively), the answer is sign(r_i - r_j) computed from the same CLIP/Elo prior for all three expert raters. Identical deterministic inputs produce identical outputs, mechanically inflating ICC and Spearman agreement regardless of whether the automatic answer agrees with human judgment. The paper never measures the accuracy of the automatically resolved comparisons against human labels. Please report auto-answer accuracy on held-out human-judged pairs, and/or recompute Table 1 using only human-resolved comparisons or using a procedure that treats automatic answers as a shared error source.","section":"Section 3, Table 1, and Section 2.1.5"},{"comment":"The abstract and Section 1 claim that EZ-Sort improves or maintains inter-rater reliability, but Table 1 shows that on DHCI the ICC drops from 0.78 (Sort comparison [4]) to 0.73 (EZ-Sort), with Spearman unchanged at 0.47. This does not support the 'maintaining reliability' claim for that dataset. Please either temper the claim to state dataset-specific behavior or provide a statistical test and explanation for the DHCI drop.","section":"Section 3, Table 1"},{"comment":"The headline annotation-count reductions are presented without variance or error bars, yet the text states 'The reported improvements are statistically significant at p < 0.05.' In addition, the implementation details state that parameters (k, theta_0, alpha, beta, delta_b, and bucket counts) were selected via cross-validation, and Table 2 is reported on FGNET, the same domain used for that selection. Some of the reported 90.5% and 19.8% reductions may therefore reflect fitted hyperparameters rather than predicted performance. Please provide a held-out evaluation, report variance across repeated runs or cross-validation folds, and describe the statistical test used.","section":"Section 3, Table 2, and Implementation details"},{"comment":"The uncertainty measure defined as uncertainty(i,j) = 1 - Priority(i,j)/log 2 is not a normalized complement of the information gain. Since Priority multiplies InfoGain by gamma (up to 1.2) and phi = 2.0 - avg_conf (up to 2.0), Priority can exceed log 2, making uncertainty negative. The behavior of the query rule in Eq. (4) for negative uncertainty is unspecified. Please redefine uncertainty so that it takes values in a calibrated range, or justify why negative values are acceptable for the routing decision.","section":"Section 2.1.4, Eqs. (2)-(4)"}],"minor_comments":[{"comment":"There is a typo in the automated prompt template: 'A voidbehavioural or contextual clues' should read 'Avoid behavioural or contextual clues.'","section":"Section 2.1.1"},{"comment":"Equation (1) uses d_i as the image-specific depth, but the text elsewhere refers to 'levels' without clearly defining the maximum depth or how image-specific depths are determined. Please make the depth notation and halting criterion explicit.","section":"Section 2.1.1, Eq. (1)"},{"comment":"The summation index in Eq. (2) is k, which conflicts with the symbol k used for the number of buckets elsewhere in the paper. Please use a different index, such as m, to avoid ambiguity.","section":"Section 2.1.4, Eq. (2)"},{"comment":"The adaptive threshold is described as '(Sec. 2.1.5)' within Section 2.1.5 itself. Please replace this self-reference with a descriptive label or a forward reference to the equation number.","section":"Section 2.1.5"},{"comment":"In the paragraph discussing the information-theoretic minimum, the text says 'at n = 100, we require 467 queries versus the 520-query lower limit,' but the n ln n bound used in the same paragraph is approximately 460.5 for n = 100. Please clarify which lower bound is meant and reconcile the discrepancy.","section":"Section 3, Human annotation cost"},{"comment":"Prompt generation uses GPT-4, which makes exact reproduction dependent on an external, non-deterministic model. Please release the exact prompt templates used for each dataset, or otherwise specify the prompts or seeds so the hierarchical pre-ordering can be reproduced.","section":"Section 3, Implementation details"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and the code release is a positive sign, but the current evidence base is not yet sufficient for the strength of the claims. The most serious gap is the absence of any measurement of automatic-comparison accuracy: without it, the reliability table is uninterpretable. A second concern is the circularity between cross-validated hyperparameters and the main efficiency table. Both are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. EZ-Sort is a reasonable engineering combination—CLIP pre-ordering, bucket-aware Elo initialization, and KL-uncertainty routing inside a MergeSort—and the measured reduction in human comparisons is consistent across dataset sizes (roughly 20–40% over the active-sorting baseline, much larger over exhaustive comparison). But the headline reliability claim is weaker than it looks: the automatic comparisons use the same CLIP/Elo prior for all three raters, so the ICC/Spearman agreement cannot validate whether those automatic answers are actually correct.\n\nWhat is genuinely new is the integration, not the individual pieces. Hierarchical prompting, Elo-based rating, and KL-based priority all build on prior work, but combining them into a single MergeSort with adaptive human/auto routing is not in the cited literature. The authors also ship code, which makes the method testable. The three-dataset evaluation with expert raters is a real step beyond synthetic benchmarks, and the FGNET age-correlation check (0.90 vs. 0.83 for flat prompting) gives some evidence that hierarchical prompting helps.\n\nThe soft spots are substantial. Most importantly, the paper never measures the accuracy of automatic comparisons against human labels. Section 3 reports that 23–31% of comparisons were auto-resolved, but those answers are computed from the same fixed model for every rater. Identical inputs produce identical outputs, so the inter-rater numbers are mechanically inflated. The FGNET correlation is an aggregate sanity check, not an error-rate estimate for auto-resolved pairs. Relatedly, Eq. (5) depends on accuracy_t, which is never defined or measured anywhere; the routing rule therefore cannot be reproduced or audited. A confidently wrong CLIP prediction can enter the final ranking with no human review, and the paper gives no estimate of how often that happens.\n\nSecondary issues: Table 2 has no error bars or variance despite p<0.05 claims; the DHCI ICC drops from 0.78 to 0.73, which conflicts with the abstract's 'improving or maintaining' phrasing; and the hyperparameters were cross-validated on the same datasets used for the headline numbers, so some of the gain may be fitted rather than predicted. These are fixable with held-out evaluation and a defined accuracy metric, but they need to be fixed before the efficiency claims can be taken at face value.\n\nWho should read this: people building annotation pipelines for subjective, medical, or historical ranking tasks, and anyone studying evaluation confounds in human-in-the-loop systems. It deserves a serious referee—the idea is worth testing, the code is available, and the claims are specific enough to check. A referee should send it back for a direct human-accuracy measurement on the auto-resolved subset, error bars, and a clear definition of accuracy_t. I would not take the current reliability numbers as evidence that the automatic comparisons are correct.","headline":"A sensible engineering combination with a real evaluation hole: the auto-resolved comparisons are never checked against human labels, so the reliability claim mostly measures the model agreeing with itself.","tokens_in":9363,"tokens_out":2210,"would_cite":true,"duration_ms":22423,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EZ-Sort cuts pairwise-comparison annotation cost by 90.5 percent using a zero-shot CLIP pre-ordering and uncertainty-guided routing.","keywords":["Pairwise comparison","Human-in-the-loop sorting","VLM-based pre-ordering","CLIP zero-shot ranking","Uncertainty-guided MergeSort","Elo rating initialization","Annotation cost"],"falsifier":"Run EZ-Sort's automatic-answer rule in isolation on a domain with known ground-truth rankings, extract every pair the rule calls confident, and count how often $\\mathrm{sign}(r_i - r_j)$ contradicts the ground truth; if the confident-error rate is non-negligible while the threshold says the pair is safe, the reported annotation savings are being paid for with unvetted inversions.","tokens_in":8229,"feed_emoji":"🏷️","tokens_out":10599,"duration_ms":88303,"temperature":0.7,"pith_summary":"The paper proposes EZ-Sort, an annotation pipeline that tries to reduce the human cost of pairwise ranking by letting a frozen vision-language model absorb most of the comparisons. CLIP first produces a rough zero-shot ordering through hierarchical binary prompts, and each item is seeded with a bucket-aware Elo rating; a standard MergeSort then runs on top, but only pairs whose KL-based uncertainty is high are sent to human annotators, while confident pairs are decided automatically by the sign of the Elo difference. On face-age, historical-photo, and retinal-quality datasets, EZ-Sort reports cutting human annotations by $90.5\\%$ against exhaustive pairwise comparison and by $19.8\\%$ against the prior human-in-the-loop sorting method at $n=100$, while keeping or improving inter-rater reliability. If true, this matters because pairwise comparison is the trusted protocol for subjective visual annotation, yet its quadratic cost currently prevents use at scale.","feed_headline":"CLIP pre-ordering cuts pairwise annotation cost by 90.5 percent","feed_subtitle":"EZ-Sort lets a frozen vision-language model answer easy comparisons and sends only uncertain pairs to human annotators.","key_machinery":"The load-bearing mechanism is the uncertainty-routing rule that turns every MergeSort comparison into a choice between a human query and an automatic answer. For a pair $(i,j)$, EZ-Sort forms a win-probability distribution from their Elo ratings, measures its KL divergence from a uniform distribution, boosts that value for cross-bucket pairs and low-confidence CLIP predictions, and compares the result to an adaptive threshold: if $\\mathrm{uncertainty}(i,j) \\geq \\theta_t$, a human answers; otherwise the outcome is $\\mathrm{sign}(r_i - r_j)$. Around this rule sit the hierarchical CLIP pre-ordering, which recursively splits images with binary prompts so that each image's binary decisions encode a group index, and bucket-aware Elo initialization, which merges the fine groups into $k$ buckets and sets an initial rating from the bucket plus a confidence-dependent term. Because every comparison the standard MergeSort would make is still made, the schedule stays $O(n \\log n)$; the routing rule only changes who answers.","core_discovery":"On the paper's own terms, the central discovery is that a zero-shot model's rough ordering can replace most human comparisons inside a sorting algorithm. EZ-Sort's hierarchical CLIP prompting assigns each image to a group whose bits seed an Elo rating, and the KL-based routing rule decides, per MergeSort comparison, whether a person or the sign of the rating difference answers. The reported consequence is that at $n=100$ the pipeline needs 467 human comparisons instead of the 4,950 exhaustive comparisons or the 582 required by the earlier sorting approach, and the reliability numbers do not drop: EZ-Sort reaches the highest inter-rater reliability on retinal image quality (ICC 0.94, Spearman 0.85) and comparable reliability on the other two datasets. The paper's claim is therefore not only that annotations can be saved, but that the savings come from automating exactly the comparisons that the model's own uncertainty says are easy.","pith_inferences":["The efficiency gain is not uniform in scale: EZ-Sort's relative advantage over the prior method peaks at $n=50$ (a 40.8% savings) and shrinks to 19.8% at $n=100$, which suggests the optimal operating regime is mid-sized batches where CLIP's pre-ordering is still mostly right; extrapolating to much larger $n$ without measuring CLIP accuracy would be risky.","A direct calibration experiment, plotting KL-based uncertainty against the empirical probability that $\\mathrm{sign}(r_i - r_j)$ matches ground truth, would tell whether the routing rule is a sound decision-theoretic threshold; the paper reports no such curve, so the rule's correctness outside the three tested domains is an open question.","The threshold update in Eq. (5) depends on an $\\mathrm{accuracy}_t$ variable that the manuscript never defines or measures; supplying that definition would make the automation schedule reproducible and would allow practitioners to set the human-query rate deliberately.","One natural extension is online calibration: have the human answers update the Elo scores and the CLIP confidence estimates as the run proceeds, so early human corrections can reduce the model's confident errors instead of only adjudicating uncertain pairs."],"forward_implications":["At $n=100$ on FGNET, EZ-Sort requires 467 human comparisons versus 582 for the prior sorting method and 4,950 for exhaustive comparison, a direct consequence of replacing confident comparisons with automatic answers.","The $O(n \\log n)$ comparison bound of MergeSort is preserved, so the savings are not traded against algorithmic complexity; the theoretical lower bound for comparison-based sorting is approached, at about $1.01 \\times n\\ln n$ for $n=100$.","Human annotations are concentrated where the model is unsure: only 23.1%, 18.4%, and 31.2% of comparisons were routed to humans at $n=30$, $50$, and $100$, respectively.","Hierarchical prompting outperforms flat prompting on the face-age task, improving Spearman correlation with true age from 0.83 to 0.90 and cutting mean absolute error by up to 2.0, so the pre-ordering itself is part of the measured gain.","On ambiguous medical images (EyePACS), EZ-Sort attains ICC 0.94 and Spearman 0.85, the best of the three annotation protocols compared, suggesting the routing rule helps in domains where absolute classification is unreliable."],"supporting_citations":[{"why":"Supplies the prior human-in-the-loop sorting baseline that EZ-Sort extends and measures its relative reduction against.","marker":"[4]"},{"why":"Supplies the frozen CLIP model that produces the zero-shot hierarchical pre-ordering.","marker":"[13]"},{"why":"Defines the exhaustive pairwise-comparison baseline whose quadratic cost EZ-Sort is designed to avoid.","marker":"[22]"},{"why":"Provides the FGNET face-age dataset and continuous age labels used for the main efficiency and correlation experiments.","marker":"[21]"},{"why":"Provides the DHCI historical-image chronology dataset used for the inter-rater reliability evaluation.","marker":"[20]"},{"why":"Provides the EyePACS retinal image quality dataset where EZ-Sort reports its highest reliability.","marker":"[19]"}],"fun_headline_variants":["EZ-Sort uses CLIP to slash pairwise comparisons by 90.5%","Zero-shot CLIP pre-ordering cuts human annotation cost 90.5%","Uncertainty-guided MergeSort with CLIP priors beats prior work by 19.8%","CLIP-based pre-ordering trims 90.5% of human comparisons, keeps reliability","Frozen CLIP + Elo-routed sorting: 90.5% fewer human clicks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that whenever the KL-based uncertainty is below the adaptive threshold $\\theta_t$, the automatic answer $\\mathrm{sign}(r_i - r_j)$ is correct; because the accuracy term inside the threshold update is never defined, the method assumes this correctness rather than measuring it.","fun_headline_variants_meta":{"raw":{"variants":["EZ-Sort uses CLIP to slash pairwise comparisons by 90.5%","Zero-shot CLIP pre-ordering cuts human annotation cost 90.5%","Uncertainty-guided MergeSort with CLIP priors beats prior work by 19.8%","CLIP-based pre-ordering trims 90.5% of human comparisons, keeps reliability","Frozen CLIP + Elo-routed sorting: 90.5% fewer human clicks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1190,"prompt_tokens":977,"completion_tokens":213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":97}},"tokens_in":593,"tokens_out":213,"duration_ms":2593,"temperature":1.0,"reasoning_tokens":97,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:40:25.965824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EZ-Sort's automatic-answer rule in isolation on a domain with known ground-truth rankings, extract every pair the rule calls confident, and count how often $\\mathrm{sign}(r_i - r_j)$ contradicts the ground truth; if the confident-error rate is non-negligible while the threshold says the pair is safe, the reported annotation savings are being paid for with unvetted inversions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP model that produces the zero-shot hierarchical pre-ordering."},{"cited_title":"Thurstone","cited_arxiv_id":null,"evidence_quote":"Defines the exhaustive pairwise-comparison baseline whose quadratic cost EZ-Sort is designed to avoid."},{"cited_title":"Lanitis, C.J","cited_arxiv_id":null,"evidence_quote":"Provides the FGNET face-age dataset and continuous age labels used for the main efficiency and correlation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DHCI historical-image chronology dataset used for the inter-rater reliability evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the EyePACS retinal image quality dataset where EZ-Sort reports its highest reliability."}],"review_version":2}