{"id":"040e8350-7027-47e8-ae64-99ca5aa4ec14","arxiv_id":"2504.17688","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new public dataset of 845,000 human pairwise similarity judgments for Hubble image cutouts, distilled into a distance matrix for benchmarking image retrieval.","lead":"The authors built a large dataset of human similarity judgments for 2,098 cutouts from Hubble Space Telescope images, using paid citizen scientists. It provides a benchmark for testing computer vision image search tools in astronomy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The distance matrix assumes context-independent similarity judgments, but phases 2/3 convert relative 'most similar' choices into absolute votes; this bias is untested and can shift pair orderings despite network redundancy.","rationale":"The central claim is that the HISP distance matrix is a valid benchmark for perceptual similarity. That claim depends critically on the preference matrix F being an unbiased reflection of pairwise similarity, which in turn depends on the independence assumption in Appendix A.1. The reader identified this as the weakest assumption, and I agree: it is load-bearing because the cosine-fingerprint distance (Eq. A7) is computed from F, and any systematic context bias in F propagates directly into D. The paper acknowledges the assumption is 'surely oversimplifications' but never quantifies the violation, even though the golden-question design provides a direct test. I add that the phase 2/3 version of this concern is more acute than the phase-1 version: the 'most similar' task is definitionally relative, and the algorithm's treatment of unchosen comparisons as negative votes can systematically penalize genuine near-neighbors whenever a slightly closer image is present. Because the phase 2/3 comparison sets are built from the distances themselves, the bias is correlated with local density and cannot be removed by the network averaging that handles random noise. The paper's internal consistency checks (Fig. 12, Fig. 13, t-SNE separation by object type) are suggestive but not dispositive: they mostly confirm that the distances correlate with the same reviews and with coarse morphological classes, which a biased distance measure could still do. The concrete golden-page test would settle whether a measurable context bias exists. If it does, the paper's claim that the distances 'faithfully reflect' subtle differences is not yet established, and the benchmark should carry uncertainty caveats; if it does not, the concern is mitigated. Since the dataset is public and valuable, and the requested revision is a targeted analysis rather than a rejection, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":23429,"tokens_out":10433,"duration_ms":104830,"concrete_test":"Using published phase1-results.fits, compare selection rates on golden pages (where the reference image is also one of the 15 comparisons) against non-golden pages for the same reference image, binning the other 14 non-golden comparison images by phase-1 or final similarity distance to the reference. If, for matched D_ij, the probability of selecting j is significantly lower on golden pages than on non-golden pages, the Appendix A.1 independence assumption is falsified and the preference matrix is context-biased. If the rates match, the phase-1 violation is not detected; the test should then be repeated on phase 2/3 data by checking whether the probability of selecting a comparison with a given final distance depends on the distances of the other comparison images on the page (e.g., logistic regression of selection on D_ij and D_min_other).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.1 assumes that a reviewer's probability of judging B similar to reference A is independent of the other comparison images on the page, and labels this 'surely oversimplifications.' In phase 1 the composition of each 15-image page is random, so the violation adds noise that the cosine-fingerprint distance (Eq. A7) can partially average down. The more serious violation is in phases 2 and 3, where the task is explicitly relative: reviewers choose the most similar image from a small set. The algorithm (Section A.3) converts each page into positive and negative votes for every comparison pair and updates Eq. A6 with a prior derived from earlier phases. But a negative vote for pair (A,B) can simply reflect that another image C on the page was even closer to A, not that B is dissimilar; similarly a positive vote can reflect a weak field rather than true similarity. Since the comparison sets are deliberately selected to be close to A (Sections 3.3-3.4), the local density of similar images is systematically correlated with the reference image, so this bias is not random. The network redundancy cancels independent noise, not density-dependent bias. The paper never tests the independence assumption, although the golden questions provide a natural control: the paper explicitly notes (end of A.1) that the reliability of secondary comparisons in golden questions 'might be useful to explore' but was not done. Convergence across phases (Fig. 13) is not decisive because phase 2/3 pages are selected using distances from earlier phases, so later phases mainly refine already-close neighborhoods. If the context bias is large, the advertised benchmark distances can reorder neighbors even when the aggregate statistics look stable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The Hubble Image Similarity Project (HISP) constructs and publicly releases a dataset of human similarity judgments on 2,098 grayscale HST cutout images drawn from 666 NGC objects (parent sample 19,916). Three review phases were run with paid Amazon Mechanical Turk reviewers recruited from the Baltimore area: Phase 1 exhaustively compared every image pair (single reviewer per page, 'select all similar' among 15 comparisons, with golden questions); Phase 2 used 'select most similar' pages of six comparisons drawn from the ~200 nearest neighbors; Phase 3 used pages of three comparisons from the ~36 nearest neighbors, with rotations and multiple reviewers. The paper's Appendix A presents an algorithm that converts votes into a preference matrix F via a Bayesian binomial estimator with power-law priors (Eq. A6) and then computes a masked cosine distance D_ij between preference 'fingerprints' (Eq. A7), yielding a fully populated symmetric distance matrix. The paper demonstrates convergence of the distances across phases (Fig. 13), shows that object types from the NGC catalog separate cleanly in a t-SNE embedding of the matrix (Figs. 14–15), and presents nearest-neighbor montages that the authors judge visually excellent (Fig. 16).","tokens_in":23721,"tokens_out":11335,"duration_ms":105863,"significance":"The released dataset is the central contribution, and it is a substantial one: nearly 850,000 paid human judgments on 2,098 carefully selected HST cutouts, with the images, review results, three stage-wise distance matrices, and the Python code all archived at MAST (doi:10.17909/0q3g-by85). Because the authors release the phase-level matrices separately, users can independently study convergence and can apply more sophisticated models (e.g., reviewer-bias or Bradley-Terry-type preference models) to the raw votes. The experimental design is strong in several respects: embedded golden questions with and without rotations, three independent reviewers in the later phases, avoidance of same-object comparisons, and the use of paid local reviewers under fair-wage conditions. The clean separation of NGC object types in the t-SNE projection (Fig. 15) is a genuinely independent anchor for the central claim, because object-type labels played no role in the image selection, reviews, or distance computation; it indicates that the distances carry real morphological information.","major_comments":[{"comment":"The conversion of phases 2 and 3 'most similar' selections into positive and negative votes treats every non-selected comparison as a negative vote for similarity. Because the comparison sets are deliberately restricted to close neighbors (N_n = 200 with D_min = 0.4 in phase 2; N_n = 36 with D_min = 0.21 in phase 3), a non-selection often means only that another image on the page was even closer, not that the pair is dissimilar; the bias is therefore correlated with the local density of similar images and is not removed by the network redundancy that cancels independent review errors (§4.1). Appendix A.1 flags the underlying context-independence assumption as a 'surely oversimplification' and notes that the golden questions could be used to test the reliability of secondary comparisons, but that test is not carried out. I ask for one of: (a) a quantitative test of page-composition effects using the golden questions; (b) a recomputation of the final distances with non-selections in phases 2/3 treated as missing rather than negative; or (c) a prominent caveat that the fine-grained distances among close neighbors may inherit this context bias.","section":"Appendix A.3, Eq. (A6); §§3.3–3.4"},{"comment":"The distance algorithm is tuned by experimentation on the same data ('we have a simple empirical approach that was determined via much experimentation on the data'), and the authors concede in §6 that a better algorithm would use cross-validation and that overfitting remains a concern. All of the quality demonstrations offered are in-sample: Fig. 13 shows convergence between phases rather than agreement with an external ground truth; the x-axis of Fig. 10 uses the final distance matrix that contains the same reviews whose consistency is being characterized; and Fig. 16's 'excellent results' rest on the authors' own visual examination, which §5 acknowledges is difficult to back up with data. The abstract's claims ('matches the accuracy of the trained eye', 'subtle differences among images faithfully reflected') therefore exceed what the presented evidence supports. I request at least one holdout-based validation (for example, building distances from a random half of pages or reviewers and testing on the remainder, or comparing the released phase-1-only matrix with the final matrix on the phase-2/3 pair set), and a recalibration of the abstract and §5 language to state explicitly that the validation is in-sample.","section":"Appendix A.3 and §6; Figs. 10, 13, 16"},{"comment":"The acknowledged error in constructing the phase 2 collection ('approximately half the image pairs were affected' by substitution of different cutouts from the same NGC object) propagates into the phase 2 preference prior, into the phase 3 neighbor selection, and hence into the final released matrix. The paper asserts that the impact is limited because the phase 2 net is broad and many substitutions are genuinely similar, but no quantitative test is provided. Given that the affected pairs are a large fraction of the phase 2 data, I request a sensitivity analysis (for example, recomputing distances with affected pairs excluded, or comparing the phase 3 neighbor sets obtained with and without the substituted pairs) and, at a minimum, a characterization of the resulting uncertainty in the MAST data documentation.","section":"Section 3.3"}],"minor_comments":[{"comment":"There is a typo in Section 4.2: 'The result is is much poorer consistency in ratings' should read 'The result is much poorer consistency in ratings.'","section":"Section 4.2"},{"comment":"Phrases such as 'The results are very impressive' (Abstract) and 'we find these results to be spectacular' (Section 5) are editorial in tone; I suggest replacing them with neutral statements of what the figures demonstrate, particularly in light of Major Comment 2.","section":"Abstract; Section 5"},{"comment":"Section 5.1 states that moderate changes in the t-SNE parameters lead to comparable results, but no quantitative support is given; I suggest reporting the parameter ranges tried and a quantitative measure of the resulting variation in the coordinates.","section":"Section 5.1"},{"comment":"Appendix A.3 notes that phase 2/3 votes are collected only in the direction where the reviewed image is the reference; a sentence explaining how this asymmetry enters the masked cosine distance in Eq. (A7) (which pairs are downweighted or lost) would help users of the released matrices.","section":"Appendix A.3"},{"comment":"The entropy cut in Section 2 (both entropy values greater than 5) removes roughly 60% of the parent sample; Section 5 should state explicitly that the distance distribution and statements such as 'the loneliest image' describe this high-contrast selected sample rather than HST imaging in general.","section":"Section 2; Section 5"},{"comment":"Phase 1 golden questions are included in the distance calculation while golden-question accuracy is also reported as a reviewer-quality diagnostic; the fraction is small (3%) and the impact likely minor, but a one-sentence justification or sensitivity check would remove a residual circularity concern.","section":"Appendix A.1"},{"comment":"The speculation that 'the t-SNE distances actually give better results than the similarity matrix distances' conflicts with the paper's framing of the matrix as the primary product; since the authors state they cannot substantiate the claim, I suggest either deleting it or explicitly labeling it as an untested hypothesis.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a data-product paper of genuine community value, and the authors' honesty about their algorithm's heuristic character is a strength. My recommended changes are intended to align the claims with the evidence: the in-sample tuning and the untested context-independence assumption in the later phases affect the confidence one can place in the fine-grained distances, which are precisely the product's selling point. The phase-separated releases mean the requested sensitivity checks are feasible with modest effort. I also note that the manuscript header states it is already accepted for publication in the Astronomical Journal; if the editorial process is complete, the abstract-calibration issue (Major Comment 2) and the context-bias caveat (Major Comment 1) could in principle be handled via erratum or documentation, but I would prefer to see them addressed in the paper itself. No concerns about authorship or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read of White & Peek. The real product is the dataset: 845k human comparisons across 2,098 HST cutouts, with distance matrices, code, and everything archived at MAST. That is new. Earlier retrieval papers (Peek 2020, Hayat 2021, Seo 2023) built CNNs but had no human-similarity ground truth to validate against. This gives the field that resource.\n\nThe paper does a lot right. The three-phase design is thoughtful: exhaustive but noisy phase 1, then phases 2/3 targeting close neighbors with multiple reviewers. Golden questions are a good check, and the error rates behave sensibly (harder in phase 3, worse with rotation). The masked cosine distance over preference vectors is a sensible way to average down noise. The t-SNE separation by NGC object type is independent evidence that the distances carry morphology. The authors are also honest: they flag the phase 2 construction error, admit the algorithm is empirical, and acknowledge overfitting without error bars. The citation pattern is appropriate; the prior work they cite is exactly what this benchmark fills a gap for.\n\nThe soft spots are real but not fatal. The strongest concern is the context-independence assumption in Appendix A.1. In phases 2 and 3, reviewers pick the most similar image from a small set. Converting that to positive/negative votes for each pair treats a 'no' as evidence of dissimilarity, when it often just means another image was closer. Because the comparison sets are deliberately dense near the reference, the bias is systematic, not random; network redundancy cancels noise, not a density-dependent shift. The authors note the golden questions 'might be useful' for this and then don't use them. That's a missed control. Separately, the validation is partly circular: the distance matrix is built from the same reviews used to demonstrate its quality, and the 'excellent results' in Fig. 16 are the authors' own eyes. Independent signals exist (NGC types, golden-question error vs distance), so the core claim probably survives, but the paper would be stronger with an external check, e.g., comparing against a small set of expert similarity ratings, or leaving some phase 1 data out when building distances.\n\nNone of this changes my recommendation. The dataset is too valuable to desk reject. Send it to review, but push the authors to test the independence assumption quantitatively and to provide uncertainty estimates or a held-out validation. The paper is for astro-informatics and archive-search people, and they will use it.","headline":"A genuinely useful human-similarity benchmark for HST images, with honest but under-tested aggregation; the dataset deserves publication, with fixes to the context-dependence validation.","tokens_in":24299,"tokens_out":2263,"would_cite":true,"duration_ms":22991,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Crowd judgments of Hubble images yield a human similarity benchmark for image search.","keywords":["image similarity","Hubble Space Telescope","citizen science","crowd-sourced reviews","distance matrix","astronomical morphology","benchmark dataset","t-SNE"],"falsifier":"Embed the same reference-comparison pair in pages of near-twin distractors and in pages of unrelated distractors, and measure whether the inferred preference for that pair shifts with context. If the positive-vote rate depends on page composition, the aggregated preference matrix—and every fingerprint distance built from it—carries a systematic bias.","tokens_in":23214,"feed_emoji":"🔭","tokens_out":7945,"duration_ms":74580,"temperature":0.7,"pith_summary":"The paper claims that nearly 850,000 quick human comparisons of small grayscale cutouts from archival Hubble images can be distilled into a reliable matrix of similarity distances for 2,098 images. The goal is to give computer-vision image search a human ground truth: astronomical archives are easily searched by metadata, but almost no labeled data say which images actually look alike. The authors show the derived distances separate galaxies from nebulae and star clusters, order spiral and elliptical galaxies on distinct branches, and group subtle textural families such as sharp-edged dust lanes together, even though object types were never shown to the reviewers. If the paper is right, the public distance matrix is a ready-made benchmark for scoring and improving search-by-image algorithms.","feed_headline":"845,000 human votes now benchmark Hubble image search","feed_subtitle":"A similarity matrix built from crowd reviews separates galaxies, nebulae, and star clusters by appearance.","key_machinery":"The carrying mechanism is the preference matrix $F$ plus the masked cosine distance computed from its rows. Each entry $F_{ij}$ is a Bayesian estimate—from aggregated positive and negative votes, with a per-object prior—of the probability that a reviewer would call images $i$ and $j$ similar. The distance $D_{ij}$ is the cosine distance between the two preference fingerprint vectors, ignoring missing entries; this network-fingerprint comparison is what turns 850,000 noisy page-level judgments into a complete $2098 \\times 2098$ matrix. Because every distance depends on thousands of indirect comparisons rather than a single vote, random errors and layout bias largely average out.","core_discovery":"The central claim is that indirect, overlapping human votes carry enough information to reconstruct a low-noise perceptual metric. Each image sits in a comparison network spanning roughly two thousand other images, so the pattern of votes an image attracts—its preference fingerprint—matters more than any single review. The paper estimates a Bayesian preference value for every pair, builds the preference matrix $F$, and defines the similarity distance between two images as the masked cosine distance between their fingerprints; two images count as similar when they vote the same way toward the surrounding network, not merely when one reviewer clicked both. The resulting distances separate the sample by object class and Hubble type, and visual inspection of nearest neighbors shows genuine agreement in morphology and texture, including hard-to-describe features.","pith_inferences":["A natural extension is to re-run the same fingerprint construction on data where reviewers rank images instead of choosing 'similar' or 'most similar,' which would isolate context effects from true preference.","The authors note that t-SNE neighbors sometimes look more convincing than raw-matrix neighbors; if so, a cross-validated version of the distance matrix would be an even stronger benchmark.","A testable prediction follows: a retrieval algorithm trained to reproduce these human distances should outperform models trained on object labels or metadata in appearance-based searches.","Since reviewers show a small but significant bias toward images placed near the reference, future crowd benchmarks should randomize layout or explicitly model position effects."],"forward_implications":["Computer-vision search systems can be scored by comparing their nearest-neighbor lists with the ordering of this human distance matrix.","The three-phase review design—exhaustive broad voting, followed by close-neighbor refinement, followed by rotated repeated comparisons—can be transplanted to other image domains.","The matrix is complete even though most image pairs were never directly compared, so it can feed dimensionality-reduction, clustering, and retrieval benchmarks directly.","The repeated and rotated reviews, together with golden-question accuracy between 84 and 96 percent, give benchmark users a built-in estimate of label noise.","Because distances were built without object-type labels, agreement with galaxy and nebula taxonomy is an independent check that the metric captures real visual structure."],"supporting_citations":[{"why":"Supplies the approval-voting framework used in phase 1, where reviewers select all similar images.","marker":"Brams & Fishburn 1978"},{"why":"Provides the t-SNE embedding used to display and verify that distance clusters match morphology.","marker":"van der Maaten & Hinton 2008"},{"why":"Representative transfer-learning search-by-image system that the benchmark is built to validate.","marker":"Peek et al. 2020"},{"why":"Self-supervised galaxy morphology method whose similarity judgments this benchmark could test.","marker":"Hayat et al. 2021"},{"why":"Convolutional autoencoder galaxy similarity search that would be scored against the human distances.","marker":"Seo et al. 2023"},{"why":"Shows the same validation gap in earth-science image search, motivating the benchmark.","marker":"Koul et al. 2020"},{"why":"Provides the kappa statistic used to quantify reviewer consistency and show judgments are reliable.","marker":"Fleiss 1971"},{"why":"Supplies the object catalog from which the diverse sample of galaxies, nebulae, and clusters was drawn.","marker":"Verga 2017"}],"fun_headline_variants":["Crowdsourced votes map Hubble's visual universe","845k human clicks capture Hubble's subtle similarities","Hubble similarity matrix from a swarm of citizen votes","Crowd's eye rivals experts on Hubble image matching","One network of votes reveals Hubble's perceptual space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire distance construction assumes that a reviewer's chance of calling image B similar to reference A does not depend on which other images are shown on the same page; the authors themselves call this assumption 'surely oversimplifications.'","fun_headline_variants_meta":{"raw":{"variants":["Crowdsourced votes map Hubble's visual universe","845k human clicks capture Hubble's subtle similarities","Hubble similarity matrix from a swarm of citizen votes","Crowd's eye rivals experts on Hubble image matching","One network of votes reveals Hubble's perceptual space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1223,"prompt_tokens":841,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":457,"tokens_out":382,"duration_ms":4611,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:34:04.691241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Embed the same reference-comparison pair in pages of near-twin distractors and in pages of unrelated distractors, and measure whether the inferred preference for that pair shifts with context. If the positive-vote rate depends on page composition, the aggregated preference matrix—and every fingerprint distance built from it—carries a systematic bias.","supporting_citations":[{"cited_title":"2008, Journal of Machine Learning Research, 9, 2579","cited_arxiv_id":null,"evidence_quote":"Provides the t-SNE embedding used to display and verify that distance clusters match morphology."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Representative transfer-learning search-by-image system that the benchmark is built to validate."},{"cited_title":"SpaceML: Distributed Open-source Research with Citizen Scientists for the Advancement of Space Technology for NASA","cited_arxiv_id":"2012.10610","evidence_quote":"Shows the same validation gap in earth-science image search, motivating the benchmark."},{"cited_title":"2017, OpenNGC, The GAVO DC team, 10.21938/Y.1EJWUD_MQ6B_EDFOVBBW","cited_arxiv_id":null,"evidence_quote":"Supplies the object catalog from which the diverse sample of galaxies, nebulae, and clusters was drawn."}],"review_version":1}