{"id":"98a51c58-1d7f-463f-8a32-6e9e9ce02ebe","arxiv_id":"2412.01555","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark shows FAISS PQ reaches 98.4% retrieval precision in 0.24 MB, while Annoy gives fastest queries at about 16 microseconds with slightly lower accuracy.","lead":"This paper benchmarks two approximate nearest neighbor libraries, FAISS and Annoy, using image embeddings from a fine-tuned ResNet50 on a fashion image dataset. It reports trade-offs between retrieval accuracy, speed, and memory, with FAISS Product Quantization reaching 98.4% precision at a small index size and Annoy offering the fastest queries.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's FAISS rows are physically impossible: Flat-L2 index reported as 1.17 MB and query time 1.67 s for 44k images, while the same table lists precision 0.0003 instead of the 0.984 claimed in prose. The central quantitative claim rests on this table and is not reproducible as printed.","rationale":"The paper's stated contribution is a quantitative trade-off map for FAISS and Annoy on fine-tuned embeddings, and the abstract's headline numbers are the entire result. Those numbers come from Table 2, and Table 2 fails elementary plausibility checks. The Flat-L2 row reports precision 0.00031565 while the text and Figure 6 say 0.984; it reports a 1.17 MB index for what must be a ≈364 MB uncompressed float32 embedding store; and it reports a 1.67 s average query time for a method the same paper lists at 803 QPS. The identical 1.67 s value also appears for FAISS (IP), suggesting a unit or formatting error, but no raw logs or code are supplied to correct it. Because every downstream comparison—PQ versus Annoy, HNSW versus Angular, and the memory/speed trade-offs—is derived from this table, the central quantitative claim is not supported as printed. The reader's formal weakest assumption is label noise in the 32-class split, which is a legitimate secondary concern, but it is not the most load-bearing issue: even a perfect ground truth cannot rescue a table whose FAISS rows contradict the paper's own figures and known FAISS behavior. A corrected table or released code could change the assessment, so the verdict remains reject with low confidence, unchanged from the reader.","tokens_in":16810,"tokens_out":5935,"duration_ms":46005,"concrete_test":"Run the authors' stated pipeline on the same Kaggle fashion dataset: fine-tune ResNet50, extract embeddings, build FAISS IndexFlatL2 over the 44,446 embeddings, and query the same 1,000 test images with top-6 neighbors. For float32 embeddings of dimension d (2048 if using the penultimate FC layer), index size must be about 44,446*d*4 bytes; for d=2048 that is ≈364 MB, not 1.17 MB. Average query time on CPU for 44k vectors should be on the order of milliseconds, not 1.67 s. Also recompute precision as the fraction of top-6 neighbors sharing the query's subcategory label; if it is 0.98 rather than 0.0003, Table 2 as printed is not the source of the reported numbers. This single-row check determines whether the abstract's PQ/Annoy comparison is trustworthy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline numbers—FAISS PQ at 98.40% precision with 0.24 MB index, and Annoy at 0.00015 s query time—come entirely from Table 2. That table is internally inconsistent. For FAISS (L2), the row lists memory usage 0.0000763 MB and precision 0.00031565, yet the same row gives recall 0.98 and F1 0.984, and Section IV-C reports FAISS L2 precision as 0.984. A flat L2 index over 44,446 ResNet50 embeddings (d=2048, float32) requires N*d*4 ≈ 364 MB, not the 1.17 MB index size listed. The same table reports average query time 1,671,185 µs (1.67 s) for both FAISS (L2) and FAISS (IP), while Figure 6 reports 803.3 QPS for FAISS (L2), which is about 1.2 ms per query. FAISS (IP) is listed with precision 0.38 although Section IV-C and Figure 6 describe it as fast but low-precision, and other rows show precision values around 0.0001 for methods the text claims reach 0.97–0.98. These are order-of-magnitude contradictions among the table, the figures, and the prose, not minor transcription slips. Because no code, raw logs, error bars, or exact-search ground-truth comparison are provided, the published precision, recall, memory, and latency numbers cannot be audited or corrected by the reader. The qualitative trade-off may be plausible, but the quantitative claim that the paper advertises is unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript benchmarks FAISS and Annoy for content-based image retrieval on a custom fashion dataset, after fine-tuning a ResNet50 to produce feature embeddings. It reports indexing time, memory usage, query time, precision, recall, F1-score, and Recall@5 for several FAISS index types (Flat-L2, Flat-IP, PQ, IVF-PQ, IVF-SQ, LSH, HNSW) and three Annoy distance metrics (angular, Euclidean, Manhattan). The central advertised claims are that FAISS PQ reaches 98.40% precision with a 0.24 MB index, and that Annoy is fastest with ~0.00015 s average query time. The paper concludes that Annoy angular offers the best overall balance for speed and memory constrained scenarios, while FAISS HNSW or PQ are preferable when precision is paramount.","tokens_in":17210,"tokens_out":2899,"duration_ms":27308,"significance":"The paper addresses a practically relevant question: how fine-tuned deep features interact with ANN index structures in an end-to-end retrieval pipeline. A careful benchmark comparing FAISS and Annoy on the same embeddings with a consistent evaluation protocol would be useful guidance for practitioners. The qualitative trade-off direction (PQ is compact and accurate, Annoy is fast) is plausible and worth stating. However, the quantitative support is not trustworthy. The central Table 2 is internally inconsistent, contradicts the prose and figures in order-of-magnitude ways, and reports physically impossible index sizes. Because the main claims are exactly the numbers in that table, and because no code, logs, or exact-search ground truth are provided, the benchmark's quantitative conclusions are currently unsupported.","major_comments":[{"comment":"Table 2 is internally inconsistent for the FAISS rows. For example, the row labeled 'Faiss (L2)' lists Precision 0.00031565 while simultaneously listing Recall 0.98 and F1-score 0.984. Since F1 is the harmonic mean of precision and recall, it cannot exceed both inputs, and a precision of 0.0003 would imply an F1 near 0.0006, not 0.984. The same pattern appears for FAISS (PQ), (IVF-PQ), (IVF-SQ), and (HNSW). These numbers are not merely transcription slips; they invalidate the headline precision values reported in the abstract, Section IV-C, and Figure 6.","section":"Table 2 (Section IV-D)"},{"comment":"The reported index sizes and query times for FAISS Flat-L2 are physically impossible for the stated data. With 44,446 images and 2048-dimensional float32 embeddings, a flat exact index requires N*d*4 ≈ 364 MB, not the 1.17 MB shown. The same row reports an average query time of 1,671,185 µs (1.67 s) for both 'Faiss (L2)' and 'Faiss (IP)', yet Section IV-C and Figure 6 report that FAISS (L2) achieves 803.3 QPS (about 1.2 ms per query). These order-of-magnitude contradictions between the table, figure, and prose mean that the precision/memory/latency trade-off claims in the abstract are unreproducible as printed.","section":"Table 2 and Section IV-C (FAISS index size and query time)"},{"comment":"The prose in Section IV-D states that 'Annoy angular achieved 53 queries per second' while the same section later reports an average query time of 16 µs. These two numbers are mutually inconsistent: 16 µs per query corresponds to roughly 62,500 QPS, not 53 QPS. If the 53 QPS figure is correct, the query time should be about 18,868 µs, not 16 µs. Because the paper's main speed claim for Annoy is 0.00015 s in the abstract, this contradiction directly affects a central result.","section":"Section IV-D (Annoy query speed)"},{"comment":"No reproducibility artifacts are provided: there is no code, no raw result logs, no random seed, no error bars or standard deviations across multiple runs, and no comparison against an exact nearest-neighbor baseline. Given that the reported numbers are not self-consistent and appear to mix different measurement units (e.g., memory values near 0.0001 MB for a 44k-vector index), the reader cannot determine which entries are correct or reconstruct the actual benchmark. This is a load-bearing gap because the paper's conclusions are exclusively quantitative.","section":"Evaluation Protocol (Section III-D); Results (Section IV)"},{"comment":"The evaluation uses the 32 merged/split subcategory labels as exact relevance ground truth and a single random sample of 1,000 queries, with no sensitivity analysis. While this is a common practical choice, the merging of visually similar categories (e.g., Scarves/Mufflers, Topwear/Shirts) and the absence of any label-noise or query-sampling robustness check means that every reported precision/recall/F1 value could shift if the label definitions or query set change. This concern is secondary to the internal numeric inconsistencies, but it should be addressed if the benchmark is to support generalizable guidance.","section":"Dataset preprocessing (Section III-A) and evaluation protocol (Section III-D)"}],"minor_comments":[{"comment":"The formula labeled 'Recall' is mislabeled as 'Precision = TP/(TP+FN)' in the rendered text; it should read 'Recall = TP/(TP+FN)'.","section":"Section III-B"},{"comment":"The acronym 'ANN' is used for both 'Artificial Neural Network' (Introduction) and 'Approximate Nearest Neighbor' (throughout the rest of the paper). This is confusing in a paper about ANN search with a fine-tuned neural network; consider using 'ANN search' or 'ANNS' consistently.","section":"Throughout the manuscript"},{"comment":"Reference [18] duplicates [12] (ParlayANN), and [37] duplicates [17] (Malkov and Yashunin). Please deduplicate.","section":"References"},{"comment":"The paper states the model was fine-tuned over 10 epochs in Figure 1 and in the text, but Table 1 lists 11 epochs; also the caption says '10 epochs' while the table has 11 rows. Please make the epoch counts consistent and check corresponding validation metrics.","section":"Figures 3, 7, and Table 1"},{"comment":"The prose in Section IV-B says Angular achieves 'a precision of 0.9443' at 5 neighbors, but Table 2 lists Annoy (Angular) precision as 0.9357. Similarly, the text in Section IV-D reports 93.56% precision for Annoy angular while Table 2 gives 0.93565575; either round consistently or clarify the discrepancy.","section":"Section IV-B and Table 2"}],"recommendation":"reject","confidential_remarks":"The manuscript's core experimental table (Table 2) contains a pattern of precision values that are incompatible with the recall/F1 values in the same rows, and several indices have physically impossible size/latency figures. The contradictions are not localized typos: they affect every advertised headline number (FAISS PQ 98.40%, Annoy 0.00015 s). Because the paper's only contribution is this empirical comparison, and because the underlying measurements are not auditable without code/logs, I cannot see a revision path short of re-running the entire benchmark, which is beyond a normal revision. The qualitative trade-off insight is worth preserving, but the current manuscript, as written, does not support its stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the qualitative story—FAISS PQ is accurate and compact, Annoy is fast—is almost certainly right and consistent with a decade of ANN benchmarking. But the paper's advertised numbers come from Table 2, and Table 2 is broken in ways that go far beyond typos. The FAISS L2 row reports 0.0003 precision in the table while the prose and Figure 6 say 0.984. It lists memory usage of 0.0000763 MB (about 76 bytes) for a 44k × 2048 float32 index, and an index size of 1.17 MB when 44,446 × 2048 × 4 bytes ≈ 364 MB. It also gives a 1.67 s average query time for L2 while Figure 6 shows 803 QPS (~1.2 ms). These are not transcription slips; they are order-of-magnitude contradictions between the table, the figures, and the text. Since no code, raw logs, or exact-search ground truth are provided, the reader cannot tell which number, if any, is real.\n\nWhat is genuinely useful here: the paper benchmarks a reasonable spread of index types (Flat-L2, Flat-IP, PQ, IVF-PQ, IVF-SQ, LSH, HNSW, and Annoy with three distances) on a public dataset, with a fine-tuned ResNet50. The experimental protocol is described in enough detail that someone could reproduce it if the code were released. The figures give a broad visual sense of the speed–accuracy trade-offs, and the discussion of when to prefer Annoy angular versus FAISS HNSW/PQ is sensible and matches known library behavior.\n\nThe softer concerns: the split is stated as 90:10:10, which sums to 110%—clearly a typo, but symptomatic. Ground truth is the 32-class subcategory label, with some merged and split categories; label noise would shift precision/recall, though probably not the qualitative ranking. And the paper's limitations section is honest about single-dataset scope, which is fair.\n\nBottom line: this is a routine benchmark, not a new algorithm or theory. That is fine for a workshop-style empirical report, but the contribution lives or dies by its numbers, and the numbers as printed cannot be audited or believed. A revised version that fixes the table, releases code and raw measurements, and adds an exact-search baseline would be worth a look. As is, I would not cite it, and I would not put it in a reading group except as an example of why results tables need sanity checks. That said, the underlying experiment is real and fixable, so I would send it to a referee rather than desk-reject outright, with a strong request for code and corrected data.","headline":"Plausible qualitative trade-offs, but the headline numbers rest on a results table with order-of-magnitude internal contradictions; not publishable as-is.","tokens_in":17720,"tokens_out":3178,"would_cite":false,"duration_ms":24775,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a fashion image retrieval benchmark, FAISS product quantization hits 98.4% precision with a 0.24 MB index while Annoy answers queries in about 16 microseconds.","keywords":["approximate nearest neighbor search","FAISS","Annoy","image retrieval","ResNet50 fine-tuning","product quantization","fashion product dataset","benchmark"],"falsifier":"Recompute the same benchmark with ground truth defined by the original 44 category labels or by human relevance judgments; if the FAISS PQ precision lead over Annoy angular falls from its reported roughly 4.8 points to near zero, the trade-off is an artifact of the 32-class label merge or the query sample.","tokens_in":16626,"feed_emoji":"🛍️","tokens_out":10965,"duration_ms":85362,"temperature":0.7,"pith_summary":"The paper asks how much the approximate-nearest-neighbor (ANN) index, rather than the feature extractor, determines end-to-end image retrieval performance. It fine-tunes a ResNet50 on a 32-class fashion product dataset and feeds the same embeddings into seven FAISS index types and three Annoy configurations. The central result is a concrete trade-off: FAISS product quantization reaches 98.4% precision with a 0.24 MB index, while Annoy's angular index is the fastest at roughly 16 microseconds per query with 93.6% precision and 95.4% recall at $k=5$. A sympathetic reading is that the paper establishes a per-application selection rule: choose Annoy when latency is critical and FAISS PQ or HNSW when accuracy dominates.","feed_headline":"Hits 98.4% precision in a 0.24 MB index with FAISS","feed_subtitle":"Fashion-image benchmark shows Annoy answers queries in 16 microseconds, trading about five points of precision.","key_machinery":"The machinery is a controlled end-to-end benchmark pipeline: a fine-tuned ResNet50 with its classification head removed produces one embedding per image, and the same embedding set is indexed by every ANN configuration. The named objects are the index types: FAISS Flat-L2, Flat-IP, product quantization (PQ), IVF-PQ, IVF-SQ, LSH, and HNSW, plus Annoy with angular, Euclidean, and Manhattan distances. Product quantization works by splitting each vector into subvectors and quantizing each part into a compact code; this is the mechanism that lets PQ match Flat-L2's precision while shrinking the index to 0.24 MB. Because every method receives identical inputs, the benchmark isolates the contribution of the index and the distance metric.","core_discovery":"On a custom 32-class fashion product dataset, after fine-tuning a ResNet50 to 98.25% validation accuracy, the paper claims that the index configuration, not the embedding alone, decides the practical trade-off among speed, accuracy, and memory. The benchmark shows FAISS Flat-L2 and FAISS product quantization both reach a precision of 0.984 at $k=5$, with PQ doing so at a 0.24 MB index; FAISS HNSW posts the highest recall (98.5%) at $k=5$, while Annoy with the angular metric reaches 93.6% precision and 95.4% recall at $k=5$ with an average query time near 16 microseconds. The claim is that no single method dominates: PQ is for memory-constrained high-accuracy pipelines, Annoy angular is for latency-critical ones, and HNSW is for recall-critical ones.","pith_inferences":["The paper leaves untested how much of the 4.8-point precision gap would close if Annoy's tree count were raised above the fixed value of 10.","Because the benchmark defines relevance by the merged 32 labels, a re-run against the original 44 categories could shrink or reverse the FAISS-over-Annoy precision lead.","The reported memory and index sizes are for one dataset split; on a full catalog, HNSW's graph structure should consume memory faster than PQ, so the resource ranking may not scale.","The low Recall@5 numbers around 0.51 suggest the top-6 voting rule underuses the recall available in the indexes, so a different voting scheme might change which method wins."],"forward_implications":["Memory-constrained search over fashion images can use FAISS PQ to get near-exact precision at a 0.24 MB index.","Latency-critical retrieval can use Annoy angular, accepting roughly five points of precision for microsecond queries.","At $k=5$, exact and compressed FAISS indexes are effectively interchangeable, so quantization does not cost accuracy at small neighbor counts.","FAISS methods degrade more than Annoy as the neighbor count grows, so the choice of $k$ should be tuned together with the index family.","For recall-critical analytics, HNSW is the strongest configuration, at the cost of the largest index and slowest query time."],"supporting_citations":[{"why":"defines the FAISS library and its index types (Flat, PQ, IVF, HNSW) that the benchmark evaluates.","marker":"[7]"},{"why":"defines the Annoy tree-based index and the distance metrics used in the comparison.","marker":"[8]"},{"why":"introduces product quantization, the compression method behind the 0.24 MB index with high precision.","marker":"[13]"},{"why":"introduces the HNSW graph index used as the high-recall configuration.","marker":"[17]"},{"why":"supplies the ANN benchmarking approach that shapes the evaluation protocol.","marker":"[20]"},{"why":"provides the fashion product image dataset that the 32-class benchmark is built from.","marker":"[28]"},{"why":"supplies the optimizer used to fine-tune the ResNet50 feature extractor.","marker":"[29]"}],"fun_headline_variants":["Annoy answers in 16 µs, FAISS PQ hits 98.4% precision","Index choice, not just embeddings, sets retrieval trade-offs","FAISS PQ: 98.4% precision with 0.24 MB index","Speed vs. precision: Annoy 16µs, FAISS PQ 98.4%","98.4% precision in 0.24 MB: FAISS PQ wins memory duel"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 32 hand-merged subcategory labels are exact relevance and that the 1,000 randomly sampled queries represent the retrieval task.","fun_headline_variants_meta":{"raw":{"variants":["Annoy answers in 16 µs, FAISS PQ hits 98.4% precision","Index choice, not just embeddings, sets retrieval trade-offs","FAISS PQ: 98.4% precision with 0.24 MB index","Speed vs. precision: Annoy 16µs, FAISS PQ 98.4%","98.4% precision in 0.24 MB: FAISS PQ wins memory duel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2975,"prompt_tokens":909,"completion_tokens":2066,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1955}},"tokens_in":525,"tokens_out":2066,"duration_ms":13217,"temperature":1.0,"reasoning_tokens":1955,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:37.876202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the same benchmark with ground truth defined by the original 44 category labels or by human relevance judgments; if the FAISS PQ precision lead over Annoy angular falls from its reported roughly 4.8 points to near zero, the trade-off is an artifact of the 32-class label merge or the query sample.","supporting_citations":[{"cited_title":"GitHub - spotify/annoy: Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk","cited_arxiv_id":null,"evidence_quote":"defines the Annoy tree-based index and the distance metrics used in the comparison."},{"cited_title":"Curator: Efficient Indexing for Multi-Tenant Vector Databases","cited_arxiv_id":"2401.07119","evidence_quote":"introduces the HNSW graph index used as the high-recall configuration."},{"cited_title":"Fashion Product Images Dataset","cited_arxiv_id":null,"evidence_quote":"provides the fashion product image dataset that the 32-class benchmark is built from."}],"review_version":1}