{"id":"50efe409-757d-4dba-be7b-840a39ed1d77","arxiv_id":"2507.17025","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Per-feature coordinate-search thresholds for binarizing BERT embeddings outperform fixed-threshold methods and occasionally match full-precision accuracy.","lead":"The authors use coordinate search to pick a different cut-off for every dimension of a BERT text embedding before converting the numbers to bits. They report that these binary codes beat fixed-threshold binarization and sometimes match full-precision embeddings while using far less memory.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split is described; reported accuracies likely compare threshold optimization and classifier evaluation on the same data, so the central outperformance claim is uninterpretable.","rationale":"The reader's stated weakest assumption is the hard-coded [-1,1] threshold search bound (Section 3.1). That is a genuine concern: if a feature dimension has values outside this range, the optimized threshold will produce a constant bit and discard information, and the paper provides no verification of the range claim. However, I judge the missing train/test split to be the single most load-bearing concern. The central claim is empirical: feature-wise thresholds outperform traditional binarization in accuracy. For that claim to be tested, accuracies must reflect generalization. The paper never describes a split, and the method description suggests thresholds are optimized using the same data on which the logistic regression is later evaluated. This would inflate all reported numbers, making the comparison with baselines and with real-valued embeddings meaningless. The reader's rationale mentions 'no train/test split is described' alongside other issues, but the reader's weakest_assumption field identifies the bounds issue instead. I therefore partially agree: the bounds issue is real but less fundamental than the evaluation-protocol gap. The concrete test I propose—rerunning with an explicit held-out split—would settle whether the reported improvements survive out-of-sample evaluation. If they do not, the paper's reject verdict stands; if they do, the authors would need to provide code and data to substantiate the claim. Thus the verdict remains unchanged from the reader's REJECT, because the missing split is enough to reject the paper's central empirical claim as currently presented.","tokens_in":19462,"tokens_out":3366,"duration_ms":36880,"concrete_test":"Require the authors to release code and data, or to rerun the full protocol with an explicit three-way split (e.g., 80/10/10): optimize thresholds on the training fold only, train the logistic regression on the binary training fold, and report accuracy on the untouched test fold for every method in Table 2. If the proposed method's test accuracy does not exceed all baseline binary methods, or if it no longer approaches or beats real-valued BERT embeddings, the central claim is unsupported. A minimal computational version of this test can be run on IMDb using standard BERT embeddings and the paper's Algorithm 1, with a single train/test split, to see whether the reported 87.84% accuracy is reproducible on held-out data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that feature-wise Coordinate Search thresholds produce binary embeddings that outperform traditional binarization and approach full-precision accuracy. Every accuracy reported in Table 2 and every post-hoc p-value depends on a valid evaluation protocol. The paper never states how data were split into training, validation, and test sets. Section 3.1 says the objective is 'classification accuracy' computed while optimizing thresholds, and Section 3 says the resulting binary embeddings are 'utilized to train a logistic regression classifier to evaluate the accuracy.' If the same samples used to optimize the 768 thresholds and train the logistic regression are also used to compute the reported accuracies, then the numbers are training-set fits, not estimates of generalization. This would explain the suspicious result in Table 2 that the binary proposed method (87.84%) beats real-valued BERT embeddings (86.68%) on IMDb, and would also explain why the proposed method shows 'significant' superiority over real embeddings in the post-hoc tests. The missing split is more load-bearing than the [-1,1] bound issue: even if every embedding dimension lies in [-1,1], the experimental comparison is invalid without a held-out test set. The anomalies the reader noted (duplicated Otsu values in Table 2, templated timings in Table 3) compound the concern but are secondary; the absence of a split is a fundamental flaw that invalidates the evidence for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Coordinate Search-based feature-wise thresholding method to convert real-valued BERT embeddings into binary 'barcode' representations. The method optimizes one threshold per feature by maximizing F1-score during a coordinate search, and the resulting binary embeddings are used to train a logistic regression classifier. Experiments on IMDb, GLUE SST-2, AG News, CoNLL-2003, and SNLI report median accuracies, memory and timing comparisons, plus Kruskal-Wallis and post-hoc statistical tests. The central claim is that the proposed per-feature thresholding outperforms traditional global thresholding methods and approaches or exceeds the accuracy of full-precision BERT embeddings while using substantially less memory.","tokens_in":19740,"tokens_out":4051,"duration_ms":41262,"significance":"If substantiated, per-feature threshold optimization for binary embeddings would be a practically useful contribution for memory-constrained NLP deployments, and framing threshold selection as a derivative-free per-coordinate optimization problem is a reasonable idea. However, the manuscript does not provide a valid evaluation protocol: no train/validation/test split is described, the reported accuracy numbers conflict with the accompanying statistical tests, and several tables contain duplicated or templated values. The paper ships no code or data, so the evidence for the central outperformance claim is not reliable. The underlying idea may merit future work, but this manuscript does not demonstrate it.","major_comments":[{"comment":"No train/validation/test split is described anywhere in the paper. Section 3.1 states that the CS algorithm evaluates the F1-score of candidate thresholds on the embeddings, and the end of Section 3 states that the resulting binary embeddings are 'utilized to train a logistic regression classifier to evaluate the accuracy.' If the same samples are used for threshold optimization, classifier training, and accuracy reporting, then all numbers in Table 2 are training-set fits rather than estimates of generalization. This is load-bearing because the central claim that the proposed method 'outperform[s] traditional binarization methods in accuracy' depends entirely on Table 2. The authors must specify the exact data split (e.g., 80/10/10 or k-fold), report metrics on a held-out test set, and ensure that thresholds and the classifier are fitted only on training data.","section":"Section 3.1 and Section 4.3"},{"comment":"The text and the statistical section claim that the proposed method achieved the highest median accuracy on all five datasets and significantly outperforms real-valued embeddings. Table 2, however, shows Real embeddings BERT higher on GLUE SST-2 (84.77 vs. 82.70), AG News (87.93 vs. 85.72), CoNLL (82.93 vs. 77.74), and SNLI (82.22 vs. 76.70). The Section 5 claim that the proposed method beats Real embeddings on every dataset is therefore internally inconsistent with the accuracy table. Additionally, the Otsu row repeats 75.05 for both GLUE SST-2 and AG News and repeats 64.95 for both CoNLL and SNLI, and the Optimization-based Otsu row repeats 68.45 for CoNLL and SNLI. These coincidences suggest data entry errors, and the accuracy evidence must be corrected and reconciled before any statistical claim can be assessed.","section":"Table 2 and Section 5"},{"comment":"The algorithm hard-codes the search bounds to L = -1 and U = 1 for every feature, with the justification that 'the BERT embeddings of our dataset lie within this range.' No verification is reported: the authors do not provide per-dimension minima and maxima, nor the fraction of values outside [-1,1] for any dataset. If any feature dimension has values outside this interval, the corresponding binary bit becomes constant (all 0 or all 1), silently discarding information, and the optimized threshold cannot adapt to the true data scale. This assumption is load-bearing for the method's claimed accuracy and efficiency, and it must be verified empirically.","section":"Section 3.1"},{"comment":"Table 3 reports proposed-method computation times of 110-150 ms depending on the dataset, but the text in Section 4.3 states 'as seen with methods like proposed at 50 ms across datasets.' The memory and timing columns also follow a visibly regular pattern (e.g., 4.98, 5.00, 5.02, ... MB and 120, 125, 130, ... ms for IMDb), which strongly suggests templated rather than measured values. The efficiency claim, which is a major advertised advantage of the method, is therefore unsupported by the reported data.","section":"Table 3 and Section 4.3"},{"comment":"The Kruskal-Wallis and post-hoc tests are applied to the same data used for threshold optimization and classifier training, so they inherit the circularity described in the first comment. Moreover, Tables 5-9 contain repeated identical p-values in unrelated cells (e.g., 7.485e-10 appears multiple times in Table 5, and 5.135e-6 appears multiple times in Table 6), and the claim that CS significantly outperforms Real embeddings on every dataset contradicts Table 2. These statistical results cannot be taken as evidence of robustness or generalizability.","section":"Section 5 and Tables 4-9"}],"minor_comments":[{"comment":"The text contains typos and duplicated words, including 'using using' in the abstract, 'acheive' in Section 1, 'intresting' in Section 1, and 'interation' in Section 3.1.","section":"Abstract and Section 1"},{"comment":"The reference list contains duplicates: Devlin et al. appears as both [9] and [34], Otsu appears as both [23] and [32], and Bidgoli and Rahnamayan appears as both [27] and [36]. These should be consolidated.","section":"References"},{"comment":"The dataset named 'SNIL' in Table 1 is written as 'SNLI' everywhere else in the paper; the table also lists GLUE-sst2 as 70,000 samples and does not clarify whether the reported numbers refer to full datasets or to train/test splits, which matters given the missing split description.","section":"Table 1"},{"comment":"The heatmaps use a color scale based only on -log10(p-value), so the row for Real embeddings appears bright even though Real loses comparisons; the text acknowledges this, but the visualization is misleading and should be redesigned, for example by showing signed effect sizes or direction-aware colors.","section":"Figure 4"}],"recommendation":"reject","confidential_remarks":"In my assessment, the manuscript contains multiple signs of unreliable data: duplicated values in Table 2, formulaic memory and timing values in Table 3, repeated p-values across unrelated cells in Tables 5-9, and direct contradictions between the accuracy table and the claimed statistical significance. These are not merely presentation issues; they call into question the integrity of the empirical results. If the authors can supply raw data and code with a properly held-out test split, a substantially revised submission might be worth considering, but in its current form the paper does not meet the standards for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper has a simple, sensible idea: instead of one global threshold for binarizing BERT embeddings, use coordinate search to pick a threshold per feature. That is a legitimate extension of existing binarization work, and the exposition of the CS algorithm is clear enough to re-implement.\n\nBut the experimental evidence is not trustworthy. No train/test split is described anywhere. The method fits 768 thresholds to the same data used to report accuracy, so the numbers in Table 2 are training-set fits, not estimates of generalization. That alone invalidates the central claim that per-feature CS beats global thresholds and even rivals real-valued BERT. The stress-test note is right: this is more load-bearing than the [-1,1] bound issue.\n\nThe oddities compound it. Table 2 has duplicated values in the Otsu row (75.05 for both GLUE and AG News, 64.95 for both CoNLL and SNLI). Table 3 looks templated: memory climbs by 0.01 MB per row, timings by 10 ms per row, while the text says the proposed method runs at 50 ms when the table says 110–150. These are not harmless typos; they suggest numbers filled in rather than measured.\n\nThe [-1,1] bound assumption is a softer issue: if any embedding dimension lies outside that range, the binary bit is constant and information is silently discarded. That is a real concern, but secondary to the missing split.\n\nWhat the paper does well: the idea is clearly stated, the algorithm is described in enough detail, and the ablation intent is thorough. But the execution undercuts it. A referee cannot assess whether the method works from the presented evidence.\n\nWho is this for? Someone curious about embedding compression might read it for the idea, but they should not use these numbers. I would desk reject in its current form. The authors need to redo the experiments with a proper held-out split, fix the tables, and release code before it deserves referee time.","headline":"Per-feature thresholding via coordinate search is a plausible idea, but the paper's experimental evidence is unreliable: no held-out split, duplicated baselines, and templated timings undercut the central claim.","tokens_in":20244,"tokens_out":2282,"would_cite":false,"duration_ms":23856,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that choosing a separate threshold for every embedding dimension, tuned by a derivative-free coordinate search, turns 768-dimensional BERT embeddings into binary codes that match or approach full-precision accuracy while…","keywords":["binary embeddings","barcode representation","coordinate search","feature-wise thresholding","BERT embeddings","text classification","embedding compression"],"falsifier":"Compute the per-dimension minimum and maximum of the BERT embeddings used in the experiments; if any dimension's range extends beyond $[-1,1]$, then the coordinate search is optimizing over a region that excludes the data, and the claim that thresholds are optimal for those features fails. A direct test is to rescale one dimension by a factor of 10 and rerun the method: a feature-wise optimizer with fixed bounds cannot compensate, so accuracy should drop relative to the unscaled run if the fixed-bound assumption is load-bearing.","tokens_in":19262,"feed_emoji":"💾","tokens_out":8156,"duration_ms":79010,"temperature":0.7,"pith_summary":"The paper tries to show that binarizing BERT embeddings does not have to cost accuracy if each of the 768 feature dimensions gets its own threshold, tuned by a derivative-free Coordinate Search directly on classification performance. Fixed-threshold methods, including optimized single-threshold variants, lose information, while per-feature thresholds let informative dimensions keep relaxed cut-points and noisy dimensions stricter ones. On IMDb the binary representation generated this way beats the full-precision BERT embedding (87.84% vs 86.68% median accuracy), and on four other NLP benchmarks it comes close while using roughly 30-fold less memory and faster inference. If the claim holds, binary embeddings become a practical drop-in replacement for float embeddings in memory-constrained applications, and the same recipe can be applied to any continuous feature representation.","feed_headline":"Per-feature thresholds push binary BERT past full-precision on IMDb","feed_subtitle":"Coordinate search per feature gets binary embeddings close to full-precision accuracy at 1-bit storage.","key_machinery":"The machinery is Coordinate Search (CS), a derivative-free optimization loop applied to the threshold vector. Starting from bounds $[-1,1]$ for every feature, the algorithm tests two center-based candidate thresholds in the current interval, keeps the one with better macro-F1 on a validation set, halves the interval, and repeats; a random permutation of dimensions orders the updates each iteration, and multiple runs start from fresh permutations. This yields $S^*$, a threshold per embedding dimension, and binarization is then $B_i = 1$ if the real value is at least $S^*[i]$, else 0. The objective being optimized is downstream classification F1, so the thresholds are tuned to the task, not to reconstruction error.","core_discovery":"The central claim is that per-feature thresholds found by Coordinate Search produce binary embeddings that preserve enough task-relevant information to match or nearly match full-precision BERT embeddings, while using one bit per value instead of 32. Concretely, the paper reports the highest median accuracy on IMDb (87.84% vs 86.68% for real BERT embeddings), competitive accuracy on GLUE SST-2, AG News, CoNLL-2003, and SNLI, and roughly 29-fold memory reduction (e.g., 4.98 MB vs 146.48 MB on IMDb). The authors argue the gain comes from giving each of the 768 dimensions its own cut-point: informative features get relaxed thresholds and noisy features get stricter ones, which a single global threshold cannot do.","pith_inferences":["A direct extension the paper does not pursue is to estimate each dimension's empirical range before searching, which would remove the fixed $[-1,1]$ search-space assumption and make the method applicable to embedding models whose values fall outside that interval.","The optimized threshold vector itself could be read as a feature-importance map: dimensions whose thresholds sit far from zero are the ones where the sign of the value is most discriminative, which could guide pruning or feature selection.","Because the objective is task accuracy rather than reconstruction error, the same coordinate-search binarization could be applied to vision or multimodal embeddings, with the expectation that per-feature cut-points will help wherever feature scales differ.","If per-feature thresholds really deliver near-full-precision accuracy, the practical ceiling for binary embeddings may be much higher than current fixed-threshold binarization suggests; testing on retrieval or ranking tasks would show whether the benefit extends beyond classification."],"forward_implications":["Binary embeddings from per-feature thresholds can replace real-valued BERT embeddings in memory-constrained deployments with little accuracy loss.","Feature-wise threshold optimization is strictly better than optimizing a single global threshold, as shown by the CS-Feature versus CS-Global comparison on every dataset.","The method applies to any continuous feature representation, since thresholding and coordinate search do not depend on the meaning of the features.","On IMDb, the binary representation exceeds full-precision BERT accuracy, so binarization is not necessarily a trade-off.","All binary methods in the reported table use dramatically less memory and compute than 32-bit embeddings, with the proposed method among the smallest and fastest."],"supporting_citations":[{"why":"Supplies BERT, the model that generates the continuous embeddings being binarized.","marker":"[34]"},{"why":"Defines Coordinate Search, the optimization method used to tune per-feature thresholds.","marker":"[24]"},{"why":"Provides the center-based sampling rationale for testing the midpoints of each interval.","marker":"[39]"},{"why":"Introduces MinMax thresholding, a per-feature baseline the proposed method must beat.","marker":"[22]"},{"why":"Provides Hybrid thresholding, a global baseline compared in the experiments.","marker":"[33]"},{"why":"Supplies Otsu thresholding, a variance-based global baseline compared in the experiments.","marker":"[41]"}],"fun_headline_variants":["Per-feature thresholds push binary BERT past full-precision on IMDb","Per-feature cut-points let binary BERT match full precision","Coordinate search tunes each BERT feature for efficient binary storage","Binary BERT with per-feature thresholds cuts memory 29x","Featurewise threshold search yields binary BERT with near-full precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes every embedding dimension's values fall inside the hard-coded search range $[-1,1]$; if a feature's values lie outside it, its binary bit becomes constant and the optimized threshold silently discards that dimension's information.","fun_headline_variants_meta":{"raw":{"variants":["Per-feature thresholds push binary BERT past full-precision on IMDb","Per-feature cut-points let binary BERT match full precision","Coordinate search tunes each BERT feature for efficient binary storage","Binary BERT with per-feature thresholds cuts memory 29x","Featurewise threshold search yields binary BERT with near-full precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001077,"raw_usage":{"total_tokens":4491,"prompt_tokens":913,"completion_tokens":3578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":3489}},"tokens_in":529,"tokens_out":3578,"duration_ms":27750,"temperature":1.0,"reasoning_tokens":3489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:57:19.402466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-dimension minimum and maximum of the BERT embeddings used in the experiments; if any dimension's range extends beyond $[-1,1]$, then the coordinate search is optimizing over a region that excludes the data, and the claim that thresholds are optimal for those features fails. A direct test is to rescale one dimension by a factor of 10 and rerun the method: a feature-wise optimizer with fixed bounds cannot compensate, so accuracy should drop relative to the unscaled run if the fixed-bound assumption is load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Otsu thresholding, a variance-based global baseline compared in the experiments."},{"cited_title":"Evolution and optimum seeking: The s ixth generation","cited_arxiv_id":null,"evidence_quote":"Defines Coordinate Search, the optimization method used to tune per-feature thresholds."},{"cited_title":"A comprehensive investigation on novel center-based sampling for large- scale global optimization","cited_arxiv_id":null,"evidence_quote":"Provides the center-based sampling rationale for testing the midpoints of each interval."},{"cited_title":"Jain and D","cited_arxiv_id":null,"evidence_quote":"Introduces MinMax thresholding, a per-feature baseline the proposed method must beat."},{"cited_title":"Hybrid thresholdi ng for image de- convolution in expectation maximization framework","cited_arxiv_id":null,"evidence_quote":"Provides Hybrid thresholding, a global baseline compared in the experiments."}],"review_version":1}