REVIEW 5 major objections 4 minor 42 references
Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddings
T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that choosing a separate threshold for every embedding dimension, tuned by a derivative-free coordinate search, turns 768-dimensional BERT embeddings into binary codes that match or approach full-precision accuracy while…
desk verdict Per-feature thresholding via coordinate search is a plausible idea, but the paper's experimental evidence is unreliable: no held-out split, duplicated baselines, and templated timings undercut the central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is Coordinate Search (CS), a derivative-free optimization loop applied to the threshold vector. Starting from bounds $[-1,1]$ for every feature, the algorithm tests two center-based candidate thresholds in the current interval, keeps the one with better macro-F1 on a validation set, halves the interval, and repeats; a random permutation of dimensions orders the updates each iteration, and multiple runs start from fresh permutations. This yields $S^*$, a threshold per embedding dimension, and binarization is then $B_i = 1$ if the real value is at least $S^*[i]$, else 0. The objective being optimized is downstream classification F1, so the thresholds are tuned to the task, not to reconstruction error.
What would settle it
Compute the per-dimension minimum and maximum of the BERT embeddings used in the experiments; if any dimension's range extends beyond $[-1,1]$, then the coordinate search is optimizing over a region that excludes the data, and the claim that thresholds are optimal for those features fails. A direct test is to rescale one dimension by a factor of 10 and rerun the method: a feature-wise optimizer with fixed bounds cannot compensate, so accuracy should drop relative to the unscaled run if the fixed-bound assumption is load-bearing.
Extended reading notes
Core claim
The central claim is that per-feature thresholds found by Coordinate Search produce binary embeddings that preserve enough task-relevant information to match or nearly match full-precision BERT embeddings, while using one bit per value instead of 32. Concretely, the paper reports the highest median accuracy on IMDb (87.84% vs 86.68% for real BERT embeddings), competitive accuracy on GLUE SST-2, AG News, CoNLL-2003, and SNLI, and roughly 29-fold memory reduction (e.g., 4.98 MB vs 146.48 MB on IMDb). The authors argue the gain comes from giving each of the 768 dimensions its own cut-point: informative features get relaxed thresholds and noisy features get stricter ones, which a single global threshold cannot do.
Load-bearing premise
The method assumes every embedding dimension's values fall inside the hard-coded search range $[-1,1]$; if a feature's values lie outside it, its binary bit becomes constant and the optimized threshold silently discards that dimension's information.
Editorial extensions
If this is right
- Binary embeddings from per-feature thresholds can replace real-valued BERT embeddings in memory-constrained deployments with little accuracy loss.
- Feature-wise threshold optimization is strictly better than optimizing a single global threshold, as shown by the CS-Feature versus CS-Global comparison on every dataset.
- The method applies to any continuous feature representation, since thresholding and coordinate search do not depend on the meaning of the features.
- On IMDb, the binary representation exceeds full-precision BERT accuracy, so binarization is not necessarily a trade-off.
- All binary methods in the reported table use dramatically less memory and compute than 32-bit embeddings, with the proposed method among the smallest and fastest.
Reading between the lines
- A direct extension the paper does not pursue is to estimate each dimension's empirical range before searching, which would remove the fixed $[-1,1]$ search-space assumption and make the method applicable to embedding models whose values fall outside that interval.
- The optimized threshold vector itself could be read as a feature-importance map: dimensions whose thresholds sit far from zero are the ones where the sign of the value is most discriminative, which could guide pruning or feature selection.
- Because the objective is task accuracy rather than reconstruction error, the same coordinate-search binarization could be applied to vision or multimodal embeddings, with the expectation that per-feature cut-points will help wherever feature scales differ.
- If per-feature thresholds really deliver near-full-precision accuracy, the practical ceiling for binary embeddings may be much higher than current fixed-threshold binarization suggests; testing on retrieval or ranking tasks would show whether the benefit extends beyond classification.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a Coordinate Search-based feature-wise thresholding method to convert real-valued BERT embeddings into binary 'barcode' representations. The method optimizes one threshold per feature by maximizing F1-score during a coordinate search, and the resulting binary embeddings are used to train a logistic regression classifier. Experiments on IMDb, GLUE SST-2, AG News, CoNLL-2003, and SNLI report median accuracies, memory and timing comparisons, plus Kruskal-Wallis and post-hoc statistical tests. The central claim is that the proposed per-feature thresholding outperforms traditional global thresholding methods and approaches or exceeds the accuracy of full-precision BERT embeddings while using substantially less memory.
Significance. If substantiated, per-feature threshold optimization for binary embeddings would be a practically useful contribution for memory-constrained NLP deployments, and framing threshold selection as a derivative-free per-coordinate optimization problem is a reasonable idea. However, the manuscript does not provide a valid evaluation protocol: no train/validation/test split is described, the reported accuracy numbers conflict with the accompanying statistical tests, and several tables contain duplicated or templated values. The paper ships no code or data, so the evidence for the central outperformance claim is not reliable. The underlying idea may merit future work, but this manuscript does not demonstrate it.
major comments (5)
- [Section 3.1 and Section 4.3] No train/validation/test split is described anywhere in the paper. Section 3.1 states that the CS algorithm evaluates the F1-score of candidate thresholds on the embeddings, and the end of Section 3 states that the resulting binary embeddings are 'utilized to train a logistic regression classifier to evaluate the accuracy.' If the same samples are used for threshold optimization, classifier training, and accuracy reporting, then all numbers in Table 2 are training-set fits rather than estimates of generalization. This is load-bearing because the central claim that the proposed method 'outperform[s] traditional binarization methods in accuracy' depends entirely on Table 2. The authors must specify the exact data split (e.g., 80/10/10 or k-fold), report metrics on a held-out test set, and ensure that thresholds and the classifier are fitted only on training data.
- [Table 2 and Section 5] The text and the statistical section claim that the proposed method achieved the highest median accuracy on all five datasets and significantly outperforms real-valued embeddings. Table 2, however, shows Real embeddings BERT higher on GLUE SST-2 (84.77 vs. 82.70), AG News (87.93 vs. 85.72), CoNLL (82.93 vs. 77.74), and SNLI (82.22 vs. 76.70). The Section 5 claim that the proposed method beats Real embeddings on every dataset is therefore internally inconsistent with the accuracy table. Additionally, the Otsu row repeats 75.05 for both GLUE SST-2 and AG News and repeats 64.95 for both CoNLL and SNLI, and the Optimization-based Otsu row repeats 68.45 for CoNLL and SNLI. These coincidences suggest data entry errors, and the accuracy evidence must be corrected and reconciled before any statistical claim can be assessed.
- [Section 3.1] The algorithm hard-codes the search bounds to L = -1 and U = 1 for every feature, with the justification that 'the BERT embeddings of our dataset lie within this range.' No verification is reported: the authors do not provide per-dimension minima and maxima, nor the fraction of values outside [-1,1] for any dataset. If any feature dimension has values outside this interval, the corresponding binary bit becomes constant (all 0 or all 1), silently discarding information, and the optimized threshold cannot adapt to the true data scale. This assumption is load-bearing for the method's claimed accuracy and efficiency, and it must be verified empirically.
- [Table 3 and Section 4.3] Table 3 reports proposed-method computation times of 110-150 ms depending on the dataset, but the text in Section 4.3 states 'as seen with methods like proposed at 50 ms across datasets.' The memory and timing columns also follow a visibly regular pattern (e.g., 4.98, 5.00, 5.02, ... MB and 120, 125, 130, ... ms for IMDb), which strongly suggests templated rather than measured values. The efficiency claim, which is a major advertised advantage of the method, is therefore unsupported by the reported data.
- [Section 5 and Tables 4-9] The Kruskal-Wallis and post-hoc tests are applied to the same data used for threshold optimization and classifier training, so they inherit the circularity described in the first comment. Moreover, Tables 5-9 contain repeated identical p-values in unrelated cells (e.g., 7.485e-10 appears multiple times in Table 5, and 5.135e-6 appears multiple times in Table 6), and the claim that CS significantly outperforms Real embeddings on every dataset contradicts Table 2. These statistical results cannot be taken as evidence of robustness or generalizability.
minor comments (4)
- [Abstract and Section 1] The text contains typos and duplicated words, including 'using using' in the abstract, 'acheive' in Section 1, 'intresting' in Section 1, and 'interation' in Section 3.1.
- [References] The reference list contains duplicates: Devlin et al. appears as both [9] and [34], Otsu appears as both [23] and [32], and Bidgoli and Rahnamayan appears as both [27] and [36]. These should be consolidated.
- [Table 1] The dataset named 'SNIL' in Table 1 is written as 'SNLI' everywhere else in the paper; the table also lists GLUE-sst2 as 70,000 samples and does not clarify whether the reported numbers refer to full datasets or to train/test splits, which matters given the missing split description.
- [Figure 4] The heatmaps use a color scale based only on -log10(p-value), so the row for Real embeddings appears bright even though Real loses comparisons; the text acknowledges this, but the visualization is misleading and should be redesigned, for example by showing signed effect sizes or direction-aware colors.
Circularity Check
Reported accuracies are the fitted objective: per-feature thresholds are optimized against classification F1 on the same data that is later scored, with no described held-out split.
-
fitted input called prediction
[Section 3.1 (Threshold Optimization), Algorithm 1; Section 3, final paragraph; Table 2 in Section 4.3]
"Let F 1X and F 1Y denote the F1-scores obtained by applying X (1) i and Y (2) i , respectively. After iterating through all dimensions for maxiter iterations, the algorithm yields an optimized threshold vector S∗. ... These binary embeddings are then utilized to train a logistic regression classifier to evaluate the accuracy of the achieved binary representation for NLP tasks."
Per-feature thresholds S∗ are chosen by Coordinate Search to maximize classification F1 on the data (Algorithm 1: F1X←Evaluate(binary_X), F1Y←Evaluate(binary_Y)). Table 2 then reports median classification accuracy obtained by training a logistic regression classifier on those same binary embeddings and evaluating accuracy. No train/validation/test split is described for the proposed method; the only mention of validation in Section 4.3 is for scalar baselines ('maximises validation macro-F1'). Thus the reported outperformance is a training-set ranking: 768 thresholds fitted to the target objective on the same samples later scored beat fixed thresholds, so the central claim reduces to a fit rather than independent prediction.
full rationale
One significant circular step of type fitted_input_called_prediction is present. The derivation chain is: define the fitness as classification F1 on the embeddings; optimize 768 per-feature thresholds against that objective; then report classification accuracy on the same embeddings via a logistic regression classifier. Because no held-out split is documented, the headline claim is not distinguishable from reporting the objective value of a high-capacity fit. The self-citations (Bidgoli and Rahnamayan on coordinate search) are not load-bearing, as Coordinate Search is a standard derivative-free algorithm and the paper does not rely on a uniqueness theorem or a self-cited empirical constant. The [-1,1] bound is an unverified assumption but not circularity. I therefore score 6: partial circularity in the central evaluation claim.
Assumptions & free parameters
free parameters (4)
- per-feature threshold vector S* =
Not reported (768 values per dataset)
- Search bounds L and U =
L = -1, U = 1
- maxiter =
Not reported
- maxNFE =
num_samples * maxiter * 2
assumptions (4)
- domain assumption BERT embeddings lie within [-1,1]
- domain assumption F1-score computed on the same data used for threshold optimization is a valid proxy for generalization
- ad hoc to paper Coordinate search with center-based sampling converges to a near-optimal threshold vector within the given budget
- domain assumption Logistic regression on binary features is an appropriate downstream evaluator
Cite this review
Pith. "Pith review of Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddings." pith.science (2026). https://pith.science/paper/WSQZFSHP
@misc{pith2026250717025,
author = {Pith},
title = {Pith review of: Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/WSQZFSHP}},
note = {Machine review of arXiv:2507.17025}
}
read the original abstract
Efficient text embedding is crucial for large-scale natural language processing (NLP) applications, where storage and computational efficiency are key concerns. In this paper, we explore how using binary representations (barcodes) instead of real-valued features can be used for NLP embeddings derived from machine learning models such as BERT. Thresholding is a common method for converting continuous embeddings into binary representations, often using a fixed threshold across all features. We propose a Coordinate Search-based optimization framework that instead identifies the optimal threshold for each feature, demonstrating that feature-specific thresholds lead to improved performance in binary encoding. This ensures that the binary representations are both accurate and efficient, enhancing performance across various features. Our optimal barcode representations have shown promising results in various NLP applications, demonstrating their potential to transform text representation. We conducted extensive experiments and statistical tests on different NLP tasks and datasets to evaluate our approach and compare it to other thresholding methods. Binary embeddings generated using using optimal thresholds found by our method outperform traditional binarization methods in accuracy. This technique for generating binary representations is versatile and can be applied to any features, not just limited to NLP embeddings, making it useful for a wide range of domains in machine learning applications.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Relation r econstructive binariza- tion of word embeddings
Feiyang Pan, Shuokai Li, Xiang Ao, and Qing He. Relation r econstructive binariza- tion of word embeddings. Frontiers of Computer Science , 16:1–8, 2022
work page 2022
-
[2]
N ear-lossless binarization of word embeddings
Julien Tissier, Christophe Gravier, and Amaury Habrard. N ear-lossless binarization of word embeddings. Proceedings of the AAAI Conference on Artificial Intelligen ce, 33(01):7104–7111, July 2019
work page 2019
-
[3]
Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models
Dongwon Jo, Taesu Kim, Yulhwa Kim, and Jae-Joon Kim. Mixture o f scales: Memory-efficient token-adaptive binarization for large lang uage models. arXiv pre- print arXiv:2406.12311 , 2024. 19
work page Pith review arXiv 2024
-
[4]
Word embedding binarization with semantic information preservation
Samarth Navali, Praneet Sherki, Ramesh Inturi, and Vanra j Vala. Word embedding binarization with semantic information preservation. In Do nia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics , pages 1256–1265, Barcelona, Spain (Online), December
-
[5]
Y. Zhao. Heavy-ball-based optimal thresholding algori thms for sparse linear inverse problems. SIAM Journal on Optimization , 30(1):31–55, 2020
work page 2020
-
[6]
S. Agrawal, R. Panda, S. Bhuyan, and B.K. Panigrahi. Tsall is entropy based optimal multilevel thresholding using cuckoo search algorithm. In Swarm and Evolutionary Computation, volume 11, pages 16–30. Elsevier, 2013
work page 2013
-
[7]
N. Meng and Y. Zhao. Newton-type optimal thresholding algo rithms for sparse optimization problems. SIAM Journal on Optimization , 30(1):31–55, 2020
work page 2020
-
[8]
Tizhoosh, Christopher Mitcheltree, Shujin Zhu, an d Shamak Dutta
Hamid R. Tizhoosh, Christopher Mitcheltree, Shujin Zhu, an d Shamak Dutta. Bar- codes for medical image retrieval using autoencoded radon t ransform. In 2016 23rd International Conference on Pattern Recognition (ICPR) , pages 3150–3155, 2016
work page 2016
Show all 42 references
-
[10]
Giuliani and E
E. Giuliani and E. Camponogara. Genetic algorithms and the ir application to oil field optimization problems. Computers & Chemical Engineering , 82:282–292, 2015
2015
-
[11]
Beck and M
A. Beck and M. Teboulle. Iterative shrinkage/threshold ing algorithm (ista) for finding sparse solutions to ill-posed inverse problems. SIAM Journal on Imaging Sciences , 2(1):183–202, 2009
2009
-
[12]
Binary neural networks: A survey
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jing kuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105:107281, 2020
2020
-
[13]
Binarized neural networks: Training deep neural netwo rks with weights and activations constrained to +1 or -1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Ben- gio. Binarized neural networks: Training deep neural netwo rks with weights and activations constrained to +1 or -1. In Advances in neural information processing systems, volume 29, pages 4107–4115, 2016
2016
-
[14]
Accelerating bert inference for sequence labeling via disc rete latent variables
Zichao Guo, Qipeng Zhang, Yichong Zhang, Xipeng Chen, an d Xuanjing Wang. Accelerating bert inference for sequence labeling via disc rete latent variables. arXiv preprint arXiv:2010.05350, 2020
2010 arXiv
-
[15]
B itnet: Scaling 1-bit transformers for large language models
Shuchen Zhang, Xiaojie Zheng, Fan Yin, and Xipeng Liu. B itnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453 , 2023. 20
2023 arXiv
-
[16]
Tizhoosh
Hamid R. Tizhoosh. Barcode annotations for medical imag e retrieval: A preliminary investigation. In 2015 IEEE International Conference on Image Processing, IC IP 2015, Quebec City, QC, Canada, September 27-30, 2015 , pages 818–822. IEEE, 2015
2015
-
[17]
Minmax radon barcodes for medical image retrieval
Hamid Tizhoosh, Shujin Zhu, Hanson Lo, Varun Chaudhari, a nd Tahmid Mehdi. Minmax radon barcodes for medical image retrieval. 10 2016
2016
-
[18]
Tizhoosh
Shujin Zhu and H.R. Tizhoosh. Radon features and barcode s for medical image re- trieval via svm. In 2016 International Joint Conference on Neural Networks (IJ CNN), pages 5065–5071, 2016
2016
-
[19]
Alexander Knipper, Md
R. Alexander Knipper, Md. Mahadi Hassan, Mehdi Sadi, and Shu bhra Kanti Kar- maker Santu. Analogy-guided evolutionary pretraining of b inary word embeddings. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, ed itors, Proceedings of the 2nd Conference of the Asia-...
2022
-
[20]
Se mantic preserving sia- mese autoencoder for binary quantization of word embedding s
Wouter Mostard, Lambert Schomaker, and Marco Wiering. Se mantic preserving sia- mese autoencoder for binary quantization of word embedding s. In Proceedings of the 2021 5th International Conference on Natural Language P rocessing and Inform- ation Retrieval, NLPIR ’21, page 30...
2021
-
[21]
Enhancing content-based histopathology image r etrieval using qr code representation
Hamidreza Rouzegar, Shahryar Rahnamayan, Azam Asilia n Bidgoli, and Masoud Makrehchi. Enhancing content-based histopathology image r etrieval using qr code representation. pages 1120–1125, 12 2023
2023
-
[22]
Jain and D
A. Jain and D. Zongker. Feature selection: evaluation, application, and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intellig ence, 19(2):153–158, 1997
1997
-
[23]
N. Otsu. A threshold selection method from gray-level h istograms. IEEE Transac- tions on Systems, Man, and Cybernetics , 9(1):62–66, 1979
1979
-
[24]
Evolution and optimum seeking: The s ixth generation
Hans-Paul Schwefel. Evolution and optimum seeking: The s ixth generation. 01 1995
1995
-
[25]
Ehsan, S
R. Ehsan, S. Rahnamayan, S.Z. Miyandoab, A.A. Bidgoli, a nd H.R. Tizhoosh. Train- ing artificial neural networks by coordinate search algorit hm. Computers & Chemical Engineering, 82:282–292, 2024
2024
-
[26]
Breheny and J
P. Breheny and J. Huang. Coordinate descent algorithm fo r nonconvex penalized regression, with application to biological feature selecti on. The Annals of Applied Statistics, 5:232–253, 2011. 21
2011
-
[27]
A. A. Bidgoli and S. Rahnamayan. Memetic differential evol ution using coordin- ate descent. Proceedings of the Genetic and Evolutionary Computation Co nference Companion, pages 359–366, 2021
2021
-
[28]
Rokhsatyazdi, S
E. Rokhsatyazdi, S. Rahnamayan, S.Z. Miyandoab, A.A. Bid goli, and H.R. Tizhoosh. Training artificial neural networks by coordinate search al gorithm. Proceedings of the IEEE Congress on Evolutionary Computation , pages 1540–1546, 2023
2023
-
[29]
Chang, C.-J
K.-W. Chang, C.-J. Hsieh, and C.-J. Lin. Coordinate descent m ethod for large- scale l2-loss linear support vector machines. Journal of Machine Learning Research , 9(45):1369–1398, 2008
2008
-
[30]
Tibshirani
R. Tibshirani. Regression shrinkage and selection via t he lasso. Journal of the Royal Statistical Society. Series B (Methodological) , 58(1):267–288, 1996
1996
-
[31]
Image Thresholding
OpenCV. Image Thresholding. Accessed: March 27, 2025
2025
-
[32]
A threshold selection method from gray- level histograms
Nobuyuki Otsu. A threshold selection method from gray- level histograms. IEEE Transactions on Systems, Man, and Cybernetics , 9(1):62–66, 1979
1979
-
[33]
Hybrid thresholdi ng for image de- convolution in expectation maximization framework
Ravi Pratap Singh and Manoj Kumar Singh. Hybrid thresholdi ng for image de- convolution in expectation maximization framework. The Imaging Science Journal , 0(0):1–18, 2024
2024
-
[34]
Bert: Pre- training of deep bidirectional transformers for language u nderstanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Tout anova. Bert: Pre- training of deep bidirectional transformers for language u nderstanding. arXiv preprint arXiv:1810.04805, 2018
2018 arXiv
-
[35]
Evolution and optimum seeking: the sixth generation
Hans-Paul Schwefel. Evolution and optimum seeking: the sixth generation . John Wiley & Sons, 1993
1993
-
[36]
Memetic d ifferential evolution using coordinate descent
Azam Asilian Bidgoli and Shahryar Rahnamayan. Memetic d ifferential evolution using coordinate descent. In 2021 IEEE Congress on Evolutionary Computation (CEC), pages 359–366. IEEE, 2021
2021
-
[37]
Coordinate search alg orithms in multilevel optimization
Emanuele Frandi and Andrea Papini. Coordinate search alg orithms in multilevel optimization. Optimization Methods and Software , 29(5):1020–1041, 2014
2014
-
[38]
Convergence of a block coordinate descent me thod for nondifferentiable minimization
Paul Tseng. Convergence of a block coordinate descent me thod for nondifferentiable minimization. Journal of Optimization Theory and Applications , 109(3):475–494, 2001
2001
-
[39]
A comprehensive investigation on novel center-based sampling for large- scale global optimization
Hanan Hiba, Shahryar Rahnamayan, Azam Asilian Bidgoli , Amin Ibrahim, and Rasa khosroshahli. A comprehensive investigation on novel center-based sampling for large- scale global optimization. Swarm and Evolutionary Computation , 73:101105, 2022. 22
2022
-
[40]
Sainarayanan, and Ratna Dahiya
Jyoti Malik, G. Sainarayanan, and Ratna Dahiya. Min max th reshold range (mmtr) based approach in palmprint authentication by sobel code me thod. Procedia Com- puter Science , 2:149–158, 2010. Proceedings of the International Conferen ce and Exhibition on Biometrics Technology
2010
-
[41]
N. Raja, V. Rajinikanth, and K. Latha. Otsu based optimal multilevel image thresholding using firefly algorithm. Modelling and Simulation in Engineering , 2014:37, 2014
2014
-
[43]
Appendix 7.1 Statistical tests on various methods used p-values obtained from the statistical test on various metho ds are shown below in tables. Table 5: Post-Hoc Comparison Results ( p-values) for IMDb Method CS CSGlobal Hybrid MinMax Otsu Simple Simple Otsu Hybrid Real Opti...
2003
-
[2020]
International Committee on Computational Linguistics
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.