Pith. sign in

REVIEW 4 major objections 6 minor 27 references

On-Device Text Representations Robust To Misspellings via Projections

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LSH projection classifiers are inherently robust to misspellings, losing only 2.94 points of accuracy on average while fine-tuned BERT loses 11.44 points.

desk verdict A useful empirical study of projection-based classifiers' robustness to character-level edits, but the edits tested are narrow and the tables disagree on the headline numbers. read the letter →

arxiv 1908.05763 v3 pith:GQXZIBNO submitted 2019-08-14 cs.CL cs.LG

Chinnadhurai Sankar , Sujith Ravi , Zornitsa Kozareva This is my paper · ORCID
classification cs.CLcs.LG
keywords locality-sensitivehashingtextclassificationmisspellingrobustnesson-deviceNLPneuralprojectionswordembeddingsadversarialperturbationsBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that text classifiers built on binary locality-sensitive hashing (LSH) projections are inherently robust to misspellings and character-level perturbations, precisely because they do not depend on word or word-piece embedding lookup tables. The authors compare two projection-based classifiers, SGNN and ProSeqo, against fine-tuned BERT and BiLSTMs on four classification tasks, corrupting test words by randomly dropping, inserting, and swapping internal characters. They report an average accuracy drop of only 2.94% for the projection classifiers, against 11.44% for fine-tuned BERT and larger drops for BiLSTM variants. A sympathetic reader would care because the same representation that makes on-device models memory-efficient would also make them stable under the noisy text users actually type.

What carries the argument

The central object is the binary Locality-Sensitive Hashing (LSH) projection function P, which maps any token or sentence to a fixed K-bit vector by extracting character- or token-level n-gram and skip-gram features and applying a modified LSH transform. A token's distributed representation is then W_K P(x)^T, with K much smaller than the vocabulary size V, so no embedding lookup table is stored. Because P is built from overlapping n-grams, a character-level edit changes only the bits associated with the affected n-grams; the paper measures this change as roughly 11% of the average inter-word Hamming distance under 20% perturbation and reports that the change is proportional to K times the perturbation probability.

What would settle it

Run the same four classifiers on test sets corrupted by realistic misspellings rather than random character edits, for example keyboard-adjacent substitutions and phonetic typos at the same per-word rate, and compare accuracy drops; if fine-tuned BERT or the BiLSTM baselines lose less accuracy than SGNN and ProSeqo, the claimed inherent robustness is an artifact of the perturbation protocol.

Watch

Extended reading notes

Core claim

The central claim is that LSH projection based neural classifiers are inherently robust to common misspellings and input perturbations, and therefore can serve as a memory-efficient alternative to embedding-lookup models without sacrificing reliability on noisy text. In head-to-head tests with a per-word perturbation probability of 0.2, the projection-based SGNN and ProSeqo classifiers show accuracy drops between about 1.91 and 3.91 points across MRDA, ATIS, Yahoo! Answers, and Amazon Reviews, while fine-tuned BERT drops 8.25 to 15.57 points and BiLSTM variants drop comparably or more. A companion perturbation analysis on a large English corpus finds that a misspelled word's binary projection moves only about 11% of the average Hamming distance between distinct words, and that the change grows linearly with projection dimension and perturbation probability. The authors conclude that the projection representation is stable and that neural layers on top of it will rarely confuse a misspelled word for an unrelated valid word.

Load-bearing premise

The robustness result rests on the assumption that randomly dropping, inserting, and swapping internal characters in 20% of test words captures the misspellings real users produce; if real misspellings follow different patterns, such as keyboard-adjacent or phonetic errors, the measured accuracy ordering may not hold.

Editorial extensions

If this is right

  • Projection-based classifiers (SGNN and ProSeqo) keep accuracy nearly intact when up to 20% of test words are randomly corrupted, with drops between roughly 1.9 and 3.9 points across tasks.
  • Fine-tuned BERT and word-piece or word-only BiLSTMs lose substantially more accuracy under the same attacks, with BERT dropping 11.44 points on average.
  • The LSH projection of a word changes by only about 11% of the average inter-word Hamming distance under 20% perturbation, so misspelled words stay near their clean representations rather than colliding with unrelated words.
  • Because the projection size K is far smaller than the vocabulary size V, the robust representation also removes the need for embedding lookup tables, shrinking memory from O(V) to O(K).
  • The linear relation Delta P(x) proportional to K times P_perturb means the effect of misspellings can be predicted from projection dimension and perturbation rate without retraining the classifier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The n-gram-overlap argument suggests projection representations should also resist out-of-vocabulary words and character-set shifts, since unseen words still share character n-grams with training words; the paper does not test this directly.
  • If LSH projections were inserted in place of embedding lookups inside Transformer-style models, the robustness benefit might carry over, but the paper only gestures at that possibility and does not test it.
  • A direct test would be to measure projection robustness under adversarial misspellings crafted to maximize classifier loss rather than random edits; the paper only uses random perturbations.
  • The stable projection distances hint that the robustness may not come at a large clean-accuracy cost, but the paper does not report clean accuracy for all models in the same table, leaving that comparison open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies the robustness of locality-sensitive hashing (LSH) based projection text representations to character-level misspellings. It compares two projection classifiers, ProSeqo and SGNN, against fine-tuned BERT and BiLSTM baselines with word-piece and word-only tokenization on four text classification datasets (MRDA, ATIS, Yahoo Answers, Amazon Reviews). Under random character dropping, insertion, and swapping applied to internal word characters with probability P_perturb = 0.2, the projection classifiers are reported to suffer substantially smaller average accuracy drops (2.94%) than fine-tuned BERT (11.44%). The paper also reports a perturbation analysis on the enwik9 corpus, claiming that the Hamming distance change in LSH projections is proportional to K * P_perturb.

Significance. If substantiated, the result is practically significant for on-device NLP, where memory-efficient representations are essential, and it would provide useful evidence about the robustness properties of hashing-based text representations. The study has clear strengths: it compares against strong external baselines (BERT, BiLSTMs) rather than only self-comparisons; the perturbed test inputs are held fixed across models; the projection classifiers show low variance across five runs; and the direct representation-level analysis in Section 4 is a useful complement to the classification experiments. However, the central claim is currently limited to a narrow class of n-gram-preserving edits, and internal inconsistencies between the two main result tables prevent the headline aggregate from being reproduced from the reported numbers. The contribution is therefore defensible in principle but needs substantive revision.

major comments (4)
  1. [Section 3, perturbation protocol (Table 2)] The perturbation protocol applies only random dropping, insertion, and swapping of internal characters; it never substitutes one character for another and never modifies word boundaries. Since the LSH projection P(x) is built from character n-grams, the paper's own assumption in Section 2 that "the fraction of n-grams affected by spelling attacks tend to be minimal" holds for these edits but not necessarily for keyboard-adjacency substitutions, phonetic substitutions, or boundary transpositions, which can change most or all n-grams of short words. The headline 2.94% versus 11.44% ordering is therefore established only for the tested n-gram-preserving edit types, not for "common misspellings" as users actually produce them. Please add substitution-based and real-misspelling evaluations, or explicitly narrow the claim in the abstract and conclusion.
  2. [Tables 2 and 3, P=20/all condition] The same condition (P=20, all operation types) yields incompatible accuracy drops for the BiLSTM baselines. For MRDA, BiLSTM-w drops 15.14 in Table 2 but only 4.75 in Table 3 (78.14 to 73.39); BiLSTM-wp drops 8.91 in Table 2 but 6.27 in Table 3. For ATIS, BiLSTM-wp drops 20.11 in Table 2 but 11.33 in Table 3. The headline aggregate is therefore not internally reproducible from the reported tables. Please reconcile the two tables and state explicitly whether the "all" operation in Table 2 is identical to the "all" operation in Table 3, including how many edits are applied per word.
  3. [Section 4, Table 4 and proportionality claim] The claimed proportionality DeltaP(x) proportional to K * P_perturb is not supported by the evidence presented. Table 4 reports only four projection dimensions and two perturbation levels, without standard deviations or baseline average Hamming distances between words. The absolute change in bits is expected to grow with K even under a fixed perturbation mechanism, because random binary projections have expected pairwise Hamming distance K/2. Please report the normalized change (for instance, DeltaP divided by the expected Hamming distance), vary P over a wider range, and provide uncertainty or regression diagnostics before stating this as a proportional law.
  4. [Table 2, missing baselines] Table 2 reports only accuracy drops and does not give the unperturbed accuracy for every model-dataset combination, even though the comparison is framed as drops from comparable baselines. Table 3 provides baselines for BiLSTM and SGNN, but not for BERT or ProSeqo, so the claim that all models were trained to comparable accuracy cannot be verified. Please add a full baseline table for all models and datasets, or otherwise report the unperturbed accuracies alongside Table 2.
minor comments (6)
  1. [Abstract and Conclusion] The phrase "inherently robust" overstates what is an empirical finding under a specific perturbation protocol; I recommend wording such as "robust under the tested character-level perturbation types."
  2. [Section 3, reproducibility] The text says that further details on perturbation operations and training are in the supplementary material, but no supplementary material is included in this version; please include it or remove the pointer.
  3. [Section 4, footnote 2] The main text refers to "enwik92" while the footnote refers to "enwik9"; this typo should be corrected.
  4. [Table 1] The table labels ATIS as "Dialog act" while the body text describes it as intent prediction; please align the terminology.
  5. [Introduction and Section 2] The citation for SGNN alternates between Ravi and Kozareva (2018) and Ravi and Kozareva (2019); please make the citation consistent with the reference list.
  6. [Table 2, statistical comparison] The paper reports means and standard deviations over five runs but does not provide paired significance tests; a paired test across datasets and runs would make the comparison between projection models and BERT more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the robustness claim rests on external comparisons to BERT and BiLSTMs under a standard perturbation protocol, and the projection self-citations are method references, not load-bearing assumptions.

full rationale

This paper does not derive its central robustness claim from its own definitions. The LSH projection representations P(x) are constructed from character n-grams in Section 2, and the paper hypothesizes that misspellings change relatively few n-grams. But that hypothesis is then tested empirically in Section 3 by comparing ProSeqo and SGNN against fine-tuned BERT and BiLSTMs on MRDA, ATIS, Yahoo, and Amazon under drop/insert/swap perturbations; the comparison is external and the reported accuracy drops are measured, not forced by construction. The many self-citations (Ravi 2017; Ravi and Kozareva 2018; Kozareva and Ravi 2019; Sankar et al. 2019) introduce the projection architecture and hyperparameters such as K=1120, but they do not contain or entail the robustness result, so they are ordinary method citations rather than load-bearing self-citation. Section 4's perturbation analysis is descriptive: it measures Hamming-distance changes in the projection space and reports a proportionality to K and P_perturb; this is an observation about the representation, not a renamed input, and it does not by itself force the classification robustness ordering. The reviewer concern that the perturbation protocol (random internal drop/insert/swap) may not represent real-world misspellings such as substitutions is a threat to external validity and should be treated as a correctness risk, not a circularity. Likewise, the inconsistency between Table 2 and Table 3 for the same P=20/all condition (e.g., BiLSTM-w MRDA drop 15.14 vs 4.75) is a reproducibility problem, not evidence that any result reduces to its inputs by definition. Under the strict standard requiring an exhibited equation-level or definition-level reduction, no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The experiments are empirical; they rely on fixed projection hyperparameters, a particular perturbation protocol, and fair baseline training. No new entities are introduced.

free parameters (2)
  • Projection dimension K = 1120
    Fixed to 1120 following Ravi and Kozareva (2018) and used across all experiments; the central robustness result is reported for this value only, and Section 4 suggests projection changes scale with K.
  • Word perturbation probability P_perturb = 0.2 (headline result)
    Per-word probability of drop/insert/swap perturbation, fixed at 0.2 for Table 2; robustness drop magnitudes depend on this choice.
assumptions (3)
  • domain assumption Character n-gram overlap between a misspelled word and its correct form is usually large, so LSH projections of misspelled and correct words remain close.
    Stated in Section 2 and used as the mechanism for robustness; not formally proved for arbitrary misspellings.
  • domain assumption The LSH transformation L maps overlapping n-gram feature sets to binary codes with small Hamming distance.
    Implicit in the projection method (Ravi 2017); the paper does not verify collision properties for adversarial misspellings beyond empirical averages.
  • domain assumption BiLSTM, BERT, and projection classifiers used in comparison are comparably trained and at similar unperturbed accuracy levels.
    The paper says 'for comparable accuracies' but does not report full unperturbed accuracy tables for all settings in Table 2, so fairness of the accuracy-drop comparison is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-Device Text Representations Robust To Misspellings via Projections." pith.science (2026). https://pith.science/paper/GQXZIBNO

@misc{pith2026190805763,
  author       = {Pith},
  title        = {Pith review of: On-Device Text Representations Robust To Misspellings via Projections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQXZIBNO}},
  note         = {Machine review of arXiv:1908.05763}
}
read the original abstract

Recently, there has been a strong interest in developing natural language applications that live on personal devices such as mobile phones, watches and IoT with the objective to preserve user privacy and have low memory. Advances in Locality-Sensitive Hashing (LSH)-based projection networks have demonstrated state-of-the-art performance in various classification tasks without explicit word (or word-piece) embedding lookup tables by computing on-the-fly text representations. In this paper, we show that the projection based neural classifiers are inherently robust to misspellings and perturbations of the input text. We empirically demonstrate that the LSH projection based classifiers are more robust to common misspellings compared to BiLSTMs (with both word-piece & word-only tokenization) and fine-tuned BERT based methods. When subject to misspelling attacks, LSH projection based classifiers had a small average accuracy drop of 2.94% across multiple classifications tasks, while the fine-tuned BERT model accuracy had a significant drop of 11.44%.

Figures

Figures reproduced from arXiv: 1908.05763 by the authors.

Figure 1
Figure 1. One-hot word embedding look-up vectors vs linear combination of LSH projection based vectors (Ravi, 2017) representing the same word. This led to interesting research by (Ravi and Kozareva, 2018; Sankar et al., 2019), who showed that word embeddings can be replaced with lightweight binary Locality-Sensitive Hashing (LSH) based projections learned on-the-fly. The projection approach surmounts the need to store any em… view at source ↗
Figure 3
Figure 3. Binary Locality-Sensitive Hashing (LSH) projection representation for text. 2 Binary LSH projections for text representations The dependency on vocabulary size V , is one of the primary reasons for the huge memory footprint of embedding matrices. It is common to represent a token, x by one-hot representation, Y(x) ∈ [0, 1]V and a distributed representation of the token is ob￾tained by multiplying the one-hot represe… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jianpeng Cheng and Mirella Lapata. 2016. https://doi.org/10.18653/v1/P16-1046 Neural summarization by extracting sentences and words . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484--494. Association for Computational Linguistics

    doi:10.18653/v1/p16-1046 2016

  4. [4]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    arXiv 2018

  5. [5]

    Bora Edizel, Aleksandra Piktus, Piotr Bojanowski, Rui Ferreira, Edouard Grave, and Fabrizio Silvestri. 2019. http://arxiv.org/abs/1905.09755 Misspelling oblivious word embeddings . CoRR, abs/1905.09755

    work page Pith review arXiv 2019

  6. [6]

    Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops, SP Workshops 2018, San Francisco, CA, USA, May 24, 2018 , pages 50--56

    work page 2018

  7. [7]

    Godfrey, Edward C

    John J. Godfrey, Edward C. Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ICASSP'92, pages 517--520. IEEE Computer Society

    work page 1992

  8. [8]

    Daniel Jurafsky, Rebecca Bates, Rachel Martin Noah Coccaro, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Audreas Stolcke, Paul Taylor, and Van Ess-Dykema. 1997. Automatic detection of discourse structure for speech recognition and understanding. In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding, pages 88--95

    work page 1997

Show all 27 references
  1. [9]

    Zornitsa Kozareva and Sujith Ravi. 2019. https://www.aclweb.org/anthology/D19-1402 P ro S eqo: Projection sequence networks for on-device text classification . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International J...

    2019

  2. [10]

    Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. http://aclweb.org/anthology/P18-1157 Stochastic answer networks for machine reading comprehension . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2018

  3. [11]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations, Workshop Track

    2013

  4. [12]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 a . Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543. Association for Computational Linguistics

    2014

  5. [13]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014 b . http://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  6. [14]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2018

  7. [15]

    Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. http://arxiv.org/abs/1905.11268 Combating adversarial misspellings with robust word recognition . CoRR, abs/1905.11268

    2019 arXiv

  8. [16]

    Sujith Ravi. 2017. http://arxiv.org/abs/1708.00630 Projectionnet: Learning efficient on-device deep networks using neural projections . CoRR, abs/1708.00630

    2017 arXiv

  9. [17]

    Sujith Ravi. 2019. http://proceedings.mlr.press/v97/ravi19a/ravi19a.pdf Efficient on-device models using neural projections . In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5370--5379

    2019

  10. [18]

    Sujith Ravi and Zornitsa Kozareva. 2018. https://www.aclweb.org/anthology/D18-1105 Self-governing neural networks for on-device short text classification . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 ...

    2018

  11. [19]

    Sujith Ravi and Zornitsa Kozareva. 2019. https://doi.org/10.18653/v1/P19-1368 On-device structured and context partitioned projection networks . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3784--3793, Florence, Italy. Assoc...

    2019 doi

  12. [20]

    Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme. 2016. http://arxiv.org/abs/1608.02214 Robsut wrod reocginiton via semi-character recurrent neural network . CoRR, abs/1608.02214

    2016 arXiv

  13. [21]

    Chinnadhurai Sankar, Sujith Ravi, and Zornitsa Kozareva. 2019. https://doi.org/10.18653/v1/N19-1339 Transferable neural projection representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2019 doi

  14. [22]

    Elizabeth Shriberg, Rajdip Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the SIGDIAL 2004 Workshop, The 5th Annual Meeting of the Special Interest Group on Discourse and Dialogue, April 30 - M...

    2004

  15. [23]

    Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language...

    2013

  16. [24]

    o khan T \

    G \"o khan T \"u r, Dilek Hakkani-T \"u r, and Larry P. Heck. 2010. What is left to be understood in atis? In Proceedings of 2010 IEEE Spoken Language Technology Workshop (SLT), pages 19--24

    2010

  17. [25]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  18. [26]

    Robert Lai, and Xuejie Zhang

    Liang-Chih Yu, Jin Wang, K. Robert Lai, and Xuejie Zhang. 2017. https://doi.org/10.18653/v1/D17-1056 Refining word embeddings for sentiment analysis . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 534--539. Association for Com...

    2017 doi

  19. [27]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649--657

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.