{"id":"ad2fc84f-ca32-4dbf-8faa-a77dafa3bb0a","arxiv_id":"1908.05763","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LSH projection classifiers lose only 2.94% average accuracy under misspelling attacks, far less than BERT or BiLSTM baselines.","lead":"The paper tests whether LSH-based projection text classifiers survive misspellings better than standard neural classifiers. It finds average accuracy drops of about 2.9% for projection models versus 11.4% for fine-tuned BERT.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Projection robustness may be an artifact of the random internal-edit protocol: excluded substitutions and edge/phonetic errors can destroy n-gram overlap, so the 2.94% vs 11.44% ordering may not hold for real misspellings.","rationale":"The reader's weakest assumption is exactly the one I would flag: the perturbation generator defines the notion of 'misspelling,' and the projection models exploit n-gram overlap. A real-misspelling test is necessary before the word 'inherently' is justified. I do not recommend rejecting the paper: within the stated protocol the experiments are consistent and the effect is large, and Table 3 independently shows SGNN/ProSeqo drops below BiLSTM drops at multiple perturbation amounts. That is real evidence even without code release. The conditional verdict is appropriate; the condition should be a demonstration with realistic misspellings. I also note the Table 2/Table 3 discrepancy (BiLSTM-w MRDA 15.14 vs 4.75 at P=20/all), which strengthens the need for a protocol-faithful recomputation, but it is secondary to the representativeness concern.","tokens_in":7251,"tokens_out":13319,"duration_ms":128994,"concrete_test":"Build perturbed test sets from a real misspelling corpus, such as the Birkbeck spelling-error corpus or a keyboard-adjacency typo generator, using the same per-word perturbation probability P_perturb=0.2. If SGNN/ProSeqo accuracy drops are not smaller than BERT-base/BiLSTM drops under this more realistic distribution of substitutions, edge transpositions, and phonetic errors, the central claim does not generalize beyond the paper's internal-edit protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the perturbation protocol of Section 3 being a valid proxy for \"common misspellings.\" The protocol only randomly drops, inserts, or swaps internal characters; it never substitutes one character for another and never touches word boundaries. Every allowed edit preserves most character n-grams, and Section 2 explicitly relies on this: \"usually the fraction of n-grams affected by spelling attacks tend to be minimal.\" For an internal edit, at most O(n) of the word's character n-grams change, so P(x) stays close to P(original). Real misspellings are not so constrained: keyboard-adjacency substitutions, phonetic replacements (\"phone\" -> \"fone\"), and edge transpositions can change most or all n-grams of short words. If the misspelling distribution does not preserve n-gram overlap, the measured robustness advantage may vanish. This is the load-bearing assumption: the headline 2.94% vs 11.44% ordering is only known to hold for the paper's n-gram-preserving edits. A secondary red flag is that Table 2 and Table 3 report inconsistent drops for the same P=20/all condition (e.g., BiLSTM-w MRDA 15.14 vs 4.75), so the headline statistic is not internally reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the robustness of locality-sensitive hashing (LSH) based projection text representations to character-level misspellings. It compares two projection classifiers, ProSeqo and SGNN, against fine-tuned BERT and BiLSTM baselines with word-piece and word-only tokenization on four text classification datasets (MRDA, ATIS, Yahoo Answers, Amazon Reviews). Under random character dropping, insertion, and swapping applied to internal word characters with probability P_perturb = 0.2, the projection classifiers are reported to suffer substantially smaller average accuracy drops (2.94%) than fine-tuned BERT (11.44%). The paper also reports a perturbation analysis on the enwik9 corpus, claiming that the Hamming distance change in LSH projections is proportional to K * P_perturb.","tokens_in":7550,"tokens_out":5202,"duration_ms":52317,"significance":"If substantiated, the result is practically significant for on-device NLP, where memory-efficient representations are essential, and it would provide useful evidence about the robustness properties of hashing-based text representations. The study has clear strengths: it compares against strong external baselines (BERT, BiLSTMs) rather than only self-comparisons; the perturbed test inputs are held fixed across models; the projection classifiers show low variance across five runs; and the direct representation-level analysis in Section 4 is a useful complement to the classification experiments. However, the central claim is currently limited to a narrow class of n-gram-preserving edits, and internal inconsistencies between the two main result tables prevent the headline aggregate from being reproduced from the reported numbers. The contribution is therefore defensible in principle but needs substantive revision.","major_comments":[{"comment":"The perturbation protocol applies only random dropping, insertion, and swapping of internal characters; it never substitutes one character for another and never modifies word boundaries. Since the LSH projection P(x) is built from character n-grams, the paper's own assumption in Section 2 that \"the fraction of n-grams affected by spelling attacks tend to be minimal\" holds for these edits but not necessarily for keyboard-adjacency substitutions, phonetic substitutions, or boundary transpositions, which can change most or all n-grams of short words. The headline 2.94% versus 11.44% ordering is therefore established only for the tested n-gram-preserving edit types, not for \"common misspellings\" as users actually produce them. Please add substitution-based and real-misspelling evaluations, or explicitly narrow the claim in the abstract and conclusion.","section":"Section 3, perturbation protocol (Table 2)"},{"comment":"The same condition (P=20, all operation types) yields incompatible accuracy drops for the BiLSTM baselines. For MRDA, BiLSTM-w drops 15.14 in Table 2 but only 4.75 in Table 3 (78.14 to 73.39); BiLSTM-wp drops 8.91 in Table 2 but 6.27 in Table 3. For ATIS, BiLSTM-wp drops 20.11 in Table 2 but 11.33 in Table 3. The headline aggregate is therefore not internally reproducible from the reported tables. Please reconcile the two tables and state explicitly whether the \"all\" operation in Table 2 is identical to the \"all\" operation in Table 3, including how many edits are applied per word.","section":"Tables 2 and 3, P=20/all condition"},{"comment":"The claimed proportionality DeltaP(x) proportional to K * P_perturb is not supported by the evidence presented. Table 4 reports only four projection dimensions and two perturbation levels, without standard deviations or baseline average Hamming distances between words. The absolute change in bits is expected to grow with K even under a fixed perturbation mechanism, because random binary projections have expected pairwise Hamming distance K/2. Please report the normalized change (for instance, DeltaP divided by the expected Hamming distance), vary P over a wider range, and provide uncertainty or regression diagnostics before stating this as a proportional law.","section":"Section 4, Table 4 and proportionality claim"},{"comment":"Table 2 reports only accuracy drops and does not give the unperturbed accuracy for every model-dataset combination, even though the comparison is framed as drops from comparable baselines. Table 3 provides baselines for BiLSTM and SGNN, but not for BERT or ProSeqo, so the claim that all models were trained to comparable accuracy cannot be verified. Please add a full baseline table for all models and datasets, or otherwise report the unperturbed accuracies alongside Table 2.","section":"Table 2, missing baselines"}],"minor_comments":[{"comment":"The phrase \"inherently robust\" overstates what is an empirical finding under a specific perturbation protocol; I recommend wording such as \"robust under the tested character-level perturbation types.\"","section":"Abstract and Conclusion"},{"comment":"The text says that further details on perturbation operations and training are in the supplementary material, but no supplementary material is included in this version; please include it or remove the pointer.","section":"Section 3, reproducibility"},{"comment":"The main text refers to \"enwik92\" while the footnote refers to \"enwik9\"; this typo should be corrected.","section":"Section 4, footnote 2"},{"comment":"The table labels ATIS as \"Dialog act\" while the body text describes it as intent prediction; please align the terminology.","section":"Table 1"},{"comment":"The citation for SGNN alternates between Ravi and Kozareva (2018) and Ravi and Kozareva (2019); please make the citation consistent with the reference list.","section":"Introduction and Section 2"},{"comment":"The paper reports means and standard deviations over five runs but does not provide paired significance tests; a paired test across datasets and runs would make the comparison between projection models and BERT more convincing.","section":"Table 2, statistical comparison"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is practically important and the experimental setup is mostly sound, but the external-validity gap in the perturbation protocol and the direct numerical contradiction between Tables 2 and 3 affect the headline result itself, so I cannot recommend acceptance in the current form. The authors should be asked to reconcile the tables and to test substitution-based and realistic misspellings; both are within the scope of a revision. I would also encourage the editor to check whether the cited supplementary material was intended to be part of this submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this is the first systematic look at how LSH projection classifiers behave under character-level noise, and for the specific edits they test—dropping, inserting, and swapping internal characters—the robustness advantage is real. The numbers are consistent across four datasets and five runs, with very low variance for the projection models. Table 4's observation that projection changes scale roughly with K*P is a nice sanity check, and the paper doesn't oversell it as more than a measured trend.\n\nThe central claim, however, is that projection classifiers are 'inherently robust to misspellings.' That is too broad for what they actually tested. Their perturbation protocol only makes internal edits that preserve most character n-grams—the very property Section 2 relies on. Real misspellings include keyboard substitutions, phonetic replacements, and edge changes that can destroy n-gram overlap for short words. The stress-test concern lands: the 2.94% vs 11.44% headline may not survive a more realistic misspelling distribution. The paper should either test those edits or soften the claim to 'robust to n-gram-preserving character edits.'\n\nThere's also an internal inconsistency that needs to be fixed. Table 2 and Table 3 report the same condition—all perturbation types at 20%—but the drops don't match. For example, BiLSTM-w on MRDA: Table 2 says 15.14% drop, Table 3's numbers imply 4.75%. That's a big gap, and it makes the headline aggregate number hard to trust until reconciled. Possibly one table uses a different random seed or perturbation protocol, but the paper doesn't say.\n\nMinor: no code release, and the comparison omits existing robust-recognition baselines like Pruthi et al., though that's not fatal.\n\nWho this is for: people building on-device text classifiers who want a memory-efficient representation with some typo tolerance. It's a legitimate empirical contribution, not a theoretical one. With the protocol narrowed and the numbers reconciled, it would be a solid paper. I'd send it to peer review, with a request that the authors address both issues.","headline":"A useful empirical study of projection-based classifiers' robustness to character-level edits, but the edits tested are narrow and the tables disagree on the headline numbers.","tokens_in":8004,"tokens_out":2697,"would_cite":false,"duration_ms":26392,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LSH projection classifiers are inherently robust to misspellings, losing only 2.94 points of accuracy on average while fine-tuned BERT loses 11.44 points.","keywords":["locality-sensitive hashing","text classification","misspelling robustness","on-device NLP","neural projections","word embeddings","adversarial perturbations","BERT"],"falsifier":"Run the same four classifiers on test sets corrupted by realistic misspellings rather than random character edits, for example keyboard-adjacent substitutions and phonetic typos at the same per-word rate, and compare accuracy drops; if fine-tuned BERT or the BiLSTM baselines lose less accuracy than SGNN and ProSeqo, the claimed inherent robustness is an artifact of the perturbation protocol.","tokens_in":7108,"feed_emoji":"🔤","tokens_out":5550,"duration_ms":50166,"temperature":0.7,"pith_summary":"This paper tries to establish that text classifiers built on binary locality-sensitive hashing (LSH) projections are inherently robust to misspellings and character-level perturbations, precisely because they do not depend on word or word-piece embedding lookup tables. The authors compare two projection-based classifiers, SGNN and ProSeqo, against fine-tuned BERT and BiLSTMs on four classification tasks, corrupting test words by randomly dropping, inserting, and swapping internal characters. They report an average accuracy drop of only 2.94% for the projection classifiers, against 11.44% for fine-tuned BERT and larger drops for BiLSTM variants. A sympathetic reader would care because the same representation that makes on-device models memory-efficient would also make them stable under the noisy text users actually type.","feed_headline":"Spelling attacks drop LSH classifiers 2.94%, BERT 11.44%","feed_subtitle":"Hash-projection text models stay stable under misspellings and skip embedding tables, fitting on-device memory limits.","key_machinery":"The central object is the binary Locality-Sensitive Hashing (LSH) projection function P, which maps any token or sentence to a fixed K-bit vector by extracting character- or token-level n-gram and skip-gram features and applying a modified LSH transform. A token's distributed representation is then W_K P(x)^T, with K much smaller than the vocabulary size V, so no embedding lookup table is stored. Because P is built from overlapping n-grams, a character-level edit changes only the bits associated with the affected n-grams; the paper measures this change as roughly 11% of the average inter-word Hamming distance under 20% perturbation and reports that the change is proportional to K times the perturbation probability.","core_discovery":"The central claim is that LSH projection based neural classifiers are inherently robust to common misspellings and input perturbations, and therefore can serve as a memory-efficient alternative to embedding-lookup models without sacrificing reliability on noisy text. In head-to-head tests with a per-word perturbation probability of 0.2, the projection-based SGNN and ProSeqo classifiers show accuracy drops between about 1.91 and 3.91 points across MRDA, ATIS, Yahoo! Answers, and Amazon Reviews, while fine-tuned BERT drops 8.25 to 15.57 points and BiLSTM variants drop comparably or more. A companion perturbation analysis on a large English corpus finds that a misspelled word's binary projection moves only about 11% of the average Hamming distance between distinct words, and that the change grows linearly with projection dimension and perturbation probability. The authors conclude that the projection representation is stable and that neural layers on top of it will rarely confuse a misspelled word for an unrelated valid word.","pith_inferences":["The n-gram-overlap argument suggests projection representations should also resist out-of-vocabulary words and character-set shifts, since unseen words still share character n-grams with training words; the paper does not test this directly.","If LSH projections were inserted in place of embedding lookups inside Transformer-style models, the robustness benefit might carry over, but the paper only gestures at that possibility and does not test it.","A direct test would be to measure projection robustness under adversarial misspellings crafted to maximize classifier loss rather than random edits; the paper only uses random perturbations.","The stable projection distances hint that the robustness may not come at a large clean-accuracy cost, but the paper does not report clean accuracy for all models in the same table, leaving that comparison open."],"forward_implications":["Projection-based classifiers (SGNN and ProSeqo) keep accuracy nearly intact when up to 20% of test words are randomly corrupted, with drops between roughly 1.9 and 3.9 points across tasks.","Fine-tuned BERT and word-piece or word-only BiLSTMs lose substantially more accuracy under the same attacks, with BERT dropping 11.44 points on average.","The LSH projection of a word changes by only about 11% of the average inter-word Hamming distance under 20% perturbation, so misspelled words stay near their clean representations rather than colliding with unrelated words.","Because the projection size K is far smaller than the vocabulary size V, the robust representation also removes the need for embedding lookup tables, shrinking memory from O(V) to O(K).","The linear relation Delta P(x) proportional to K times P_perturb means the effect of misspellings can be predicted from projection dimension and perturbation rate without retraining the classifier."],"supporting_citations":[{"why":"Introduces the LSH projection function and neural projection networks that the paper's robust representations are built on.","marker":"(Ravi, 2017, 2019)"},{"why":"Defines the SGNN model and fixes the projection dimension K = 1120 used in the experiments.","marker":"(Ravi and Kozareva, 2018)"},{"why":"Defines the ProSeqo projection sequence network evaluated as one of the robust classifiers.","marker":"(Kozareva and Ravi, 2019)"},{"why":"Documents that fine-tuned BERT and BiLSTM classifiers are brittle under misspelling attacks, motivating the robustness hypothesis.","marker":"(Pruthi et al., 2019)"},{"why":"Supplies the drop, insert, and swap character perturbation operations used to generate misspelled test inputs.","marker":"(Gao et al., 2018)"},{"why":"Provides the BERT-base model that is fine-tuned and used as the strongest comparison baseline.","marker":"(Devlin et al., 2018)"},{"why":"Provides the Amazon Reviews and Yahoo! Answers long-text classification datasets used in the main comparison table.","marker":"(Zhang et al., 2015)"},{"why":"Provides the MRDA dialog act corpus used in the classification experiments.","marker":"(Shriberg et al., 2004)"}],"fun_headline_variants":["Hash-Projection Models Survive Misspellings Better Than BERT","LSH Text Nets: 2.94% Drop vs BERT's 11.44% Under Spelling Attacks","On-Device Text Models: Misspelling-Proof via Projections","Projection Classifiers Shrug Off Typos, BERT Stumbles","Misspelling Attacks: LSH Nets Drop 2.94%, BERT 11.44%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The robustness result rests on the assumption that randomly dropping, inserting, and swapping internal characters in 20% of test words captures the misspellings real users produce; if real misspellings follow different patterns, such as keyboard-adjacent or phonetic errors, the measured accuracy ordering may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Hash-Projection Models Survive Misspellings Better Than BERT","LSH Text Nets: 2.94% Drop vs BERT's 11.44% Under Spelling Attacks","On-Device Text Models: Misspelling-Proof via Projections","Projection Classifiers Shrug Off Typos, BERT Stumbles","Misspelling Attacks: LSH Nets Drop 2.94%, BERT 11.44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3104,"prompt_tokens":929,"completion_tokens":2175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2063}},"tokens_in":545,"tokens_out":2175,"duration_ms":14236,"temperature":1.0,"reasoning_tokens":2063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:21:18.914808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four classifiers on test sets corrupted by realistic misspellings rather than random character edits, for example keyboard-adjacent substitutions and phonetic typos at the same per-word rate, and compare accuracy drops; if fine-tuned BERT or the BiLSTM baselines lose less accuracy than SGNN and ProSeqo, the claimed inherent robustness is an artifact of the perturbation protocol.","supporting_citations":[{"cited_title":"ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections","cited_arxiv_id":"1708.00630","evidence_quote":"Introduces the LSH projection function and neural projection networks that the paper's robust representations are built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SGNN model and fixes the projection dimension K = 1120 used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MRDA dialog act corpus used in the classification experiments."}],"review_version":1}