{"id":"d00a1a86-e35b-45a1-9f4f-fd8d779b9126","arxiv_id":"2412.12072","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark shows existing NLP systems find almost no novel dog whistles in social media, while the proposed EarShot pipeline raises F0.5 scores to 14.6 on synthetic Reddit, 5.7 on Gab, and 4.6 on Twitter.","lead":"This paper introduces FETCH!, a benchmark for discovering new coded hate phrases, called dog whistles, in social media posts, and tests whether existing systems can find them. It finds standard tools fail badly and proposes EarShot, a pipeline using sentence embeddings, vector search, and language models, which performs better but still at low absolute accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FETCH! scores are computed against regex/GPT-4 labels from a fixed known lexicon, so out-of-lexicon predictions are always false positives; the reported F-scores measure retrieval of known dog whistles, not discovery of novel ones.","rationale":"The reader's weakest assumption correctly identifies lexicon-derived labels as the core risk. I sharpen it: because the labels are generated from a fixed known lexicon, the benchmark cannot by construction reward genuinely novel dog whistles; Appendix G's qualitative novelty check is excluded from the reported metrics. This supports the reader's CONDITIONAL verdict rather than overturning it, since the paper may still be salvageable with a human-annotated novelty evaluation. A secondary issue, also noted by the reader, is that best thresholds and n-gram ranges are selected on the test set (Section 5, Appendix F), which likely inflates EarShot's advantage over baselines; fixing this with a held-out validation split is necessary before the quantitative comparisons can be trusted.","tokens_in":20314,"tokens_out":5141,"duration_ms":49228,"concrete_test":"Build a held-out evaluation set of 1,000-2,000 posts from Gab, Twitter, and Reddit dated after the Mendelsohn et al. lexicon was compiled, with human annotations for any dog whistles not in that lexicon; run EarShot's best configurations with fixed thresholds and compute precision/recall against these human labels. If precision on out-of-lexicon terms is at or near zero, the FETCH! scores do not support the novelty-discovery claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EarShot 'efficiently and effectively identif[ies] new dog whistles' on FETCH!. But the benchmark cannot score genuinely novel discoveries: in the Balanced and Realistic scenarios, positive labels are generated by regex-matching the Mendelsohn et al. (2023) lexicon against Gab/Twitter posts (Section 3.1.3), and even the Synthetic scenario uses GPT-4 annotations from Silent Signals rather than human labels. Any prediction that is not a surface form in that lexicon is counted as a false positive, so a system that finds a truly new dog whistle receives no credit for it. The only novelty evidence is the manual review in Appendix G, which is not part of the reported precision/DPR/F0.5 numbers. Thus the headline F-score improvements (e.g., 14.55 Synthetic, 5.70 Balanced, 4.63 Realistic) are retrieval scores over a closed lexicon, not evidence for discovery of emergent dog whistles. The paper's Section 9 concedes the lack of human-labeled corpora, but the problem is more than label noise: the evaluation design makes out-of-lexicon true positives impossible, so the task's central construct is not measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces FETCH!, a benchmark for discovering novel dog whistles in social media corpora, with three scenarios: Synthetic (Reddit/Silent Signals), Balanced (Gab), and Realistic (Twitter). Given a corpus and seed dog whistles, systems must propose additional dog whistles. The authors evaluate Word2Vec/Phrase2Vec, MLM fill-in-the-blank methods, EPD, and their EarShot pipeline, which combines sentence embeddings, nearest-neighbor retrieval, LLM/BERT filtering, and keyword extraction or direct LLM prompting. They report that EarShot outperforms the baselines by 2 to 20 F0.5 points, with best scores of 14.55 on Synthetic, 5.70 on Balanced, and 4.63 on Realistic. The paper also includes a manual review of novel dog whistles in Appendix G.","tokens_in":20467,"tokens_out":7589,"duration_ms":65300,"significance":"The task is well motivated and the resource could be useful to the community. EarShot is described in enough detail to be reimplemented, code is released, and the use of Data Potential Recall is a thoughtful adjustment for sparse positive settings. However, the evaluation design measures retrieval of known lexicon items rather than discovery of novel dog whistles, because the positive labels are produced by regular-expression matching of the Mendelsohn et al. (2023) lexicon or by GPT-4 annotations, and scoring requires exact matches to known surface forms. This construct-validity gap is acknowledged in Section 9 but not resolved, and it affects every headline number in the paper. If the paper is reframed as a retrieval baseline over known dog whistles and supplemented with a human-validated evaluation of novel discoveries, it would be a solid contribution.","major_comments":[{"comment":"The benchmark cannot credit out-of-lexicon discoveries. In the Balanced and Realistic scenarios, ground truth is generated by regular-expression matching of the Mendelsohn et al. (2023) lexicon (Section 3.1.3), and in the Synthetic scenario by GPT-4 annotations in Silent Signals (Section 3.1.1). Section 3.3 then scores a prediction only if it exactly matches a root or surface form. As a result, any prediction outside the curated lexicon is counted as a false positive, and the F0.5 numbers in Tables 2-5 measure retrieval of known dog whistles rather than discovery of emergent ones. The manual review in Appendix G is the only evidence of novel discovery, but it is not scored, not systematic, and in the Realistic scenario it found no new dog whistles (Appendix G.3). Section 9 concedes the lack of human-labeled corpora, but this is not only a label-noise problem: the evaluation design makes novel true positives impossible. I recommend adding a held-out human-annotated set of genuinely new dog whistles, or making manually verified discoveries a primary reported metric.","section":"Sections 3.1.3, 3.3, 9"},{"comment":"The reported best F0.5 scores are selected by sweeping prediction thresholds (k = 50 to 25,600 for MLM/EPD and EarShot) and taking the maximum on the test set, with no development split or a priori threshold rule. This makes the headline 2-to-20-point improvements optimistic and difficult to reproduce. The authors should either fix thresholds before test evaluation, report development-set selection, or clearly label the selected maxima as upper bounds. This issue is load-bearing because the superiority claim rests on these selected values.","section":"Tables 3-5 and Appendices D-F"},{"comment":"The abstract's claim that state-of-the-art systems fail to achieve meaningful results is stronger than the evaluation supports. The compared methods are Word2Vec/Phrase2Vec, MLM, and EPD, which are general embedding and euphemism-detection baselines; the most relevant prior dog-whistle systems (Mendelsohn et al. 2023 and Kruk et al. 2024, discussed in Section 2.4) are not evaluated as systems. The claim should be narrowed to the tested methods, or those prior systems should be included in the comparison.","section":"Abstract and Section 4.1"}],"minor_comments":[{"comment":"The abstract and introduction say state-of-the-art systems achieve less than 5% F-score, but the unigram Word2Vec model on Synthetic in Table 2 reaches F0.5 = 5.91; the claim should be qualified to the specific baselines or specific scenarios.","section":"Abstract/Table 2"},{"comment":"The first sentence of Section 6 contains a grammatical error ('We differences between models'), and there are several other typos, including 'phenomenia' in Section 2.3 and 'an modification' in Appendix G.2.","section":"Section 6"},{"comment":"The final sentence of Section 3.2 is garbled and should be split into two sentences to explain what metadata is used for stratified splitting and why the Synthetic scenario is handled differently.","section":"Section 3.2"},{"comment":"The statement that the best Realistic system found no new dog whistles should be acknowledged in the main discussion, because it substantially qualifies the discovery claim for the realistic setting.","section":"Appendix G.3"},{"comment":"The paper does not report the exact number of Gab posts collected after scraping or the composition of the corpus after filtering; additional data-collection details would improve reproducibility.","section":"Section 4.1.4 and Section 3.1.2"}],"recommendation":"major_revision","confidential_remarks":"The closed-lexicon evaluation issue is real and central; I would not accept the paper in its current form. With a human-validated novel-term evaluation, a fixed threshold-selection protocol, and a more careful comparison, the benchmark and EarShot baseline could be a useful community resource. Major revision seems appropriate rather than rejection because the code, task formulation, and reproducible baseline have value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FETCH! is a genuinely useful new benchmark for dog whistle discovery, and EarShot is a reasonable, reproducible baseline. The paper is worth engaging with, but the headline F-scores are retrieval numbers over a closed lexicon, not evidence of discovering new dog whistles.\n\nThe authors do several things right. The task formalization—seed whistles, a corpus, discover other whistles—is clear, and the three scenarios (Synthetic/Reddit, Balanced/Gab, Realistic/Twitter) give the community a shared testbed. The evaluation against Word2Vec, MLM, EPD, and the EarShot variants is systematic, and the code is on GitHub. They also deserve credit for stating plainly in Section 9 that there is no human-labeled corpus and that LLM contamination cannot be ruled out.\n\nThe soft spot is structural. In the Balanced and Realistic scenarios, positive labels come from regex-matching the Mendelsohn et al. (2023) lexicon; in the Synthetic scenario, from GPT-4 annotations. Any prediction not in that closed set is counted as a false positive, so a system that surfaces a genuinely new dog whistle gets no credit. The reported F-scores therefore measure retrieval of known surface forms, not discovery. The authors' own manual review in Appendix G is the only real novelty evidence, and it is kept outside the main metrics. That is a bigger problem than label noise: the central construct is not being measured.\n\nThere are smaller issues. Best thresholds and n-gram ranges are selected on the test set, which inflates the reported numbers. The abstract claims baselines achieve less than 5% F-score, but Table 2 shows Word2Vec at 5.91 on Synthetic. These are fixable. The main qualitative conclusion—current systems are weak and EarShot is stronger—holds up on the paper's own numbers, so I would not reject the work on that basis.\n\nThis paper is for content moderation and hate speech researchers who need a shared evaluation. With a held-out validation split, a human-annotated sample, fixed thresholds, and an honest statement that FETCH! scores retrieval of known whistles rather than discovery, it would be a solid contribution. As is, it deserves serious refereeing: the benchmark has value, and the evaluation weaknesses are repairable rather than fatal. I would send it out.","headline":"Useful benchmark, honest limitations, but the headline F-scores measure retrieval from a closed lexicon, not discovery of novel dog whistles.","tokens_in":21084,"tokens_out":2170,"would_cite":true,"duration_ms":18262,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces FETCH!, a benchmark for discovering novel dog whistles in social media, and EarShot, a baseline the authors claim raises $F_{0.5}$ by 2-20 points over prior methods.","keywords":["dog whistles","coded language","hate speech detection","euphemism detection","social media","large language models","sentence embeddings","benchmark"],"falsifier":"Have independent human annotators label a random sample of FETCH! positive posts as genuine dog whistles or not, then recompute EarShot's precision and $F_{0.5}$ against the corrected labels; if precision drops substantially on the Balanced or Realistic sets, the reported 2-20 point gains are partly artifacts of noisy automatic ground truth.","tokens_in":20040,"feed_emoji":"🐕","tokens_out":15568,"duration_ms":116614,"temperature":0.7,"pith_summary":"Dog whistles are coded phrases that say something innocuous to most readers while signaling a second, often hateful message to an in-group, which lets them slip past moderation filters. The paper tries to establish that this problem can be tackled as a discovery task: given a corpus and a small set of known seed dog whistles, a system should surface other dog whistles that are not in any lexicon. To that end it introduces FETCH!, a benchmark with three social-media case studies (dense synthetic Reddit posts, high-prevalence Gab, sparse Twitter), and EarShot, a baseline that retrieves posts similar to seed-whistle posts and then filters and extracts candidate terms. The paper reports that all three existing methods perform poorly on FETCH!, with the best prior $F_{0.5}$ at 5.91 on Synthetic and below 3 on the other two, while EarShot's best configurations reach 14.55, 5.70, and 4.63 on Synthetic, Balanced, and Realistic. A sympathetic reader would care because curated dog-whistle lexicons are slow to update, and an automated discovery pipeline could help moderators and researchers track coded hate speech as it emerges.","feed_headline":"EarShot beats prior methods at finding new dog whistles","feed_subtitle":"On FETCH!, a new benchmark spanning Reddit, Gab, and Twitter, EarShot gains 2 to 20 F-points over prior methods.","key_machinery":"FETCH! is the evaluation object: a corpus plus seed dog whistles with a stratified train/test split and Data Potential Recall as the denominator. EarShot is the mechanism that carries the argument. Its first stage maps each post to a vector using a sentence encoder, so posts similar in meaning are close together, and stores these vectors in a vector database. For every post that contains a seed whistle, the system retrieves its nearest neighbors from the database, capturing posts that share meaning, sentiment, or intent without sharing exact words. In the PREDICT path, those neighbors are filtered by a hate/toxicity classifier or an LLM yes/no prompt, then passed to a keyword extractor that returns top-k candidate terms; in the DIRECT path, an LLM reads the neighbor posts and returns dog whistles in a structured list. The load-bearing idea is that novel whistles live in the semantic neighborhood of known whistles, not in token overlap.","core_discovery":"The paper's central claim is that dog whistle discovery can be formalized and measurably improved. FETCH! splits known dog whistles into seed and held-out sets, then asks a system to find the held-out whistles in a corpus, with recall normalized by which whistles actually occur in that corpus (Data Potential Recall). On this benchmark, Word2Vec/Phrase2Vec, masked-language-model fill-in, and euphemistic-phrase detectors all perform poorly: their best $F_{0.5}$ is 5.91 on Synthetic and 2.47 or lower on the other two, which the paper reads as evidence that lexicon-based and token-filling approaches do not transfer to discovery. EarShot, by contrast, embeds every post, retrieves nearest neighbors of posts containing seed whistles, and either asks an LLM directly for the whistles or filters candidates with a hate/toxicity classifier or LLM and then extracts keywords; the best configurations reach $F_{0.5}$ of 14.55 on Synthetic, 5.70 on Balanced, and 4.63 on Realistic. The paper presents EarShot not as a finished detector but as a strong baseline that outperforms prior methods by 2 to 20 F-points while leaving subtle, recent, and emoji-based whistles largely undiscovered.","pith_inferences":["One implication the paper leaves implicit is that its automatically generated labels make the reported scores an upper-bound estimate; a human-adjudicated version of FETCH! would likely shift both precision and recall.","A natural next experiment, not run in the paper, is to apply EarShot longitudinally on time-sliced corpora and check whether its retrievals predate a whistle's appearance in curated lexicons, which would directly test emergent discovery.","The bias toward overt slurs under hate-speech filtering suggests that a filter-free variant ranking neighbors by embedding distance alone could recover recency-dependent whistles like 'jogger' or 'vegan cat' that the paper reports missing.","The same retrieval-plus-filter architecture could transfer to other coded-language domains with a seed vocabulary, such as drug slang or underground-economy jargon, because it does not depend on a particular lexicon."],"forward_implications":["Any future discovery method can be tested on the same three habitats with the same seed/holdout split, so results across systems are directly comparable.","Lexicon-based euphemism detectors such as Word2Vec, masked language models, and phrase detectors perform poorly on FETCH!, with best $F_{0.5}$ at 5.91 on Synthetic and below 3 on Balanced and Realistic, so they are unlikely to work as drop-in discovery tools.","EarShot's best configurations favor precision over recall, so a human reviewer sees relatively few false candidates, but the tradeoff is that explicit slurs are found while subtle and recent phrases are missed.","Direct LLM prompting recovers far more true dog whistles (higher Data Potential Recall) at much lower precision, and in the sparse Realistic corpus the best DIRECT run scores 0 in $F_{0.5}$.","Because recall is computed against whistles that actually appear in each corpus, a method cannot gain credit by listing known terms that are absent, so lexicon coverage alone will not inflate scores."],"supporting_citations":[{"why":"Supplies the hand-curated dog whistle lexicon used as ground truth and seed terms for the Balanced and Realistic scenarios.","marker":"Mendelsohn et al. (2023)"},{"why":"Provides the Silent Signals Reddit dataset that forms the Synthetic scenario and the prompting format EarShot reuses.","marker":"Kruk et al. (2024)"},{"why":"Supplies the sentence encoder that turns posts into vectors for EarShot's nearest-neighbor stage.","marker":"Reimers and Gurevych (2019)"},{"why":"Defines the Word2Vec/Phrase2Vec baseline and its prediction-expansion procedure.","marker":"Mikolov et al. (2013)"},{"why":"Defines the MLM baseline that samples sentences and masks seed dog whistles.","marker":"Zhu et al. (2021)"},{"why":"Defines the EPD baseline that combines phrase extraction with a masked language model.","marker":"Zhu and Bhat (2021)"},{"why":"Provides the ToxiGen BERT filter used as one EarShot-PREDICT filtering option.","marker":"Hartvigsen et al. (2022)"},{"why":"Provides the RoBERTa R4 hate-speech filter used in EarShot's best Balanced and Synthetic configurations.","marker":"Vidgen et al. (2021)"},{"why":"Provides the HateXplain BERT filter used in EarShot's best Realistic configuration.","marker":"Mathew et al. (2022)"}],"fun_headline_variants":["EarShot finds new dog whistles where old methods fail","FETCH! benchmark shows EarShot beats prior detectors by 2-20 F-points","New baseline EarShot tops dog whistle discovery on FETCH!","EarShot outperforms previous methods at finding emergent dog whistles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is produced automatically from an existing word list and from AI annotations, with no human-verified labels, so if those automatic labels are wrong every score in the paper is mismeasured.","fun_headline_variants_meta":{"raw":{"variants":["EarShot finds new dog whistles where old methods fail","FETCH! benchmark shows EarShot beats prior detectors by 2-20 F-points","New baseline EarShot tops dog whistle discovery on FETCH!","EarShot outperforms previous methods at finding emergent dog whistles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1531,"prompt_tokens":970,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":586,"tokens_out":561,"duration_ms":5851,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:17:39.805903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators label a random sample of FETCH! positive posts as genuine dog whistles or not, then recompute EarShot's precision and $F_{0.5}$ against the corrected labels; if precision drops substantially on the Balanced or Realistic sets, the reported 2-20 point gains are partly artifacts of noisy automatic ground truth.","supporting_citations":[],"review_version":1}