{"id":"22521cdc-df16-4e64-a2bb-192cc7da2587","arxiv_id":"2507.20066","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Cosine similarity between tweets and a target disinformation narrative, computed with a sentence transformer, can trace and characterize disinformation on a continuous scale, but with hand-tuned thresholds and occasional false positives.","lead":"This thesis builds a dashboard that scores how similar social media posts are to a known disinformation narrative on a continuous scale, then tests it on tweets about the 2020 election and transgender issues. A smart generalist might read it to see whether semantic similarity can track misleading narratives at scale without relying on exact keywords.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cosine similarity is stance-blind: the same-topic opposite-sentiment case in Fig. 8 (0.590 vs. 'Transgender people are harmful to society') above the 0.38 threshold is a direct counterexample, and STS-B validation does not test oppositional pairs.","rationale":"The reader's weakest assumption is precisely the load-bearing point: cosine similarity to a single narrative sentence conflates topical overlap with endorsement. Section 5.4 and Table 7 confirm this is an acknowledged failure mode, and Figure 8 gives a concrete instance where a tweet with opposite sentiment scores 0.590, above the 0.38 threshold used in Case Study 2. The STS-B validation (Section 4.1) is a general semantic-similarity benchmark; its 0.87 Pearson correlation does not measure stance sensitivity, and the rubric lacks oppositional pairs. The paper's assertion that such false positives are 'rare' is unsupported by any labeled frequency estimate. Because the central claim is that the metric is 'useful to detect, trace, and characterize' disinformation narratives, and because the dashboard's thresholds and timelines depend on this metric, the concern is load-bearing. However, the thesis is explicitly modest, documents its limitations, and provides real tooling; a CONDITIONAL verdict requiring a stance-validated evaluation is the appropriate outcome. The proposed concrete test—AUC for support vs. oppose on a stance benchmark—would settle whether the failure is systematic or rare. If AUC is high, the concern dissolves; if low, the metric needs a stance-flipping layer or a scaled-back claim.","tokens_in":25449,"tokens_out":6087,"duration_ms":62902,"concrete_test":"Evaluate the tracing model on a stance-labeled dataset (e.g., SemEval-2016 Task 6, where tweets are labeled support/oppose/neutral toward a claim). For each claim and text, compute cosine similarity exactly as in the tracing tool, then measure the AUC separating support from oppose labels. If AUC is near 0.5, cosine similarity is stance-blind and the tool does not measure disinformation alignment; if AUC is well above chance, the Figure 8 failure is an isolated anomaly rather than a systematic confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that cosine similarity between a tweet and a single target-narrative sentence quantify alignment with that narrative. The paper's own Figure 8 (Section 5.4) is a concrete counterexample: a New York Times tweet supporting transgender people, but containing the word 'harmful', scores 0.590 against 'Transgender people are harmful to society' — above the 0.38 threshold used in Case Study 2 and inside the 0.5–0.75 bracket that Table 7 warns often contains 'opposite sentiment' texts. This is not an edge case: the STS-B validation (Section 4.1) measures general semantic equivalence, not stance; the STS-B rubric's 'same topic' (0.2) category does not include oppositional pairs, so the benchmark cannot detect stance-blindness. The paper states such false positives are 'rare' without frequency evidence from an opposition-labeled set; the residual error analysis on STS-B does not measure stance agreement. Consequently, the Case Study 2 timelines for The Guardian (10 tweets) and The New York Times (20 tweets) above threshold may consist largely of critical or neutral mentions of the narrative, and the reported Fox-vs-left ratios are not established measures of disinformation spread. The same confound affects Case Study 1: tweets merely discussing the election could be counted as endorsing the stolen-election narrative. Thus the claim that the metric 'detects... disinformation alignment' is unsupported until stance-confounded similarity is separated from genuine agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a continuous semantic-similarity-based method for detecting, tracing, and characterizing disinformation narratives in social media text. A sentence-transformer embedding model is used to compute cosine similarity between tweets and a user-supplied target narrative; a tracing tool plots similarity over time, and a narrative synthesis tool clusters high-similarity tweets and generates summaries using an LLM. The similarity model is validated on GLUE STS-B (Pearson r = 0.8696, MAE = 0.1383), and the combined dashboard is demonstrated on two case studies: tracing the '2020 election was stolen' narrative in Donald Trump's tweets, and tracing 'Transgender people are harmful to society' across four news outlets. The paper claims that the continuous metric can detect, trace, and characterize disinformation without time-intensive qualitative analysis, with the caveat that some misclassifications occur.","tokens_in":25804,"tokens_out":4379,"duration_ms":43339,"significance":"If the central claim were established, the paper would contribute a practical, repeatable pipeline that complements keyword-based disinformation detection: the STS-B validation is a genuine external check on the similarity model, and the open dashboard and documented code are useful resources. The two case studies also demonstrate an accessible way to visualize narrative alignment over time and across sources. However, the strongest claim—that cosine similarity to a single target sentence measures disinformation alignment—is not supported by the current evidence, because the metric is stance-blind and the case studies lack ground truth labels for narrative agreement. The paper itself acknowledges this limitation but does not quantify its impact. The contribution is therefore promising but the empirical validation needs substantial rework before the detection/tracing claims can be accepted.","major_comments":[{"comment":"The similarity metric is stance-blind: the paper's own example shows a New York Times tweet supporting transgender people, but containing the word 'harmful', scoring 0.590 against 'Transgender people are harmful to society'. This score lies above the 0.38 threshold used in Case Study 2 and inside the 0.5–0.75 bracket that Table 7 explicitly warns 'often' contains opposite-sentiment texts. Consequently, the timelines in Section 4.4 (e.g., 10 Guardian and 20 NYT tweets above threshold) and the Fox-vs-left ratios in Table 8 are not established measures of disinformation spread; they may include critical or neutral mentions of the narrative. The claim that such false positives are 'rare' is unsupported because no opposition-labeled evaluation is reported, and the STS-B validation in Section 4.1 measures general semantic equivalence, not stance agreement. The authors should either add a stance-separated evaluation (e.g., an annotated dataset of same-topic opposing-sentiment pairs) or re-scope the claims to 'topical similarity' rather than 'disinformation alignment'.","section":"Section 5.4, Figure 8"},{"comment":"The Case Study 1 validation is circular. Tweets are selected by their similarity to the target narrative, the narrative synthesis tool summarizes the selected tweets, and the high similarity of the generated narratives to the target narrative is then cited as 'useful validation of the accuracy of the trace' (Section 4.3). Because both the selection and the confirmation use the same cosine-similarity function, resemblance is built into the procedure and does not provide independent evidence that the trace captures disinformation. An external ground truth is needed—for example, expert-coded labels of whether each tweet endorses the stolen-election claim, or a comparison against a pre-existing hand-validated timeline—before the tool can be said to detect disinformation as opposed to topical mention.","section":"Section 4.3 and Section 5.1"},{"comment":"The similarity thresholds are chosen post hoc without a principled or reproducible criterion. The text states that the 0.38 threshold was selected because it 'reduced the unrelated tweets while retaining the greatest amount of related tweets during testing', but no testing details, stability analysis, or separate validation set are provided. The 0.45 threshold in Case Study 1 is not justified at all. Since every quantitative result in Section 4.4 (Figures 4–6, Table 8) depends directly on these thresholds, and since the STS-B error analysis in Table 1 shows the largest errors occur in the 0.2–0.4 human-score range where these thresholds lie, the threshold choice is load-bearing. The authors should report results across a range of thresholds or derive thresholds from a labeled development set.","section":"Section 3.3"},{"comment":"The narrative synthesis tool is not validated for the claimed characterization contribution. The paper relies on the base model's performance on unrelated commonsense benchmarks and asserts that hallucination 'did not appear to occur', but no systematic evaluation of the generated narratives against human expert coding is reported. Section 5.4 acknowledges this gap and calls for expert comparison as future work. For a paper whose stated contributions include characterization, this is a load-bearing omission; a small manual evaluation comparing, say, the six generated narratives in Case Study 1 and Table 13 with independent human summaries would materially strengthen the claim.","section":"Section 5.1 and Section 5.4"},{"comment":"The STS-B validation is a useful external benchmark, but it does not address the operational regime of the case studies. The model's absolute error is highest for sentence pairs humans rate as dissimilar (average absolute error 0.1424 at human score 0.0 and 0.2043 at 0.25), which is exactly the region where the case-study thresholds (0.38 and 0.45) are set. Reporting only the aggregate Pearson r and MAE obscures this. The authors should report performance separately for the low-similarity bracket and discuss how the measured overestimation bias affects threshold-based detection.","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"The figure numbering is inconsistent: Figure 3 is used both for the dashboard features in Section 4.2 and for the Case Study 1 timeline in Section 4.3. Please renumber the figures throughout.","section":"Section 4.2 and Section 4.3"},{"comment":"The column headers 'Ratio of Total Tweets to Fox News' and 'Tweets>0.38 Similarity Ratio to Fox News' are reversed relative to the interpretation in the text (the text says Fox tweets are at a 1.6/1 ratio to The Gateway Pundit, i.e., Fox/outlet, not outlet/Fox). Please clarify the direction of the ratios.","section":"Table 8"},{"comment":"The specific similarity model is not identified (only described as a distilled sentence transformer with 384-dimensional embeddings). Please name the model and version, since STS-B results are model-specific.","section":"Section 3.1"},{"comment":"The sentence 'Don't do that. I won't grant you access if you plan to do that.' in the ethics discussion is informal for a journal article; consider rewording to a professional statement of the access-control policy.","section":"Section 5.4"},{"comment":"The paper does not specify the exact dates for which Trump's tweets were retrieved beyond '01/01/2020 to 01/01/2021', but the text in Section 5.2 refers to a tweet from 05/01/2020; please ensure the date format is consistent.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a master's thesis, and its tool-building contributions are genuine and reproducible. The main concern is that the headline claim—that the metric detects and traces disinformation—is not yet supported because of the stance-blindness confound and the lack of ground-truth validation in the case studies. I believe the paper can be repaired within its own scope by adding a stance-labeled evaluation, a principled threshold analysis, and a small expert comparison for the narrative synthesis tool, so I do not recommend rejection. The journal should also consider whether the descriptive, thesis-style presentation meets its standards for empirical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chaytan,\n\nRead the thesis. My take: it's a solid, clearly written tool-building paper with an honest limitations section, but the main empirical claim is softer than the abstract suggests. The stress-test note is right about the stance-blind issue: the paper's own Figure 8 and Table 7 admit that same-topic opposite-sentiment text lands in the 0.5–0.75 bracket, and the New York Times counterexample scores 0.590. Since thresholds are set post hoc and the case studies lack ground-truth labels, the reported Fox-vs-left ratios may be counting critical and neutral coverage as alignment. That is not a fatal flaw in the tool's design, but it does mean the paper has not actually shown that the metric separates disinformation alignment from topical overlap.\n\nWhat it does well: the GLUE STS-B validation (Pearson 0.8696, MAE 0.1383) is a real external check on the similarity model, and the residual error analysis is genuinely informative. The dashboard itself is a practical contribution, and the author is transparent about the narrative synthesis tool's limits, the hallucination risk, and the confirmation-bias danger. The literature review is reasonable and the framework of five foundational questions is a useful organizing device. Also, the limitation passage about the 'harmful' keyword is explicit, not buried; that should count in the author's favor.\n\nThe circularity is the bigger problem. Using similarity to the target to select tweets, then citing generated narratives that resemble the target as validation of the trace, doesn't add independent evidence. The case studies are demonstrations, not validation. The narrative synthesis tool is unvalidated against expert coding, and the hyperspecific Guardian narratives show why.\n\nBottom line: this deserves a serious peer-review look, but would need major revision before acceptance. The author needs an opposition-labeled or stance-labeled evaluation set, a held-out validation where thresholds are set on training data, and ideally a comparison of traced narratives against human-coded ground truth. The tool is useful enough that reviewers should engage with it, but the paper as written overclaims what the metric establishes.\n\nI'd bring it to reading group as a case study in semantic similarity pitfalls, though I wouldn't cite it in my own work until the stance issue is addressed.","headline":"A cleanly written thesis with a genuinely useful dashboard and honest limitations, but the central detection claim is undercut by stance-blind cosine similarity and circular case-study validation; worth sending out, but only with major revisions.","tokens_in":26283,"tokens_out":1700,"would_cite":false,"duration_ms":21347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A similarity score computed from a single target narrative sentence can detect, trace, and characterize disinformation across large social media datasets, and does so in general agreement with human similarity judgments, subject to…","keywords":["disinformation detection","semantic similarity","cosine similarity","sentence transformers","narrative tracing","social media analysis","continuous measurement","STS-B benchmark"],"falsifier":"Count the failure class directly: take a few thousand labeled tweets on a contested topic in which half endorse a narrative and half oppose it using overlapping vocabulary, run them through the tracing tool, and measure the false-positive rate among the opposing half at the paper's thresholds; the paper already exhibits one instance (a pro-trans tweet scoring $0.590$ against the anti-trans narrative), but whether the method's detection claim stands depends on how often such inversions occur — a number the paper does not report.","tokens_in":25265,"feed_emoji":"📊","tokens_out":15804,"duration_ms":133157,"temperature":0.7,"pith_summary":"This paper tries to establish that a continuous similarity score — computed by embedding a tweet and a target narrative sentence, then taking their cosine similarity — can detect, trace, and characterize disinformation in large social media datasets without time-intensive qualitative coding. The claim matters because disinformation typically mixes falsehoods with partial truths, so a binary in-or-out label misses the gray area; a continuous measure of how close a post is to a known narrative would let researchers see the strength and evolution of alignment. The author builds two tools on this idea: a tracing tool that scores every tweet against a target narrative and graphs the scores as a timeline, and a narrative synthesis tool that clusters high-scoring tweets and summarizes their dominant themes. The scoring model is validated on the STS-B benchmark, where its scores correlate at $0.8696$ with human similarity judgments, and both tools are demonstrated on two cases: the \"2020 election was stolen\" narrative in Donald Trump's tweets, and the \"transgender people are harmful to society\" narrative across four news outlets. The author's stated finding is that this form of detection is generally aligned with human similarity judgments, although some misclassifications occur.","feed_headline":"One narrative sentence can trace disinformation across social media","feed_subtitle":"A continuous score validated against human judgment lets researchers trace a narrative's spread without manual coding.","key_machinery":"The load-bearing mechanism is a sentence-transformer similarity model that encodes each tweet and the target narrative into 384-dimensional numerical vectors (embeddings), so that \"how close is this tweet to the narrative\" reduces to the cosine similarity of two vectors. Because the resulting score is continuous, every tweet receives a degree of alignment rather than a yes-or-no label; a similarity threshold (set to $0.45$ for the election case and $0.38$ for the anti-trans case) selects the tweets that form a traceable timeline. The same embeddings feed a K-means clustering step, and the tweets in each cluster are summarized by an instruction-tuned generation model into \"dominant narratives,\" which is how the tool characterizes what the detected disinformation actually says. Validation of the similarity model against human judgments on the STS-B benchmark is what licenses the whole pipeline: the $0.8696$ correlation is the evidence that the continuous scores mean something to humans.","core_discovery":"The paper's central claim is that disinformation is better measured as a continuous quantity than classified as a binary label: the cosine similarity between a tweet's embedding and the embedding of a known disinformation narrative tells you how strongly that tweet aligns with the narrative, even when the tweet shares only part of the narrative's meaning. On this basis, the author claims, researchers can detect, trace, and characterize subtle patterns of disinformation spread across datasets too large for qualitative coding, and the resulting scores will generally agree with human judgments about textual similarity. The evidence offered has two parts. The similarity model achieves a $0.8696$ Pearson correlation with human ratings on the STS-B benchmark, with error concentrated on pairs humans rate as most dissimilar, which the model overestimates. The two case studies then demonstrate the claim in the wild: the election-hoax timeline spikes in November 2020 but also shows doubt-sowing months before the election, and Fox News produces anti-trans narrative content at rates far exceeding its share of total tweets, with high-similarity tweets numerous only for the two right-leaning outlets. The author's summary finding is that this continuous form of detection aligns with human similarity judgments in general, subject to documented misclassifications, most notably same-topic tweets with opposite sentiment.","pith_inferences":["If the metric works as claimed, a monitoring use follows that the paper does not develop: track a target narrative's mean or median similarity across a corpus over time and flag anomalous rises, turning the tracing tool into an early-warning signal rather than a post-hoc analysis.","A direct testable extension is the paper's own hinted polarity flip: attach a stance classifier to each high-scoring tweet and invert the similarity score when stance opposes the narrative; the open question is whether the anti-trans case study's New York Times false positives drop below threshold without losing true positives.","Because the score measures similarity to one chosen sentence, results will depend on how the target narrative is phrased; a practical consequence is that cross-study comparisons will need canonical narrative phrasings, a standardization problem the paper does not address.","The same continuous framing transfers naturally to other \"partial truth\" domains — climate, vaccine, or migration narratives — where the main validation burden is to confirm that the embedding model's similarity judgments match the distinctions a domain expert would draw."],"forward_implications":["Whole-sentence target narratives replace keyword lists: the tool catches tweets such as \"I won the election!\" (scoring $0.489$) or mentions of \"unsecured\" ballots that a keyword search would likely miss.","The timelines expose temporal structure that frequency counts flatten — Case Study 1 shows pre-election \"seeds of doubt\" followed by a post-election spike, consistent with prior findings on the election-hoax narrative.","Cross-outlet comparison becomes quantitative: Fox News exceeds a $0.38$ similarity threshold at $2.0\\times$ the rate of The Gateway Pundit, $7.1\\times$ the New York Times, and $14.1\\times$ The Guardian — far beyond its $1.6$–$2.5\\times$ share of total tweets — and tweets above $0.5$ similarity are numerous only for the two right-leaning outlets.","The method transfers to any textual data — other platforms, longer documents, video transcripts — because the pipeline only requires a body of text and a target narrative.","Detection thresholds must absorb the model's documented bias: it overestimates similarity on low-similarity pairs (signed error $0.1276$ at human score $0$), so thresholded detection will over-flag rather than under-flag, and same-topic opposite-sentiment tweets near $0.5$ are the known failure mode."],"supporting_citations":[{"why":"Supplies the information-disorder definitions (disinformation, misinformation, malinformation) and the framing the paper revises into its five foundational research challenges.","marker":"(Wardle & Derakhshan, 2017)"},{"why":"Introduced the STS-B benchmark and the human 0–5 similarity rubric that the scoring model is validated against.","marker":"(Cer et al. 2017)"},{"why":"Introduced the GLUE benchmark suite that frames STS-B as a standard NLP validation task.","marker":"(Wang et al., 2019)"},{"why":"Defines semantic similarity and cosine similarity of embeddings, the core metric the tracing tool computes.","marker":"(Chandrasekaran & Mago, 2021)"},{"why":"The 30-million-tweet mixed-methods disinformation study that motivates the paper by showing the scale and cost of hand-curated narrative detection.","marker":"(Kennedy et al., 2022)"},{"why":"The Trump Twitter Archive that supplies the 12,236 tweets for Case Study 1.","marker":"(Brown, 2016)"},{"why":"Source of the four news outlets' tweets analyzed in Case Study 2.","marker":"(Junkipedia, 2024)"},{"why":"Prior experimental evidence that Trump's election tweets undermined democratic trust, used as ground truth that Case Study 1's patterns are real.","marker":"(Bowler et al. 2022)"},{"why":"Prior study of The Gateway Pundit's posting behavior, the reference point for selecting that outlet in Case Study 2.","marker":"(Starbird, DiResta & DeButts, 2023)"}],"fun_headline_variants":["A similarity score turns tweets into disinformation timelines","Semantic similarity maps disinformation spread across tweets","Continuous similarity measures capture partial disinformation","LLM embeddings trace narratives from truth to disinformation","From election hoax to anti-trans: similarity-based tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cosine similarity between a tweet's embedding and a single target-narrative sentence measures how much that tweet actually endorses the narrative; the paper's own results show this premise fails for same-topic, opposite-sentiment tweets, which can score as high as $0.590$ (a pro-trans New York Times tweet against \"Transgender people are harmful to society\"), and if that failure is frequent, the timelines and thresholds are not measuring narrative alignment as claimed.","fun_headline_variants_meta":{"raw":{"variants":["A similarity score turns tweets into disinformation timelines","Semantic similarity maps disinformation spread across tweets","Continuous similarity measures capture partial disinformation","LLM embeddings trace narratives from truth to disinformation","From election hoax to anti-trans: similarity-based tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1643,"prompt_tokens":1064,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":680,"tokens_out":579,"duration_ms":5793,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:50:14.313323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the failure class directly: take a few thousand labeled tweets on a contested topic in which half endorse a narrative and half oppose it using overlapping vocabulary, run them through the tracing tool, and measure the false-positive rate among the opposing half at the paper's thresholds; the paper already exhibits one instance (a pro-trans tweet scoring $0.590$ against the anti-trans narrative), but whether the method's detection claim stands depends on how often such inversions occur — a number the paper does not report.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the information-disorder definitions (disinformation, misinformation, malinformation) and the framing the paper revises into its five foundational research challenges."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the GLUE benchmark suite that frames STS-B as a standard NLP validation task."},{"cited_title":"S., Garcia-Camargo, I., Spiro, E","cited_arxiv_id":null,"evidence_quote":"The 30-million-tweet mixed-methods disinformation study that motivates the paper by showing the scale and cost of hand-curated narrative detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Trump Twitter Archive that supplies the 12,236 tweets for Case Study 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the four news outlets' tweets analyzed in Case Study 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior experimental evidence that Trump's election tweets undermined democratic trust, used as ground truth that Case Study 1's patterns are real."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior study of The Gateway Pundit's posting behavior, the reference point for selecting that outlet in Case Study 2."}],"review_version":1}