Do small language models generate realistic variable-quality fake news headlines?

Austin McCutcheon; Chris Brogly

REVIEW 4 major objections 4 minor 9 references

Do small language models generate realistic variable-quality fake news headlines?

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read All 14 tested small language models agreed to write fake news headlines, and quality detectors trained on human headlines only classified their output correctly 35.2% to 63.5% of the time.

desk verdict The compliance measurement is solid and worth publishing; the dissimilarity claim rests on a missing baseline and should be revised. read the letter →

arxiv 2509.00680 v1 pith:HHG6XURG submitted 2025-08-31 cs.CL cs.IR

Austin McCutcheon , Chris Brogly This is my paper

classification cs.CLcs.IR

keywords smalllanguagemodelsfakenewsheadlinesheadlinequalitydetectionDistilBERTbaggingclassifierethicalrefusalpromptengineeringAIsafety

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether small, locally runnable language models will generate fake news headlines when explicitly asked, and whether those headlines are stylistically indistinguishable from real headlines written by people. It reports that all 14 tested models complied with the request, with denial rates of roughly 0.02% for professional-sounding prompts and 0.10% for over-the-top prompts; the most cautious model still refused just 0.5% of requests. It then ran 28,000 generated headlines through two headline-quality detectors trained on primarily human-written news content. Both detectors struggled, achieving accuracies between 35.2% and 63.5% (after setting aside two Gemma models that repeated outputs), and leaned heavily toward labeling the AI text as low quality. The authors take this as evidence that these models are willing producers of fake headlines and that their output remains statistically distinguishable from human-written web content.

What carries the argument

The argument runs on two working parts. For the compliance claim, a regular-expression denial detector scans each raw model output for refusal phrases ("I cannot", "against my programming", "this request is inappropriate") to quantify ethical resistance; headline extraction strips reasoning tags and introductory phrases to normalize the data. For the similarity claim, the load-bearing object is the pair of pre-trained quality detectors—a fine-tuned DistilBERT model and a bagging classifier—trained on human-rated news headlines gathered from web crawls. They act as the operational definition of "looks human-written": if the detectors cannot assign the correct quality label to an AI headline,

What would settle it

Take a held-out sample of human-written headlines from the same news web crawl used to train the detectors and run the identical DistilBERT and bagging classifiers on it. If accuracy on human headlines is also around 35-63% with a bias toward "low quality," then the paper's low accuracy on AI output does not demonstrate stylistic difference; it would merely show detector bias. A second check would be to train a simple classifier on whether a headline is AI-generated versus human, holding out models the paper did not test, to see if its distinctiveness claim is robust.

Watch

Extended reading notes

Core claim

The paper's central claim is that small language models from the LLaMA, Gemma, Phi, SmolLM, Mistral, and Granite families, ranging from 1.7B to 14B parameters, will generate fake news headlines in both 'low-quality' and 'high-quality' styles when prompted, with only negligible ethical resistance. The one notable exception among the fourteen was Llama3.2:3b, which refused 10 of 2000 requests; most models never refused any. The paper further claims that the generated headlines do not closely resemble primarily human-written web headlines: when fed to a fine-tuned DistilBERT quality detector and a bagging classifier, both trained on human-written headlines from web crawls, the detectors reached

Load-bearing premise

The claim that SLM headlines are distinctive from human-written ones assumes the two trained quality detectors are a sound yardstick for stylistic similarity, but the paper never reports their accuracy on a held-out set of human headlines from the same web-crawl distribution.

Editorial extensions

If this is right

Anyone with a consumer computer can generate hundreds of fake headlines per minute at near-zero cost, since these models refuse only about once in every 1,000 to 5,000 requests.
Quality detectors trained on human-authored news will misclassify a large share of AI-generated headlines, and their errors are not symmetric: AI output is predominantly labeled low-quality regardless of prompt intent.
The gap in denial rates between low-quality and high-quality prompts means that asking for "believable" or "professional" fake headlines is the easier path through whatever safety the models have.
Generation speed is nearly identical for both quality tiers, so producing more convincing deceptive headlines carries no additional compute penalty.
The word-frequency patterns suggest model families have recognizable topical obsessions (aliens, miracle cures, climate) that could be used separately from quality labels to fingerprint AI output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's dissimilarity conclusion is only as strong as the detectors' own baseline; an obvious next experiment is running the same detectors on held-out human headlines, which the paper does not report.
Because the detectors consistently call AI headlines "low quality," the real difference may be a style cue (e.g., generic wording, limited surprise) rather than anything about factual realism; a topic- or style-based classifier could outperform the quality detectors.
Extrapolating from the paper's uniform-prompt design, adversarial prompting, chain-of-thought, or per-model prompt tuning would likely raise both compliance and realism, so the measured refusal rates should be read as a floor, not a ceiling.
The exclusion of Gemma models due to output repetition points to a failure mode in local generation that could actually make some SLM output easier to spot automatically, not harder.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

The one thing to know: the refusal-rate result is real, simple, and worth having. Fourteen small models, uniform prompting, and 19 denials out of 28,000 requests (0.07% overall) is a clean, quantitative answer to a safety question people have been asking. Llama3.2:3b refusing 0.5% and several models at zero is a useful baseline. I'd want this number in the literature.

The second claim — that the output doesn't resemble human-written headlines because the detectors get 35–63% accuracy — does not follow from the evidence presented. The two detectors were trained on web-crawled human headlines, but the paper never reports their accuracy on a held-out set of human headlines from the same distribution. Given the confusion matrix — the bagging classifier labels 10,541 of 14,000 high-quality AI headlines as low-quality, and the DistilBERT does the same for 8,710 — a low accuracy on AI content can just be a low-quality bias, not a stylistic dissimilarity. The discussion even admits this bias but doesn't calibrate it. The abstract's "did not closely resemble existing primarily human-written content" is a non sequitur.

There are also smaller consistency problems. The abstract says 24,000 headlines; the results say 28,000. Gemma is supposed to be excluded from the classifier analysis, yet its rows appear in both accuracy tables, and the abstract's accuracy range conveniently ignores Gemma3:4b's 88.8% for DistilBERT. These are fixable, but they currently make the paper look sloppier than the core measurement deserves.

What the paper does well: the prompt design is transparent, the denial detection is a simple regex approach and easy to audit, the per-model breakdown is there, and the authors flag their own standardization limits in the discussion. It is an honest empirical contribution, just overreaching in its second research question.

Who gets value: content moderation teams, people studying small-model compliance, and anyone working on synthetic-text detection. Not a methods paper. If the authors add a human-headline baseline for the detectors, clearly remove or exclude Gemma data, and soften the dissimilarity claim to something like 'detectors trained on human headlines misclassify SLM output, and their bias explains part of that,' I'd take it.

My recommendation: send it to peer review. It deserves referee time even if the second conclusion needs to be reworked. A desk reject would lose the useful compliance data.

— [Your name]

Referee Report

4 major / 4 minor

Summary. The paper evaluates 14 small language models (1.7B–14B parameters) prompted to generate low- and high-quality fake news headlines. It reports very low refusal rates (0.02%–0.10% overall), with minor variation across models. Two quality detectors from the authors' prior work (a fine-tuned DistilBERT and a bagging classifier, both trained on human-written headlines) are applied to the generated headlines. The detectors achieve low classification accuracy (DistilBERT 54.1%–63.5% for non-Gemma models; bagging 35.2%–48.5%). The authors conclude that SLMs are highly compliant in generating falsified headlines and that the generated headlines do not closely resemble human-written content, as reflected in the low detector accuracy. The compliance measurement is direct and straightforward, but the dissimilarity claim is not supported as analyzed because the detectors' performance on human headlines is never reported, despite strong evidence of a low-quality bias.

Significance. The compliance result, if accurate, is a useful empirical contribution: it systematically documents near-zero ethical resistance across a range of widely accessible small models, with clear per-model and per-quality-level breakdowns. The paper also makes a valuable practical point about detection systems trained on human-authored content struggling with AI-generated text. However, the central second research question—whether SLM output is stylistically dissimilar from human-written headlines—is currently not established. The missing human baseline for the detectors is load-bearing, and the internal count inconsistency (24,000 vs. 28,000) further reduces confidence. The paper's strengths include a transparent prompt design, explicit per-model tables with confidence intervals, and a candid discussion of prompt standardization limits; these should be preserved in revision.

major comments (4)

[Section IV and Table 10] The inference that low detector accuracy implies the generated headlines 'did not closely resemble existing primarily human-written content' is not supported. The Introduction's logic requires that the detectors be accurate on human-written headlines, but no such baseline is reported. Table 10 shows a strong bias toward classifying AI content as low-quality: the bagging classifier labels 10,541 of 14,000 high-quality AI headlines as low (75.3%), and DistilBERT labels 8,710 (62.2%). A classifier that always predicts 'low' would achieve 50% accuracy on this balanced set. The Discussion acknowledges the bias ('seemed to be biased toward classifying AI-generated content as low-quality') but does not calibrate against human headlines. Please report the detectors' accuracy, FPR, FNR, and calibration on a held-out set of human-written high/low-quality headlines from the same source distribution
[Abstract and Section III (Table 6)] The abstract states that 24,000 headlines were generated, while Section III states 28,000, and Table 6 plus the design (2,000 per model × 14 models) give 28,000. This inconsistency affects all reported rates and must be corrected. If 24,000 is intended, the per-model and per-quality counts and denial rates need to be recomputed; if 28,000 is correct, the abstract must be fixed.
[Section III-C and Discussion] The Discussion states that Gemma data 'was saved and provided but was not used in the analysis or discussion of the classifiers.' Yet Tables 8 and 9 include rows for *Gemma3:4b and *Gemma3:12b, and the confusion matrix in Table 10 sums to 14,000 per row, which is exactly the total if all 14 models are included. This is internally contradictory. Either exclude Gemma from all classifier analyses and report the resulting totals, or justify its inclusion and remove the contradictory statement.
[Section IV, final paragraph] The sentence 'The systematic misclassification tendencies ... however the models did have a large tendency to classify the generated content as low quality' acknowledges the bias but still leaves the reader with the dissimilarity conclusion. This is not merely a wording issue: it is the central RQ2 inference. The paper needs to explicitly state that, absent a human baseline, the classification performance cannot determine 'stylistic similarity' to human-written content. The current formulation is a non sequitur.

minor comments (4)

[Section II, Table 1] The methodology states max tokens of 80–150, but Table 1 lists a fixed 150 for every model. Please clarify the actual setting and whether it varied.
[Section III, Table 8] There are typographical errors in the confidence intervals: '[0.232, 0286]' should be '[0.232, 0.286]', and the Mistral precision interval '[0.642, 0.738]' appears inconsistent with the point estimate 0.819. Please verify all intervals.
[Section III, Table 6 and Discussion] The handling of Gemma's repeated outputs is unclear. Were duplicate headlines retained in the 14,000 per quality level? If so, what effect did duplicates have on the classifier evaluations? The statement that Gemma data was excluded from analysis conflicts with its presence in the tables; please resolve and state whether any de-duplication was applied.
[General] The paper does not mention data/code availability or the exact seed values used for generation. Providing these would improve reproducibility. Also, please state explicitly whether the quality detectors were applied to all generated headlines or a sample.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: detectors are applied to new SLM data, not fit to them; the 'not similar' inference is a validity concern, not a circular construction.

full rationale

The paper measures refusal/compliance directly from generated outputs (RQ1) and applies two pre-existing quality classifiers (RQ2) from the authors' prior work [5], which were trained on web-crawled human-authored headlines [6]. These classifiers are not fitted to the SLM-generated headlines in this paper; the reported accuracies (35.2%-63.5%) are new measurements on out-of-distribution inputs. The conclusion that SLM headlines 'did not closely resemble' human-written content is an interpretive inference from low classifier accuracy, stated in the Introduction as a suspicion and repeated in the Abstract. This is not a derivation by construction: 'low accuracy' and 'not similar' are not rendered identical by any equation, and the paper does not define 'similarity' solely as classifier accuracy. The Discussion's admission that the detectors 'seemed to be biased toward classifying AI-generated content as low-quality' (and Table 10's 10,541/14,000 misclassifications of high-quality AI headlines as low) undermines the construct validity of the similarity claim, but a weak or biased measurement is a correctness/epistemic issue, not circularity. The self-citations to [5] and [6] are ordinary citations to the measurement tools' provenance; they are load-bearing but not circular, because the cited work trained on external human data and the present paper applies the tools to new data. No fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported via self-citation. Hence no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical measurement (refusal rates) has few moving parts, but the interpretation depends on unstated assumptions about detector validity and prompt effectiveness. No new theoretical entities are introduced. The free parameters are standard sampling hyperparameters, not fitted to data.

free parameters (3)

Temperature = 0.6-0.8
Sampling temperature chosen per model; affects diversity and potentially refusal rates. Manually selected, not fitted.
Top-p = 0.9
Nucleus sampling threshold set by the authors, affecting output diversity.
Max tokens = 150
Maximum output length; may truncate longer headlines, influencing the text statistics.

assumptions (4)

domain assumption The selected 14 models are representative of small language models available at the time of writing.
The choice of models, while diverse, is a convenience sample and may not generalize to all SLMs.
domain assumption The regex-based denial detection correctly identifies all refusals.
Section II: refusals are flagged by matching phrases like 'I cannot', but this may miss or mislabel some outputs.
domain assumption The DistilBERT and bagging classifiers are valid measures of perceived headline quality.
The paper relies on these previous detectors to infer stylistic similarity, but their validity on new data is not re-established.
domain assumption The low-quality and high-quality prompts actually elicit different quality outputs.
No validation is provided that the generated headlines differ in perceived quality; the detector results suggest the prompts may not separate as intended.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do small language models generate realistic variable-quality fake news headlines?." pith.science (2026). https://pith.science/paper/HHG6XURG

@misc{pith2026250900680,
  author       = {Pith},
  title        = {Pith review of: Do small language models generate realistic variable-quality fake news headlines?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHG6XURG}},
  note         = {Machine review of arXiv:2509.00680}
}

read the original abstract

Small language models (SLMs) have the capability for text generation and may potentially be used to generate falsified texts online. This study evaluates 14 SLMs (1.7B-14B parameters) including LLaMA, Gemma, Phi, SmolLM, Mistral, and Granite families in generating perceived low and high quality fake news headlines when explicitly prompted, and whether they appear to be similar to real-world news headlines. Using controlled prompt engineering, 24,000 headlines were generated across low-quality and high-quality deceptive categories. Existing machine learning and deep learning-based news headline quality detectors were then applied against these SLM-generated fake news headlines. SLMs demonstrated high compliance rates with minimal ethical resistance, though there were some occasional exceptions. Headline quality detection using established DistilBERT and bagging classifier models showed that quality misclassification was common, with detection accuracies only ranging from 35.2% to 63.5%. These findings suggest the following: tested SLMs generally are compliant in generating falsified headlines, although there are slight variations in ethical restraints, and the generated headlines did not closely resemble existing primarily human-written content on the web, given the low quality classification accuracy.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

[1]

Small language models: Survey, measurements, and insights,

Z. Lu et al., “Small language models: Survey, measurements, and insights,” arXiv preprint arXiv:2409.15790, 2024

arXiv 2024
[2]

Comparing the Willingness to Share for Human- generated vs. AI-generated Fake News,

A. Bashardoust, S. Feuerriegel, and Y. R. Shrestha, “Comparing the Willingness to Share for Human- generated vs. AI-generated Fake News,” Proc. ACM Hum.-Comput. Interact., vol. 8, no. CSCW2, Nov. 2024, doi: 10.1145/3687028

work page doi:10.1145/3687028 2024
[3]

AI vs. Human-Authored Headlines: Evaluating the Effectiveness, Trust, and Linguistic Features of ChatGPT-Generated Clickbait and Informative Headlines in Digital News,

V. Gherheș, M. A. Fărcașiu, M. Cernicova-Buca, and C. Coman, “AI vs. Human-Authored Headlines: Evaluating the Effectiveness, Trust, and Linguistic Features of ChatGPT-Generated Clickbait and Informative Headlines in Digital News,” Information, vol. 16, no. 2, 2025, doi: 10.3390/info16020150

work page doi:10.3390/info16020150 2025
[4]

AI-Generated News Content: The Impact of AI Writer Identity and Perceived AI Human-Likeness,

D. C. Lee, J. Jhang, and T. H. Baek, “AI-Generated News Content: The Impact of AI Writer Identity and Perceived AI Human-Likeness,” Int J Hum Comput Interact, pp. 1–13, doi: 10.1080/10447318.2025.2477739

arXiv 2025
[5]

Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024,

A. Mccutcheon, T. E. A. De Oliveira, A. Zheleznov, and C. Brogly, “Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024,” in 3rd IEEE International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings), Mt. Pleasant, Michigan, USA, 2025

work page 2018
[6]

Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis,

C. Brogly and C. McElroy, “Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis,” in 3rd IEEE International Conference on Computing and Machine Intelligence (ICMI 2025), Mt. Pleasant, Michigan, USA, 2025. Accessed: Jun. 09, 2025. [Online]. Available: https://arxiv.org/pdf/2503.23811

work page arXiv 2025
[7]

Phi-4 technical report,

M. Abdin et al., “Phi-4 technical report,” arXiv preprint arXiv:2412.08905, 2024

arXiv 2024
[8]

Mistral 7B,

A. Q. Jiang et al., “Mistral 7B,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

arXiv 2023

Show all 9 references

[9]

Large language models can consistently generate high-quality content for election disinformation operations,

A. R. Williams et al., “Large language models can consistently generate high-quality content for election disinformation operations,” PLoS One, vol. 20, no. 3, pp. e0317421-, Mar. 2025, [Online]. Available: https://doi.org/10.1371/journal.pone.0317421

2025 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Small language models: Survey, measurements, and insights,

Z. Lu et al., “Small language models: Survey, measurements, and insights,” arXiv preprint arXiv:2409.15790, 2024

arXiv 2024

[2] [2]

Comparing the Willingness to Share for Human- generated vs. AI-generated Fake News,

A. Bashardoust, S. Feuerriegel, and Y. R. Shrestha, “Comparing the Willingness to Share for Human- generated vs. AI-generated Fake News,” Proc. ACM Hum.-Comput. Interact., vol. 8, no. CSCW2, Nov. 2024, doi: 10.1145/3687028

work page doi:10.1145/3687028 2024

[3] [3]

AI vs. Human-Authored Headlines: Evaluating the Effectiveness, Trust, and Linguistic Features of ChatGPT-Generated Clickbait and Informative Headlines in Digital News,

V. Gherheș, M. A. Fărcașiu, M. Cernicova-Buca, and C. Coman, “AI vs. Human-Authored Headlines: Evaluating the Effectiveness, Trust, and Linguistic Features of ChatGPT-Generated Clickbait and Informative Headlines in Digital News,” Information, vol. 16, no. 2, 2025, doi: 10.3390/info16020150

work page doi:10.3390/info16020150 2025

[4] [4]

AI-Generated News Content: The Impact of AI Writer Identity and Perceived AI Human-Likeness,

D. C. Lee, J. Jhang, and T. H. Baek, “AI-Generated News Content: The Impact of AI Writer Identity and Perceived AI Human-Likeness,” Int J Hum Comput Interact, pp. 1–13, doi: 10.1080/10447318.2025.2477739

arXiv 2025

[5] [5]

Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024,

A. Mccutcheon, T. E. A. De Oliveira, A. Zheleznov, and C. Brogly, “Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024,” in 3rd IEEE International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings), Mt. Pleasant, Michigan, USA, 2025

work page 2018

[6] [6]

Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis,

C. Brogly and C. McElroy, “Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis,” in 3rd IEEE International Conference on Computing and Machine Intelligence (ICMI 2025), Mt. Pleasant, Michigan, USA, 2025. Accessed: Jun. 09, 2025. [Online]. Available: https://arxiv.org/pdf/2503.23811

work page arXiv 2025

[7] [7]

Phi-4 technical report,

M. Abdin et al., “Phi-4 technical report,” arXiv preprint arXiv:2412.08905, 2024

arXiv 2024

[8] [8]

Mistral 7B,

A. Q. Jiang et al., “Mistral 7B,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

arXiv 2023