{"id":"4817feb1-d65c-462d-9930-462d4a8a28fa","arxiv_id":"2505.23276","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Arabic text written by LLMs carries detectable stylometric signatures, and fine-tuned XLM-RoBERTa detectors reach near-perfect F1 on academic abstracts but degrade on social media.","lead":"Researchers built Arabic academic and social media datasets, generated text with four large language models, and found stylometric patterns that separate machine-written from human-written Arabic. They then trained detectors that reach up to 99.9% F1-score on academic abstracts but struggle more on informal social media text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-reference ASJP abstracts are unvalidated: multilingual segmentation has no measured error rate and no AI-contamination check, so the near-perfect detection results could be artifacts of a corrupted gold standard.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: every stylometric contrast and detector label in the academic half of the paper is defined relative to the ASJP-derived human corpus. The authors are transparent about the multilingual single-block format and the need for custom segmentation (Section 3.1), but transparency without validation does not establish that the pipeline works. An unmeasured error rate in segmentation is not a minor implementation detail: Arabic script extraction from PDF is notoriously error-prone, and the same extraction pipeline was used both for the human corpus and for the content-conditioned generation prompts, so any systematic artifact could be shared by both sides or could differentially contaminate the human side. Because XLM-RoBERTa is powerful enough to exploit shallow format cues, the 99.9% F1 result cannot be interpreted as evidence about stylometric fingerprints until the gold standard is audited. The proposed manual audit directly tests this: if the corrected subset reproduces the results, the concern is resolved; if not, the academic-domain claims should be weakened. The social-media results rest on BRAD/HARD, which are established human-written collections and are not under the same suspicion. I therefore keep the reader's CONDITIONAL verdict rather than escalating; the burden is a data-authentication check, not a demonstrated contradiction.","tokens_in":15703,"tokens_out":8561,"duration_ms":100037,"concrete_test":"Take a random sample of 200 ASJP-derived 'human' abstracts from the released dataset; two native Arabic speakers, with access to the original ASJP page, mark whether each record is exactly one abstract, entirely Arabic, and aligned with the paper metadata, and measure the error rate. Also run a sentence-level language identifier on all 3,000 abstracts, flagging any English/French fragments in the 1,619 'Arabic-only' subset. If the error rate exceeds 5% or contamination appears, rerun the Section 5 stylometric comparisons and Tables 9-10 detection experiments on the manually corrected subset; a material drop in F1 or Zipf separation would confirm that the reported signatures were partly segmentation artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the 'human' side of the human-vs-machine contrast is genuinely human Arabic academic prose. Section 3.1 says the ASJP site stores Arabic, English, and French abstracts in a single text block and that the authors wrote 'custom segmentation scripts' using statistical and rule-based methods, but no error rate, manual audit, or inter-annotator agreement is reported for those scripts. The only AI-contamination control is the 2010-2022 publication window; no check for machine-translated or machine-assisted abstracts is mentioned. If segmentation merges, truncates, or mislabels language segments, the human class contains mixed-language or formatting artifacts, and a 279M-parameter XLM-RoBERTa can learn to separate such artifacts from clean LLM output without learning anything about human Arabic writing style. The near-perfect F1 scores in Tables 9-10 and the Zipf/frequency contrasts in Section 5 would then be artifacts of a corrupted gold standard rather than evidence of an 'Arabic AI fingerprint.' The social-media half uses established review corpora (BRAD/HARD) and is less exposed, but the headline academic results depend on ASJP.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether Arabic text generated by large language models carries detectable stylometric signatures. The authors build two corpora: 3,000 human academic abstracts from the Algerian Scientific Journals platform (ASJP), with corresponding LLM-generated abstracts produced from three strategies (title-only, title-plus-content, and polishing) using ALLaM, Jais, Llama 3.1, and GPT-4; and 3,500 human social-media reviews from BRAD and HARD, with LLM-generated 'polished' versions. Stylometric analyses examine word-length statistics, top frequent words, Zipf-like frequency distributions, and BLEU/METEOR/ROUGE-L/BERTScore similarities. Detection experiments fine-tune XLM-RoBERTa for binary cross-model generalization and multi-class model identification. The paper reports near-perfect detection in academic abstracts (up to 99.9% F1) and degraded, highly variable cross-model performance in social media content. The authors claim this is the most comprehensive Arabic machine-generated text detection study to date and release datasets and code publicly.","tokens_in":15959,"tokens_out":3234,"duration_ms":36656,"significance":"If the core results are reliable, the paper is a useful contribution to Arabic NLP and AI-text detection. Its strengths include the breadth of generation strategies and model architectures, the combination of stylometric analysis with detector evaluation, and the public release of datasets and generation notebooks, which supports reproducibility. The finding that formal Arabic academic text is highly detectable while informal social media text resists cross-model generalization is a concrete, falsifiable claim of practical value. However, the significance is conditional on the validity of the human-reference corpus and on the stability of the reported near-perfect scores, both of which need strengthening before the headline claims can be accepted.","major_comments":[{"comment":"The human academic corpus is not adequately validated. The ASJP platform stores Arabic, English, and French abstracts in a single text block, and the authors state that they wrote 'custom segmentation scripts' using statistical and rule-based methods, but they report no error rate, no manual audit, and no inter-annotator agreement for these scripts. The only AI-contamination control is the 2010-2022 publication window; no check for machine-translated or machine-assisted abstracts is described. Because the near-perfect F1 scores in Tables 9 and 10 and the stylometric contrasts in Section 5 are built directly on this human gold standard, corrupted or mis-segmented labels could produce the observed separation without reflecting genuine human-Arabic-writing characteristics. Please add a measured segmentation accuracy on a manually labeled sample, an AI-contamination check (for example, an independent detector or perplexity filter), and rerun the key experiments on the validated subset.","section":"Section 3.1"},{"comment":"The abstract claims that 'Our cross-domain analysis confirms generalization challenges previously reported in the literature,' but no cross-domain experiment appears in Section 6. The academic (Section 6.1) and social media (Section 6.2) experiments are each trained and tested within their own domain; there is no training-on-academic/testing-on-social-media or the reverse. Either add a direct cross-domain evaluation or revise the abstract and the relevant sentence to state that generalization challenges are observed across models within each domain, not across domains.","section":"Abstract and Section 6"},{"comment":"All detection results are reported from a single 75/15/15 split with no error bars, no multiple seeds, and no significance testing. The near-perfect scores in Table 9 (for example, ALLaM-trained detector achieving 99.96% F1 on its own test set) could be split-specific or reflect a particular random initialization. Please report results over at least three to five random splits or seeds with means and standard deviations, and specify whether the reported precision, recall, and F1 refer to the machine-generated class or to a macro average.","section":"Section 6"},{"comment":"The academic detection dataset is imbalanced at roughly 1:3 human-to-machine ratio, yet the binary detection tables report only accuracy, precision, recall, and F1 without a class-wise breakdown or balanced metrics. Accuracy can be misleading under this imbalance, and the reader cannot tell how well the detector recognizes human text. Please report per-class precision/recall/F1, or macro-F1 and balanced accuracy, for the binary experiments.","section":"Section 6.1"}],"minor_comments":[{"comment":"The Arabic word forms in Tables 4 and 7 appear garbled or reversed (for example, 'ةسﺍﺭدلﺍ'), likely a PDF-extraction artifact; please ensure the tables render correct Arabic script in the final version.","section":"Tables 4 and 7"},{"comment":"Reference [2] and reference [31] contain placeholder '????' fields; these should be completed before publication.","section":"References"},{"comment":"In the description of the title-plus-content generation method, the phrase 'We happened to the prompt' should read 'We appended to the prompt.'","section":"Section 4.1.1"},{"comment":"The stylometric analysis is descriptive and relies on visual inspection of plots and tables; adding statistical tests or effect sizes for the frequency and length differences would strengthen the claims.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of Expert Systems with Applications and addresses a timely problem with reproducible artifacts. The main risk is the unvalidated human academic corpus, which is load-bearing for the headline near-perfect detection results. The required fixes (segmentation error measurement, contamination check, multi-split evaluation, direct cross-domain experiments) are feasible within the manuscript's scope and would substantially increase confidence. I do not see evidence of intentional circularity; the detectors are evaluated on held-out splits. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a genuinely useful dataset paper for Arabic machine-generated text detection, but the headline numbers on academic abstracts rest on a human-reference corpus whose quality is not established. The abstract also claims a cross-domain analysis that I could not find in the body.\n\nWhat is new: they built the first Arabic benchmark that combines four LLMs (ALLaM, Jais, Llama, GPT-4), three generation strategies, and two domains, and they released the datasets publicly. The stylometric analysis is careful and the Zipf plots showing LLM overuse of high-frequency words and steeper long-tail drop-off are informative. The detection experiments are competently run: fine-tuning XLM-RoBERTa, reporting precision/recall, and the cross-model generalization results clearly show the expected gaps.\n\nSoft spots, in order of seriousness. First, the ASJP human abstracts. Section 3.1 admits the site dumps Arabic/English/French abstracts in one block and that they used custom segmentation scripts, but no error rate, manual audit, or AI-contamination check is reported beyond the 2010-2022 publication window. If the segmentation or contamination is bad, the 99.9% F1 could be separating formatting artifacts rather than human Arabic prose. That is a real validity threat, not a nitpick. Second, the abstract says \"cross-domain analysis\" but the paper does not train on one domain and test on the other; it treats each domain separately. That is an overclaim. Third, in the social media experiments, human posts average 867 words while generated posts average 225-627 words, so a detector could be exploiting length rather than style. No length-controlled experiment is reported. Fourth, all detection results come from a single train/validation/test split with no error bars or repeated runs, so we have no sense of variance.\n\nThat said, the social media half uses established human corpora (BRAD/HARD), and the qualitative finding that informal text is harder to detect is probably robust. The paper is not a takedown case; it is a resource paper that needs validation work.\n\nWho this is for: people working on Arabic NLP, text detection, or benchmarking. The dataset alone is worth engaging with. I would send this to peer review, but with a requirement that the authors validate the ASJP segmentation, add length-controlled detection, and correct the abstract. As it stands, it is a strong resource with an unverified foundation.","headline":"A useful public dataset and a competent detection study, but the academic human reference is unvalidated and the abstract overclaims cross-domain evidence.","tokens_in":16440,"tokens_out":2689,"would_cite":false,"duration_ms":30993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Arabic text generated by large language models carries measurable stylometric fingerprints that allow near-perfect detection in academic writing, but not in social media.","keywords":["Arabic NLP","machine-generated text detection","stylometric analysis","cross-domain generalization","XLM-RoBERTa","LLM text detection","academic abstracts","social media text"],"falsifier":"Take a set of Arabic abstracts whose human authorship is individually verified (for example, hand-written scans from before the LLM era), match them to generated abstracts by topic and length, and rerun the frequency-distribution contrast and the detector: if the near-perfect academic F1 collapses or the long-tail gap vanishes, the claimed signature is an artifact of corpus differences rather than a stable property of AI text.","tokens_in":15536,"feed_emoji":"🔎","tokens_out":5168,"duration_ms":51001,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models produce measurable stylistic fingerprints in Arabic text, and that these fingerprints are strong enough to support automated detection in formal writing. Using a newly built corpus of Arabic academic abstracts and social media reviews, the authors generate text from four LLMs with three prompting strategies, then compare word-frequency distributions, vocabulary diversity, and semantic similarity against human writing. They find that machine text overuses the most frequent words, drops off more sharply in the low-frequency tail, and is substantially shorter in social media. A fine-tuned XLM-RoBERTa detector separates machine from human academic abstracts with F1 scores up to 99.9 percent, while social media detection is weaker and cross-model transfer is often poor. The upshot is that Arabic AI text is detectable in academic settings, but informal, dialectal, or style-preserving contexts remain a frontier.","feed_headline":"Arabic AI writing carries a detectable fingerprint","feed_subtitle":"Formal Arabic abstracts are easy to flag via word-frequency patterns; social media and cross-model transfer remain hard.","key_machinery":"The load-bearing object is the stylometric signature, operationalized as the top-100 word-frequency distribution and its Zipf-like slope, together with a fine-tuned XLM-RoBERTa classifier that learns to separate human from machine distributions. The frequency analysis is the discovery side: humans show a smooth power-law decay while LLM outputs concentrate probability mass on the highest-ranked words and fall off faster in the tail. The detector is the application side: it reproduces these contrasts in supervised form. Reference-based metrics (BLEU, METEOR, ROUGE-L, BERTScore) serve as a secondary instrument for measuring how close generated text stays to human wording.","core_discovery":"The central claim is that Arabic text produced by LLMs carries consistent, domain-dependent stylometric signatures, despite being human-like in surface quality. The evidence is a systematic comparison showing that all four models overuse high-frequency words, underuse mid-frequency vocabulary, and show a steeper decay in the long tail of the top-100 word-frequency distribution compared to human academic abstracts; in social media, generated posts are 25 to 72 percent shorter than human posts yet preserve semantic meaning. On the basis of these patterns, the authors fine-tune a multilingual BERT-style classifier (XLM-RoBERTa) and report near-perfect detection in academic abstracts, up to 99.9 percent F1 for GPT-4 and above 99 percent for most cross-model pairs trained on Jais or Llama, while social media cross-model F1 can fall to roughly 29 percent. They interpret this as confirming that detection works where the linguistic signature is strong, and that cross-domain and cross-model generalization remain unresolved.","pith_inferences":["If the frequency-distribution signature is genuine, the same analytical lens could be applied to other low-resource languages to test whether the pattern is language-specific or a general property of LLM sampling.","The unverified human-reference corpus means the headline accuracy should be re-measured on a manually confirmed gold standard before being treated as a deployable guarantee.","A direct extension the authors do not pursue is watermarking or decoding-phase constraints: the brittleness they find in informal registers suggests that detector-only defenses will not scale, and generation-side interventions may be needed."],"forward_implications":["Academic Arabic abstracts can be screened automatically for machine authorship with near-perfect accuracy, so institutions can deploy this detection in plagiarism and integrity pipelines.","Detectors lose much of their power in social media: cross-model F1 can drop to 29 percent, so reliable informal-text detection likely needs per-model ensembles rather than a single classifier.","Stylometric signatures differ across models and generation strategies, meaning detection systems must be retrained as new Arabic LLMs appear.","Text refinement (polishing) is the hardest-to-detect generation method, since it stays closest to human wording; content-aware generation sits in between."],"supporting_citations":[{"why":"Supplies the quantitative comparison of human vs. LLM linguistic patterns that the paper extends to Arabic.","marker":"[12]"},{"why":"Provides the Jais Arabic-focused model used as one of the four text generators.","marker":"[13]"},{"why":"Provides the ALLaM Arabic-focused model used as one of the four text generators.","marker":"[15]"},{"why":"Establishes the Arabic detection benchmark and documents prior detectors' struggles with Arabic script.","marker":"[16]"},{"why":"Provides the prior Arabic AI detector using encoder-based transformers that the paper builds on and extends.","marker":"[17]"},{"why":"Supplies the HC3 dataset and its finding of distinctive AI stylometric patterns, a key point of comparison.","marker":"[28]"},{"why":"Demonstrates that detection degrades out-of-domain, the generalization challenge the paper confirms for Arabic.","marker":"[30]"},{"why":"Provides the multilingual XLM-R model that the paper fine-tunes for its detectors.","marker":"[33]"}],"fun_headline_variants":["Arabic AI text carries a word-frequency fingerprint","Formal Arabic AI writing is easily detectable","Arabic LLM text unmasked by stylometry","AI Arabic text: formal easy, social hard to detect","Word-frequency reveals Arabic AI writing in abstracts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison assumes the 2010 to 2022 ASJP abstracts are truly human-written and that the custom segmentation scripts for splitting Arabic, English, and French abstracts made no systematic errors; no contamination check is reported.","fun_headline_variants_meta":{"raw":{"variants":["Arabic AI text carries a word-frequency fingerprint","Formal Arabic AI writing is easily detectable","Arabic LLM text unmasked by stylometry","AI Arabic text: formal easy, social hard to detect","Word-frequency reveals Arabic AI writing in abstracts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2979,"prompt_tokens":1013,"completion_tokens":1966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1896}},"tokens_in":629,"tokens_out":1966,"duration_ms":15145,"temperature":1.0,"reasoning_tokens":1896,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:48:38.933721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of Arabic abstracts whose human authorship is individually verified (for example, hand-written scans from before the LLM era), match them to generated abstracts by topic and length, and rerun the frequency-distribution contrast and the detector: if the near-perfect academic F1 collapses or the long-tail gap vanishes, the claimed signature is an artifact of corpus differences rather than a stable property of AI text.","supporting_citations":[{"cited_title":"Muñoz-Ortiz, C","cited_arxiv_id":null,"evidence_quote":"Supplies the quantitative comparison of human vs. LLM linguistic patterns that the paper extends to Arabic."},{"cited_title":"Alshammari, A","cited_arxiv_id":null,"evidence_quote":"Establishes the Arabic detection benchmark and documents prior detectors' struggles with Arabic script."},{"cited_title":"Alshammari, A","cited_arxiv_id":null,"evidence_quote":"Provides the prior Arabic AI detector using encoder-based transformers that the paper builds on and extends."}],"review_version":1}