{"id":"7071b7df-b2b4-4a0c-a7b1-27029870a73a","arxiv_id":"2608.09539","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 996-tweet Arabic benchmark, Mawqif-XT, provides three held-out targets for evaluating cross-target stance detection and reports transformer and LLM baselines.","lead":"This paper introduces Mawqif-XT, 996 Arabic tweets about three new topics, labeled for stance, sentiment, and sarcasm, as a held-out test set for models trained on the earlier Mawqif data. It also reports baseline scores for Arabic transformers and zero-shot LLMs to measure how well models transfer to related and previously unseen targets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No cross-corpus overlap check is the load-bearing gap: if Mawqif-XT tweets resemble original Mawqif training tweets, reported Favg2 scores may measure memorization rather than cross-target generalization.","rationale":"The reader correctly flags moderate inter-annotator agreement and possible leakage. I agree that the leakage/overlap issue is the most load-bearing, because the benchmark's defining property is that it is held out; if it is not disjoint from training, every downstream comparison is uninterpretable as a test of cross-target generalization. The IAA concern is real but less decisive: moderate kappa (0.564) is disclosed, disagreements concentrate on None-vs-polar, and adjudication with a fourth annotator is standard; moreover all baselines are scored on the same adjudicated labels, so ranking differences are less likely to be purely noise artifacts, though absolute scores are affected. The absence of a release URL is an important practical blocker, but it is a logistics issue that does not by itself invalidate the scientific claim. The near-duplicate hypothesis can be settled by a concrete overlap analysis described above. Since the reader already made the verdict CONDITIONAL and my concern reinforces that condition rather than overturning it, I leave the verdict unchanged.","tokens_in":7781,"tokens_out":3198,"duration_ms":31953,"concrete_test":"Run an exact and near-duplicate overlap analysis between the 996 Mawqif-XT tweets and the original Mawqif training/dev splits (CV, DT, WE). Normalize both sets as in Section 3.1.2 (remove diacritics, tatweel, non-Arabic chars, repeated chars; replace URLs/mentions with tokens), then use exact text matching plus a near-duplicate detector (e.g., MinHash with Jaccard threshold >= 0.8) after removing hashtags. Report the number of test tweets with a match in the training set, broken down by target. If any matches exist, recompute Table 10 Favg2 on the deduplicated subset; if Qwen 2.5's WD score or MARBERT's TR score shifts materially, the held-out generalization claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Mawqif-XT is a held-out test set for cross-target generalization. For that claim, the test tweets must be new instances, not duplicates or near-duplicates of the original Mawqif training/development tweets. Section 2.2 reports removal of duplicates within each collected target ('Duplicate tweets were then eliminated') and filtering by account type and keywords, but no step compares the 996 new tweets against the original Mawqif dataset used in Section 3.1.2 and Table 7. Section 3.1.3 defines the protocols without any leakage or overlap analysis. The risk is concrete: Women Driving is deliberately 'closely related' to Women Empowerment, a training target, and is collected from the same platform over overlapping years (2016-2026 for WD; original Mawqif is also Twitter-based), so exact or lightly edited near-duplicates are plausible. If even a small fraction of test tweets have near-duplicates in training, the reported gains of Qwen 2.5 and MARBERT reflect retrieval of memorized surface forms rather than generalization to a new target. The dataset release URL is also absent, which prevents independent verification, but the leakage issue is the more load-bearing scientific gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Mawqif-XT is a dataset paper introducing a held-out Arabic stance-detection test set of 996 manually annotated tweets across three targets: Women Driving (semantically related to Women Empowerment in the original Mawqif corpus), E-Cars, and Trimester System (both previously unseen targets). The paper preserves the original Mawqif annotation scheme and labels each tweet for stance, sentiment, and sarcasm. It defines two evaluation protocols—related-target generalization (Protocol 1) and unseen-target generalization (Protocol 2)—and reports baselines for seven encoder-based models and three zero-shot LLMs, with Qwen 2.5 (72B) achieving the best overall scores (Favg2 = 73.66 on Women Driving under Protocol 1 and 74.75 overall under Protocol 2). The central claim is that the dataset, together with the original Mawqif train/dev splits, provides a benchmark for evaluating cross-target generalization in Arabic stance detection.","tokens_in":8079,"tokens_out":6155,"duration_ms":57184,"significance":"If the dataset is released with the stated variables and protocols, it would be a useful, carefully scoped Arabic resource for cross-target stance detection, complementing existing datasets by explicitly separating the related-target and unseen-target transfer scenarios. Strengths of the paper include the preservation of a published annotation scheme, explicit reporting of Fleiss kappas and disagreement patterns, clear baseline configurations, and two protocols that operationalize different transfer settings. The main risks are that no cross-corpus overlap analysis is reported, no dataset link or full data dictionary is supplied, and the exact zero-shot prompt and decoding setup is not given; these must be addressed before the benchmark claim is fully supported.","major_comments":[{"comment":"Section 2.2 reports within-target duplicate elimination, and Section 3.1.3 defines the held-out protocols, but no analysis compares the 996 Mawqif-XT test tweets against the original Mawqif training and development tweets used in Table 7. Since Women Driving is deliberately related to Women Empowerment and is collected from the same platform over a 2016–2026 window, exact or lightly edited near-duplicates are plausible. If any such overlaps exist, the Protocol 1 and Protocol 2 scores in Table 10 may partly reflect memorization of training surface forms rather than cross-target generalization. Please report a cross-corpus overlap analysis (e.g., exact tweet-ID matching, normalized-text exact match, and a near-duplicate similarity search), remove any overlapping instances, and quantify the effect.","section":"3.1.3 / 2.2"},{"comment":"Although the paper calls Mawqif-XT a 'released' benchmark and provides usage notes in Section 4, it contains no URL, repository name, DOI, license, or access instructions anywhere in the manuscript. For a dataset paper, the artifact itself is the central claim, so an availability statement is not optional. Add a Data Availability section with a persistent download link, license, and versioning information.","section":"Abstract / Sections 2, 4"},{"comment":"Table 5 lists the released variables as id, tweet_id, target, stance, sentiment, and sarcasm, but does not list the tweet text. If the release contains only tweet IDs, most researchers cannot reconstruct the tweets because of X API restrictions and content deletion, which would make the benchmark unusable in practice. Please clarify in Table 5 (or a data dictionary) whether the tweet text is included, and if it is not, add it to the planned release.","section":"Table 5"},{"comment":"Section 3.1.1 states that the three zero-shot LLMs receive 'the target and tweet as input and [are] instructed to predict one of the three stance labels,' but the exact prompt template (including the Arabic wording), temperature, maximum output length, and number of inference runs are not reported. Zero-shot LLM results are sensitive to these choices, so Table 10 cannot be reproduced from the information given. Provide the full prompts and decoding configuration, or release the inference script.","section":"3.1.1–3.1.2"}],"minor_comments":[{"comment":"The sentence 'The retained tweets were manually following the annotation guidelines' is missing a word; it should read 'manually annotated following the annotation guidelines.'","section":"2.3"},{"comment":"The dataset is called 'Mawqif-v2 Extension' in the abstract but 'Mawqif-XT Extension' throughout the rest of the paper; please standardize the name.","section":"Abstract and passim"},{"comment":"There is a punctuation and capitalization error in 'regardless of the author’s stance towards the target. while sarcasm was annotated'; the period before 'while' should be a comma or semicolon.","section":"2.3"},{"comment":"The word 'usefull' in the final sentence of Section 3.2 is a typo and should be 'useful.'","section":"3.2"},{"comment":"The Women Driving collection states a search period of January 1, 2016 to February 16, 2026, but later describes the initial collection as spanning January 2, 2016 to November 27, 2025; please reconcile the two date ranges.","section":"2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope as a language-resource contribution, and the core idea of separating related-target and unseen-target transfer is sound. The main scientific risk is the missing cross-corpus overlap analysis, which is directly fixable with a supplementary experiment; the missing URL and prompt details are also addressable. I do not see a reason to reject at this stage, but the release and leakage issues should be resolved before the benchmark claim is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Pyramidal take: Mawqif-XT is a legitimate, clearly described Arabic dataset extension for cross-target stance detection, with a sensible design and honest reporting. The main gap is the absence of an overlap analysis against the original Mawqif training set, which matters because one of the new test targets is deliberately related to a training target.\n\nWhat is new: the paper adds 996 manually annotated tweets on three targets, following the existing Mawqif annotation scheme, and explicitly sets up two protocols: a related-target transfer (Women Driving vs. Women Empowerment) and an unseen-target generalization (E-Cars, Trimester System). That design is genuinely useful. The IAA scores are moderate (Fleiss stance kappa 0.564) but the authors dig into the disagreements and show that most involve neutral-vs-polar decisions, which is a familiar and credible pattern. They also report baselines from a range of Arabic and multilingual models, plus zero-shot LLMs. The limitations section is straightforward and doesn't overclaim. The data, while small, is adequate for a benchmark, and the citation pattern is fine — they reference the prior Arabic stance datasets and the original Mawqif without red flags.\n\nWhere it's soft:\n\n- The bigger issue: there's no check for duplicates or near-duplicates between the new test tweets and the original Mawqif training tweets. Section 2.2 talks about removing duplicates within each collection, but not across corpora. Since Women Driving was collected from the same platform over a period that overlaps with the original Mawqif data and is semantically close to Women Empowerment, the risk is concrete. If some test tweets have near-copies in training, the reported Favg2 numbers could partially reflect memorization rather than generalization. This is fixable: run a deduplication check (exact and near-duplicate) against the original corpus and report the results, or at least explain why it's not needed.\n\n- The paper says the dataset is released but gives no URL or access instructions. That's a practical problem for a resource paper. It also prevents an independent check of the leakage issue.\n\n- Minor: the manual review step for Women Driving is described as a random sample of 500 tweets, whereas the other two targets went through full manual review. That discrepancy is confusing and should be clarified.\n\n- The baseline numbers would be more convincing with seed-level variance, but that's a minor point for a dataset paper.\n\nNet assessment: The core resource claim is plausible and the paper is honest with its limitations. The leakage gap is load-bearing enough that it should be addressed before the benchmark is widely adopted, but it's not a fatal flaw; a proper overlap analysis would likely settle it. This paper deserves a serious referee and a request for revision rather than a desk reject. It will be useful to Arabic NLP researchers and the stance detection community. I'd probably not bring it to a general reading group, but if you work on Arabic or cross-target stance, it's worth a look.\n\nRecommendation: send to peer review with the expectation that the authors add the dataset link, run and report a cross-corpus overlap analysis, and clarify the manual review procedure.","headline":"Mawqif-XT is a useful, honestly documented Arabic stance dataset extension, but the missing cross-corpus overlap check against the original Mawqif training set is a load-bearing gap that needs to be addressed before the benchmark is adopted.","tokens_in":8535,"tokens_out":3531,"would_cite":true,"duration_ms":32053,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"996 Arabic tweets form a held-out benchmark for cross-target stance detection.","keywords":["Arabic dataset","Stance detection","Cross-target generalization","Target-specific stance detection","Multi-task learning","Sentiment analysis","Sarcasm detection","Arabic tweets"],"falsifier":"Recompute the baseline $F_{\\text{avg}2}$ scores after removing all tweets whose stance label lacked full three-way annotator agreement; if model rankings change substantially, the benchmark's moderate agreement is driving the results rather than cross-target ability. Also run a duplicate and near-duplicate check between the original Mawqif training tweets and the new test tweets; finding near-duplicates would mean the held-out scores are inflated by leakage.","tokens_in":7564,"feed_emoji":"📊","tokens_out":6519,"duration_ms":56913,"temperature":0.7,"pith_summary":"This paper introduces Mawqif-XT, a held-out Arabic benchmark of 996 manually annotated tweets for stance, sentiment, and sarcasm across three targets: Women Driving, E-Cars, and Trimester System. The dataset is designed to be a test set only, with the original Mawqif dataset reserved for training and development, so that cross-target generalization can be measured cleanly. Two evaluation protocols are defined: one that tests transfer to a target semantically related to the training data, and one that tests generalization to unseen domains. The paper also reports baselines from seven transformer models and three zero-shot LLMs, with Qwen 2.5 (72B) obtaining the highest overall scores under both protocols. A sympathetic reader would care because Arabic stance resources with a controlled train/test separation are scarce, and this benchmark gives the field a common yardstick for comparing related-target transfer and unseen-target generalization.","feed_headline":"996 Arabic tweets benchmark cross-target stance detection","feed_subtitle":"A held-out test set and baselines let models be compared on related versus unseen Arabic targets.","key_machinery":"The load-bearing object is the Mawqif-XT Extension itself: a test-only dataset of 996 tweets built on the original Mawqif annotation scheme, which labels each tweet for stance (Favor, Against, None), sentiment (Positive, Negative, Neutral), and sarcasm (Sarcastic, Non-sarcastic). The evaluation design rests on two protocols: Protocol 1 trains on all original Mawqif targets and tests on Women Driving, a related target; Protocol 2 trains on COVID-19 Vaccine and Digital Transformation, validates on Women Empowerment, and tests on E-Cars and Trimester System, two unseen domains. The separation of training, development, and test targets is what makes measured performance interpretable as cross-target generalization rather than in-domain memorization.","core_discovery":"The central claim is that Mawqif-XT provides a valid held-out evaluation set for cross-target stance detection in Arabic. Each of its 996 tweets carries stance, sentiment, and sarcasm labels following the original Mawqif scheme; three annotators label every tweet and a fourth resolves disagreements. The benchmark's design separates the original Mawqif targets (COVID-19 Vaccine, Digital Transformation, Women Empowerment) as training and development data from three new test targets, with Women Driving chosen to be semantically related to Women Empowerment and E-Cars and Trimester System chosen as unseen domains. Baseline experiments show that no model wins on every target: Qwen 2.5 (72B) achieves the best overall scores ($F_{\\text{avg}2} = 73.66$ on Women Driving under Protocol 1 and $74.75$ overall under Protocol 2), while MARBERT is the strongest encoder on the unseen-target protocol. The paper's claim is that, together with the original Mawqif dataset, this extension constitutes a standardized benchmark for studying cross-target generalization in Arabic stance detection.","pith_inferences":["Since most stance disagreements involve the None label and the reported $F_{\\text{avg}2}$ metric ignores None, the metric may partly immunize the baselines against the noisiest annotation decisions; reporting $F_{\\text{avg}2}$ on only full-agreement tweets would test this directly.","A natural safeguard for users is to run a duplicate or near-duplicate check between the original Mawqif training tweets and Mawqif-XT before treating the scores as generalization results; the paper does not report such an overlap analysis.","The observed disagreement pattern suggests that a third, weaker label like 'leaning favor' might reduce noise, but that would require changing the original annotation scheme.","One could extend the same protocol to test whether models trained jointly on stance, sentiment, and sarcasm beat single-task models on cross-target transfer."],"forward_implications":["Future Arabic stance systems can report comparable $F_{\\text{avg}2}$ scores on a fixed held-out test set instead of constructing their own splits.","The two protocols allow researchers to separate the difficulty of transferring to a semantically related target from the difficulty of generalizing to an unseen domain.","Zero-shot LLMs, particularly Qwen 2.5, are competitive on Arabic stance, while encoder models such as MARBERT remain strong on specific unseen targets.","Because every tweet also has sentiment and sarcasm labels, the benchmark supports multi-task studies of how these labels interact with stance."],"supporting_citations":[{"why":"Defines the task of detecting stance toward a specified target, which the benchmark operationalizes.","marker":"[1]"},{"why":"Supplies the original Arabic stance, sentiment, and sarcasm annotation scheme and the training/development splits reused in both evaluation protocols.","marker":"[2]"},{"why":"Provides the threshold used to describe the Fleiss kappa agreement values as moderate.","marker":"[9]"},{"why":"Serves as an Arabic pretrained encoder baseline under both protocols.","marker":"[10]"},{"why":"Serves as an Arabic dialectal encoder baseline under both protocols.","marker":"[11]"},{"why":"Serves as an Arabic encoder baseline; it achieves the best overall encoder score under Protocol 2.","marker":"[12]"},{"why":"Serves as a zero-shot LLM baseline.","marker":"[16]"},{"why":"Serves as a zero-shot LLM baseline; it attains the best overall scores in both protocols.","marker":"[17]"},{"why":"Serves as an Arabic-centric zero-shot LLM baseline.","marker":"[18]"}],"fun_headline_variants":["Mawqif-XT: 996 tweets benchmark cross-target stance","Arabic stance detection: new 996-tweet cross-target test","Cross-target Arabic stance benchmark: 996 tweets","New Arabic dataset challenges cross-target stance models","Mawqif-XT: evaluating Arabic stance on unseen targets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity depends on the manually assigned labels being accurate enough that score differences reflect real cross-target generalization, even though annotators reached full three-way agreement on only 62.95% of stance labels.","fun_headline_variants_meta":{"raw":{"variants":["Mawqif-XT: 996 tweets benchmark cross-target stance","Arabic stance detection: new 996-tweet cross-target test","Cross-target Arabic stance benchmark: 996 tweets","New Arabic dataset challenges cross-target stance models","Mawqif-XT: evaluating Arabic stance on unseen targets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3139,"prompt_tokens":909,"completion_tokens":2230,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2148}},"tokens_in":525,"tokens_out":2230,"duration_ms":13076,"temperature":1.0,"reasoning_tokens":2148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:13:48.644194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the baseline $F_{\\text{avg}2}$ scores after removing all tweets whose stance label lacked full three-way annotator agreement; if model rankings change substantially, the benchmark's moderate agreement is driving the results rather than cross-target ability. Also run a duplicate and near-duplicate check between the original Mawqif training tweets and the new test tweets; finding near-duplicates would mean the held-out scores are inflated by leakage.","supporting_citations":[{"cited_title":"Mohammad, S","cited_arxiv_id":null,"evidence_quote":"Defines the task of detecting stance toward a specified target, which the benchmark operationalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the original Arabic stance, sentiment, and sarcasm annotation scheme and the training/development splits reused in both evaluation protocols."},{"cited_title":"Team, Qwen2.5: A party of foundation models (September 2024)","cited_arxiv_id":null,"evidence_quote":"Serves as a zero-shot LLM baseline; it attains the best overall scores in both protocols."}],"review_version":2}