{"id":"da898a39-c780-4cb5-aa19-63d3b0adf340","arxiv_id":"2505.06184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM pipeline that filters millions of Persian political tweets and generates abstractive and extractive profiles per stance claim outperforms existing profiling baselines on a human-validated stance QA task.","lead":"This paper uses large language models to turn millions of Persian political tweets into short, readable user profiles, built around 15 stance claims. It reports that these profiles beat prior LLM and retrieval baselines on a stance-detection test, and introduces a new Persian political tweet dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Extractive profile size may not be standardized: Section 4.2 allows multiple supporting tweets per claim, while baselines are capped at one tweet per statement, so the reported F1 gap could be a context-volume effect.","rationale":"The reader's weakest assumption focused on LLM-evaluator bias and the narrow 15-claim sample, while the rationale also mentioned 'unequal profile-size standardization.' I consider the size-standardization issue the single most load-bearing concern because it threatens the internal validity of the headline comparison, not just its generalizability. The paper's strongest evidence is the human-validated annotation set, the use of McNemar tests, and the detailed pipeline description, but the one paragraph intended to establish fairness is ambiguous at the critical point: Section 4.2 permits a set of supporting tweets per claim, and Section 5 only explicitly caps 'other selection methods' at one tweet per statement. If the proposed extractive method returns multiple tweets per claim, it has an unfair advantage in the open-book QA evaluation through additional context, making the reported 0.6668 and 9.8% figures uninterpretable as evidence of better profiling. This is a concrete, checkable concern rather than a conceptual disagreement, and it is fixable by reporting and matching profile sizes. Since the concern does not by itself show the method is inferior, only that the current comparison is not controlled, the reader's CONDITIONAL verdict remains appropriate; no change in disposition is needed.","tokens_in":13611,"tokens_out":9007,"duration_ms":90958,"concrete_test":"Check the actual size of the extractive profile in the released code, prompts, or logs: count the number of tweets in Eu,i per user and per statement. If the average is greater than 1 tweet per statement (i.e., more than 15 tweets per user), rerun the comparison with the extractive profile truncated to exactly one tweet per statement (e.g., the highest-cited tweet per claim), keeping all baselines at 15 tweets. If the macro-F1 advantage over BM25, SemAE, and Random Selection narrows or disappears, the reported superiority is a profile-size effect rather than a profiling-quality effect. Also report the total token count per profile for every method to verify that the standardization was actually enforced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on a fair comparison, but Section 4.2 defines the extractive profile as a set of supporting tweets Eu,i for each of the 15 stance claims, with no stated cardinality cap. Section 5's 'Standardized Profile Size' paragraph says only that 'we limited other selection methods to retrieve the top-1 result for each statement, resulting in 15 tweets per user.' It never states that the proposed extractive method is limited to one tweet per statement. If each Eu,i contains multiple cited tweets, the extractive profile provides the GPT-4o evaluator with substantially more context than the 15-tweet baselines, so the 0.6668 macro F1 and the 9.8% headline gain could reflect information volume rather than profile quality. The comparison is therefore not clearly controlled on the variable the paper claims to standardize. This is not an accusation of intent; it is a missing detail in the exact place where fairness must be demonstrated. If the extractive profile is indeed larger, the McNemar significance tests in Table 3 are comparing unequal conditions, and the central claim of superior profiling is underdetermined until the profile sizes are matched.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage LLM-based user profiling pipeline for Persian political Twitter. In the first stage, a semi-supervised filtering method uses a Wikidata- and Wikipedia-derived Persian political knowledge base with BGE-M3 embeddings to label and filter political tweets, yielding the PersianPol6M dataset of over 1.7 million political tweets. In the second stage, the method generates 15 domain-defining stance claims from a first split of users, then profiles users in a second split by producing, for each claim, an abstractive factual summary and an extractive set of supporting tweets. Profile quality is evaluated by an open-book stance-detection QA task in which an LLM reads the profile and answers the 15 claims; answers are compared against human-annotated ground truth for 100 users. The authors report that the extractive profile achieves macro F1 0.6668 with GPT-4o as evaluator, outperforming baselines including random selection, BM25, semantic retrieval, Amazon summarization/RAG, and SemAE, with statistical significance in most comparisons, and the abstract states a headline improvement of 9.8%.","tokens_in":13864,"tokens_out":3200,"duration_ms":32379,"significance":"If the central comparison is fair, the paper makes a useful contribution: it introduces a new Persian political Twitter dataset, a semi-supervised filtering pipeline that reduces annotation cost, and a natural-language profile representation that is interpretable and usable by LLMs. The evaluation design has genuine strengths: the ground-truth labels are produced by human annotators from pooled tweet histories and are thus independent of profile generation; the authors provide detailed appendices with prompts, annotation guidelines, and implementation parameters; and the use of two LLM evaluators (GPT-4o and Gemini Flash 1.5) is a reasonable attempt at robustness. The strongest claim that extractive, citation-grounded tweet selection preserves stance-relevant information better than abstractive summarization or retrieval baselines is plausible and worth investigating. However, the paper's central quantitative claims are weakened by an uncontrolled profile-size comparison, an unreported derivation of the 9.8% headline number, and an evaluation setup in which the 15 claims are simultaneously the profile-generation dimensions and the evaluation queries.","major_comments":[{"comment":"The comparison is not demonstrably controlled on profile size. Section 4.2 defines the extractive profile as a set of supporting tweets Eu,i for each stance claim si, with no stated cardinality cap on Eu,i. Section 5 states only that 'we limited other selection methods to retrieve the top-1 result for each statement, resulting in 15 tweets per user,' which explicitly caps the baselines but not the proposed extractive profile. If each Eu,i contains more than one tweet, the extractive profile gives the LLM evaluator substantially more context than the 15-tweet baselines, and the reported macro F1 gap could reflect information volume rather than profile quality. The abstractive profile, which generates 'a few sentences for each of the 15 statements,' is likewise not matched in length to the baselines. Please report the mean and maximum number of tweets per statement in the extractive profiles, and add an ablation in which the extractive and abstractive profiles are truncated or padded to match the 15-tweet baseline size.","section":"§4.2 and §5, 'Standardized Profile Size'"},{"comment":"The headline '9.8%' improvement is not derived anywhere in the text. Section 6.2 reports a macro F1 of 0.6668 for the extractive method with GPT-4o as evaluator and states that this 'significantly outperforms all other methods,' but no table or equation gives the baseline F1 values or the pairwise differences that produce 9.8%. Because the profile-size issue in the previous comment affects any quantitative gap, the paper should report the full F1 table for both evaluators and explicitly state which comparison yields the 9.8% figure.","section":"Abstract and §6.2"},{"comment":"The evaluation is partially circular with respect to the claim of general profile quality. The same 15 stance claims serve both as the dimensions along which profiles are generated (Section 4.2) and as the queries in the open-book QA evaluation (Section 4.3), so the profiles are explicitly constructed to encode the information being tested. A high score on this task shows that the profiling method preserves the target information, but it does not establish that the profiles are generally superior for other stances or downstream tasks. Moreover, the GPT-4o evaluator belongs to the same model family as the GPT-4o-mini profile generator; the observed gap may partly reflect format or family compatibility. Please add an evaluation on held-out stance claims that were not used in profile construction, and report results with an evaluator from a different family or with a non-LLM classifier to test for evaluator bias.","section":"§4.3, Figure 7, and §4.2"},{"comment":"The statistical significance reporting conflates significance with effect size and does not account for multiple comparisons. Describing the GPT-4o extractive result as '100% statistical significance' because all McNemar p-values are below 0.05 is misleading: with 14 comparisons, the family-wise error rate is not controlled, and several p-values (e.g., 0.0002, 0.0003) are close to the threshold after repeated testing. Please report corrected p-values (e.g., Benjamini-Hochberg) or state that the comparisons are exploratory, and also report effect sizes or confidence intervals for the F1 differences.","section":"§4.3, Appendix A.3, Table 3"}],"minor_comments":[{"comment":"The caption says 'Percentage changes in the total tweets, users, retweets, likes, and tweet lengths after filtering,' but the table rows show Total Twts, Total Users, Avg Twt/User, Avg Twt Len, and Median Twt Len; retweets and likes are not actually listed. The caption and table content should be aligned.","section":"Table 1"},{"comment":"There is a typographical error in the citation '(Kanoje, Girase, and Mukhopadhyay 2015))' with an extra closing parenthesis; also the sentence beginning 'Ongoing research focuses...' is a fragment that should be merged with the preceding sentence.","section":"Section 2.2"},{"comment":"The random-selection baseline is described as having two variations, one operating on the pooled tweets and one on the complete user tweet history, but the 'complete user history' variation is not clearly defined in terms of how many tweets are used and whether it is subject to the 15-tweet standardization. Please clarify.","section":"Section 5, Baseline descriptions"},{"comment":"The caption states that results are for 'Gemini Flash 1.5 and GPT 4o mini,' but Section 6.2 refers to 'GPT-4o as evaluator'; the naming should be made consistent (GPT-4o vs. GPT-4o-mini) to avoid confusion about which model produced the reported 0.6668 F1.","section":"Figure 8"},{"comment":"The table is only referenced indirectly in the main text; please add an explicit pointer in Section 6.2 so readers can find the McNemar results without hunting in the appendix.","section":"Appendix A.3, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the dataset is a useful resource for the Persian NLP/social-media community. The main risk is that the comparative evaluation is not yet fair enough to support the headline claims; the profile-size standardization gap is the most serious issue and should be probed by an ablation. I would also encourage the authors to make the 9.8% calculation explicit and to temper the '100% statistical significance' language. These are fixable with additional experiments and reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful system paper with an evaluation that is more careful than most in the social-media profiling space, but the headline performance claim is not yet air-tight. The 9.8% advantage over baselines could be real, or it could be a side-effect of unequal context size and an evaluation task that mirrors profile construction.\n\nThe genuinely new pieces: a semi-supervised domain filtering pipeline that uses a knowledge base and a BERT classifier to reduce millions of tweets to a manageable pool without large labeled sets, and a profiling method that uses LLM-generated domain-defining statements (political stance claims, here) to produce both abstractive summaries and extractive tweet selections. The evaluation framework, with human-annotated ground truth, Cohen's kappa, and a clear QA prompt, is a step beyond the typical intrinsic BLEU/ROUGE comparisons. The Persian political dataset, while gated, is a contribution.\n\nThe soft spots, in order of how much they bother me:\n\n1. The standardized-profile-size claim is incompletely specified. Section 4.2 defines the extractive profile as a set of supporting tweets per claim with no stated cap. Section 5 says baselines were limited to top-1 per claim, yielding 15 tweets. Nothing says the proposed extractive method was similarly capped. If the extractive profile packs in multiple tweets per claim, the extra context alone could explain part of the F1 gap. This is the kind of detail that has to be nailed down before I trust the 9.8%.\n\n2. The 9.8% headline number does not appear in the results section. I can infer it from the bar chart, but there's no traceable arithmetic. That's an easy fix but it matters.\n\n3. The evaluation has a mild circularity: the 15 stance claims are used to build the profiles and also as the QA queries. So the profiles are explicitly tailored to encode the very dimensions being probed. Baselines like random or BM25 don't have that advantage. The authors acknowledge this partially, but it's real.\n\n4. The LLM evaluator is GPT-4o, and the profiles are generated with GPT-4o-mini. Same-family bias is possible, though the authors do include Gemini as a second evaluator, which helps.\n\nThe good news: the human annotation work is real, the limitations section is honest about abstractive bias, and the authors don't oversell to the point of absurdity. I'd send this to review, but with the expectation that the revision must show equalized profile sizes and a full-history oracle baseline. As it stands, the paper's contribution is more the pipeline and dataset than the specific F1 gap.","headline":"A solid, well-engineered profiling pipeline whose headline 9.8% claim depends on an incompletely specified size equalization and a mildly circular evaluation; deserves review but needs revision.","tokens_in":14369,"tokens_out":3466,"would_cite":false,"duration_ms":32323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage LLM pipeline turns millions of tweets into compact profiles that preserve political stance information better than existing user-profiling methods, reporting a 9.8% improvement over baselines.","keywords":["user profiling","large language models","stance detection","semi-supervised filtering","knowledge base","extractive summarization","abstractive summarization","Persian political Twitter"],"falsifier":"Run the same 1,500 user-statement pairs with an evaluator from a different model family—for example an open-weight LLM or a fine-tuned Farsi stance classifier—and check whether the extractive profile's lead over random and BM25 selection persists. If the gap narrows or reverses, the claim that the profiles preserve stance information better than the baselines is not robust to evaluator choice. A second decisive check is to build a fresh set of stance claims by independent experts and see whether the F1 gap survives on claims that were never used to construct the profiles.","tokens_in":13415,"feed_emoji":"🗳️","tokens_out":10145,"duration_ms":82257,"temperature":0.7,"pith_summary":"The paper sets out to establish that a user's political stance on Twitter (X) can be distilled into a short, human-readable profile without losing the information that matters. The proposed method first filters a user's timeline with a semi-supervised knowledge-base similarity step, then uses an LLM to generate, for each of 15 expert-selected stance claims, a short factual summary and a set of cited original tweets. The authors argue these profiles are flexible across domains, need no large labeled datasets, and outperform existing summarization, retrieval, and latent-embedding baselines on an open-book stance-detection evaluation. The headline result is a 9.8% improvement over baselines, with the extractive profile variant reaching macro F1 of 0.6668 under a GPT-4o judge.","feed_headline":"9.8% better: LLM profiles beat baseline user-profiling methods","feed_subtitle":"Two-stage pipeline distills millions of tweets into profiles that preserve stances for downstream tasks.","key_machinery":"The central object is the set of 'domain-defining statements'—15 expert-curated stance claims generated by zero-shot LLM prompting from a held-out split of the data. These statements serve triple duty: they are the aspects around which the profile is organized, the queries for the open-book QA evaluation, and the anchors for human annotation of ground truth. The complementary machinery is the two-stage pipeline: a semi-supervised filter that scores tweets against a Wikidata-derived Persian political knowledge base using BGE-M3 embeddings and a fine-tuned TookaBERT classifier, and the profile generator that produces cited abstractive summaries and extractive tweet selections per statement. The evaluation compares stance-detection accuracy when context is the full pooled tweet history versus the compact profile, treating the gap as a measure of information loss.","core_discovery":"The paper's central claim is that profiling a user by anchoring on domain-defining statements—here, 15 stance claims about Persian politics—and generating both abstractive summaries and extractive tweet selections for each claim preserves stance-relevant information better than the alternatives tested. On 1,500 human-annotated user-statement pairs, the extractive profiles reach a macro F1 of 0.6668 with GPT-4o as evaluator, significantly ahead of random selection, BM25, semantic retrieval, SemAE, and the Amazon LLM-personalization baselines; the abstractive variant also leads under the Gemini Flash evaluator. The authors attribute the extractive advantage to the fact that it keeps the user's original wording, which avoids cultural and political biases that appear when the model paraphrases in its own words.","pith_inferences":["Beyond the paper: if the evaluator-bias hypothesis is correct, the extractive lead should shrink when the judge is an open-weight LLM or a fine-tuned Farsi stance classifier; that is a cheap, direct test of the robustness of the reported gap.","Beyond the paper: the 15 claims were selected by experts and used for both generation and evaluation; a second set of claims from independent annotators would reveal whether the method's advantage is specific to the chosen claims or generalizes to other stances.","Beyond the paper: because extractive profiles keep the original tweet text with citations, they could double as auditable evidence for downstream tasks such as misinformation detection, where traceability is a requirement.","Beyond the paper: the semi-supervised filter's 93% precision on positive examples suggests the same knowledge-base-similarity labeling could bootstrap classifiers for other high-signal domains without manual annotation."],"forward_implications":["A compact profile built from 15 stance-anchored statements can stand in for a user's full tweet history in stance-based analyses, which makes large-scale social-network reasoning computationally feasible.","The same pipeline transfers to a new domain by rebuilding the knowledge base and regenerating defining statements; the paper demonstrates the knowledge-base construction using Wikidata for European football.","Extractive profiles are the safer default when the target domain is politically or culturally specific, because abstractive summaries inherit the LLM's Western-centric biases.","The PersianPol6M dataset and the human-validated LLM evaluation framework give later work a benchmark for comparing profiling methods on stance preservation."],"supporting_citations":[{"why":"Supplies BGE-M3 embeddings used to compute tweet–knowledge-base similarity in the filtering stage.","marker":"(Chen et al. 2024)"},{"why":"Provides TookaBERT-Base, the Persian encoder fine-tuned into the political-content classifier.","marker":"(SadraeiJavaheri et al. 2024)"},{"why":"Source of the Wikidata entities and graph traversal that forms the Persian political knowledge base.","marker":"(Vrandečić and Krötzsch 2014)"},{"why":"Louvain community detection produces the representative user sample from the filtered tweet graph.","marker":"(Blondel et al. 2008)"},{"why":"Prior Persian stance-detection work that frames the difficulty of the task and the inter-annotator agreement level.","marker":"(Zarharan et al. 2025)"},{"why":"The Amazon LLM personalization method is the main abstractive baseline the proposed method must beat.","marker":"(Richardson et al. 2023)"},{"why":"SemAE is the extractive aspect-summarization baseline using sparse coding and mean embeddings.","marker":"(Basu Roy Chowdhury, Zhao, and Chaturvedi 2022)"},{"why":"Defines BM25, the sparse-retrieval baseline that outperformed dense retrieval in the stance-claim matching.","marker":"(Amati 2009)"},{"why":"Documents the lost-in-the-middle long-context limitation that motivates the compact-profile design and the extractive advantage.","marker":"(Liu et al. 2024)"}],"fun_headline_variants":["LLM profiling via stance claims beats baselines by 9.8%","Extractive beats abstractive in LLM-based user profiling","Domain-defining statements unlock better LLM user profiles","Anchoring on stance claims boosts LLM profiling accuracy","Persian political tweets: extractive LLM profiles win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on the assumption that scoring a profile against 15 expert-selected stance claims with an LLM judge is a fair and unbiased test of profile quality; if the judge simply prefers original tweet text or outputs from its own model family, the reported lead could be an artifact of the evaluation rather than a property of the profiles.","fun_headline_variants_meta":{"raw":{"variants":["LLM profiling via stance claims beats baselines by 9.8%","Extractive beats abstractive in LLM-based user profiling","Domain-defining statements unlock better LLM user profiles","Anchoring on stance claims boosts LLM profiling accuracy","Persian political tweets: extractive LLM profiles win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1526,"prompt_tokens":958,"completion_tokens":568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":574,"tokens_out":568,"duration_ms":5646,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:46.039558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,500 user-statement pairs with an evaluator from a different model family—for example an open-weight LLM or a fine-tuned Farsi stance classifier—and check whether the extractive profile's lead over random and BM25 selection persists. If the gap narrows or reverses, the claim that the profiles preserve stance information better than the baselines is not robust to evaluator choice. A second decisive check is to build a fresh set of stance claims by independent experts and see whether the F1 gap survives on claims that were never used to construct the profiles.","supporting_citations":[{"cited_title":"T.; and Foster, J","cited_arxiv_id":null,"evidence_quote":"Prior Persian stance-detection work that frames the difficulty of the task and the inter-annotator agreement level."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SemAE is the extractive aspect-summarization baseline using sparse coding and mean embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BM25, the sparse-retrieval baseline that outperformed dense retrieval in the stance-claim matching."}],"review_version":1}