{"id":"b183a0cd-0a2a-43b0-97b4-b5892cd5fc23","arxiv_id":"2411.12196","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Group polarization is measured through a Community Sentiment Network built by a team of LLM agents, with a Community Opposition Index score; only the stance detection step is empirically tested.","lead":"The authors describe a system where a team of AI agents reads social media comments, identifies opposing groups, and builds a sentiment graph to score how polarized a discussion is. The AI team performed well on stance detection tests, but the new polarization score itself was never directly tested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"COI metric is underspecified: internal cohesion t_i is never defined (Eqs. 2-3), and Algorithm 2's stochastic imputation makes the index non-reproducible; stance detection results do not validate it.","rationale":"The reader's weakest_assumption identifies the undefined t_i and the stochastic imputation in Algorithm 2 as the fragile premises. I find these are the decisive internal gaps. The stance detection results are plausible but peripheral: they validate parts of the pipeline, not the polarization index itself. Since COI cannot be computed or reproduced, the paper's central claim is unsupported even if every stance detection number is correct. The absence of a polarization benchmark does not excuse an undefined metric; a metric must be defined before it can be validated. Therefore the REJECT verdict is appropriate, and I would not soften it. The paper could be repaired by supplying an explicit definition of t_i, making the imputation deterministic or averaging over repeated samples, and testing COI on synthetic or annotated polarization data. As written, the contribution is a sketch rather than a measurable approach.","tokens_in":11364,"tokens_out":2977,"duration_ms":30995,"concrete_test":"Use Section 3.3 alone as the specification to implement COI for a toy three-subgroup CSN; record where t_i cannot be computed. Then run Algorithm 2 twice on identical incomplete-triplet data with different random seeds and compare the resulting COI values; any difference shows the metric is non-deterministic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the proposed approach provides a usable, accurate, interpretable measure of group polarization. That claim depends entirely on the Community Opposition Index, which is introduced in Section 3.3. The index is defined as COI = sum_i (n_i/N) * t_i * sum_j (-e_ij) * 1_{e_ij <= 0}, but t_i is merely said to be 'the internal cohesion of subgroup i' with no equation, algorithm, or operationalization anywhere in the manuscript. Equations (1)-(3) display t_i symbolically, but the reader cannot compute it from any described data. Consequently COI is not a well-defined function of the Community Sentiment Network; the metric that is the paper's third contribution does not exist in a testable form. The empirical section tests only zero-shot stance detection of the agent pipeline, which measures classification against gold labels on SEM16, P-Stance, and VAST; it never measures COI or compares COI values to any external criterion of polarization. The paper itself acknowledges the absence of a polarization benchmark (Section 4), but substituting stance detection does not validate the metric. A second internal gap compounds this: Algorithm 2 hardcodes 10x10 matrices and imputes missing personal stances by sampling from the observed count frequencies (lines 19-32), with no random seed or repeated averaging, so two runs on identical data can yield different edge weights and thus different COI values. The metric is therefore not only undefined but also irreproducible. These are omissions inside the manuscript, not disagreements with scholarly consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a group polarization measurement approach built on a multi-agent LLM system. It introduces a Community Sentiment Network (CSN) to represent subgroups and inter-subgroup sentiments, and a Community Opposition Index (COI) as a scalar metric. The system's components are evaluated only through zero-shot stance detection on SEM16, P-Stance, and VAST, where it reports strong results. The central claim is that this approach provides a usable, accurate, and interpretable polarization measurement.","tokens_in":11691,"tokens_out":5829,"duration_ms":57108,"significance":"If the COI were well-defined and validated, the paper would offer a practical pipeline for large-scale social media polarization analysis, and the strong stance detection results suggest the extraction component is competitive. However, the current manuscript's significance is severely limited: the COI is not computable as written because internal cohesion t_i is undefined, and no experiment tests the metric against any external polarization criterion. The conceptual idea of combining internal cohesion and inter-group hostility in a graph index is plausible, but the paper does not yet provide evidence that the index behaves as claimed.","major_comments":[{"comment":"The Community Opposition Index is not a well-defined function of the described inputs because the internal cohesion term t_i is never defined or operationalized. The text states only that t_i represents 'the internal cohesion of the subgroup i' (Eq. 2), and no equation, algorithm, or data source for t_i appears anywhere in the manuscript. Since the final COI in Eq. (3) is a weighted sum over t_i, readers cannot compute the index from the CSN or any described dataset. This is load-bearing: the abstract and introduction claim that the approach provides accurate and interpretable polarization measurement, and COI is the metric that operationalizes that claim.","section":"Section 3.3, Eqs. (1)-(3)"},{"comment":"The empirical evaluation does not validate the polarization measurement claim. The experiments test zero-shot stance detection on SEM16, P-Stance, and VAST, which evaluate per-comment classification against gold labels. No experiment computes COI, compares COI to any ground-truth polarization measure, or tests whether COI distinguishes polarized from non-polarized discussions. The paper's own statement that 'there is no established benchmark' (Section 4) explains the choice but does not justify it; stance detection accuracy is a proxy for the extraction component, not evidence about the validity of the aggregation metric.","section":"Section 4"},{"comment":"The CSN construction algorithm is non-reproducible. Incomplete triplets are completed by sampling a subgroup from the observed stance frequencies (lines 23-26), but the sampling has no random seed and no repeated averaging is performed. Two runs on identical input data can therefore produce different adjMatrix values and different COI outputs. A measurement method should be deterministic; as written, the method cannot yield a stable, comparable index. Additionally, the hardcoded 10x10 matrix dimensions arbitrarily assume at most 10 subgroups, which is not justified.","section":"Algorithm 2, lines 19-32"},{"comment":"The comparison with baselines is confounded by the choice of base model. The authors state that GPT-3.5 Turbo is used for the agents in the Background Mining and Semantic Analysis stages, while GPT-4 is employed as the Polarization Assessor (Section 4.2). Several baselines, such as GPT-3.5 and GPT-3.5+COT, use GPT-3.5 alone. The reported improvements over these baselines may therefore reflect the stronger base model rather than the multi-agent architecture. To support the architecture claim, the authors should compare systems using the same underlying LLM, or ablate the agent composition with a fixed base model.","section":"Section 4.2 and Table 3"}],"minor_comments":[{"comment":"The manuscript contains many typographical and formatting errors, including 'F or' in the Section 3.2 heading, 'Background' in Figure 2, 'V ariable' in Table 1, '63,4' in Table 3, and concatenated words in the introduction (e.g., 'Withthedevelopmentofinternettechnology'). A thorough proofreading pass is needed.","section":"Throughout"},{"comment":"The threshold for uncertainComments and the HumanExpertHelp procedure are not specified; a reader cannot determine when human intervention occurs or how reproducible the results are.","section":"Section 3.2"},{"comment":"The variable sg is populated in the first loop but is not clearly connected to the initialization of the agents; the role of SocialMediaVeteran(sg, bg) could be clarified.","section":"Algorithm 1"},{"comment":"The relationship between the 'directed cyclic graph' description and the matrix construction in Algorithm 2 is not explicit, particularly regarding how self-loops are formed.","section":"Section 3.1"},{"comment":"The 'P-Stance' column header is run together ('TrumpBiden Sanders') and should be separated for readability.","section":"Table 3"},{"comment":"No code, prompts, or detailed agent instructions are provided, making it impossible to reproduce the stance detection results or the CSN construction.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The paper's central contribution is a measurement index, but the index is undefined and the evaluation does not test it. This is not a local fix; a substantial rework of both the definition and the validation would be required to make the central claim credible. The reader's assessment aligns with my reading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The central contribution, the Community Opposition Index, is not actually defined: the internal cohesion term t_i appears in every version of the formula but never gets an equation or operationalization. And the empirical section never tests that metric; it tests a zero-shot stance detection pipeline instead. Take the stance detection results at face value though. On VAST the system is 8.4 points above the best baseline, and on most SEM16 targets it beats COLA, the leading role-based agent baseline. That is a genuine result, and the five-agent architecture (social media veteran, linguistic expert, sentiment expert, polarization assessor, plus background mining) is a plausible reason for it. The graph representation idea, extending the Sentiment Thermometer to a multi-group directed graph, is also new and worth keeping. Where it falls apart is the metric and the protocol. Equations (1)-(3) show t_i only symbolically; no algorithm computes it. A reader cannot reproduce COI from the description. Algorithm 2 hardcodes 10x10 matrices and fills in missing personal stances by sampling from observed frequencies with no seed, so the edge weights and the COI change between runs. The comparison is also confounded: 'Ours' uses GPT-4 as the final assessor while the GPT-3.5 baselines use GPT-3.5 only, so the improvement may come from the stronger model, not the agent structure. No code or prompts are released, so the pipeline cannot be reproduced from the text. And the paper's own statement that no polarization benchmark exists is accurate, but substituting stance detection does not validate a polarization measurement. Stance classification measures agreement with gold labels; COI measures something else and that something else is uncalibrated against any external criterion. I don't think the authors are hiding anything; the paper reads as an honest draft that overclaims what has been established. The right fix is a real definition and test of t_i, deterministic imputation or a reported seed with variance estimates, a controlled model comparison (same final model for all systems), and at least one external validation of COI against a known polarization case. For a reader: if you work on LLM agents for stance detection or graph-based sentiment aggregation, this paper has useful results and a worthwhile architecture. If you work on polarization measurement, treat it as a proposal with a missing center. I would not desk reject it; the stance detection work is substantive and the CSN idea deserves referee time to be turned into something testable. But it needs major revision before it can be considered a polarization measure.","headline":"Strong stance detection results wrapped around an undefined polarization metric; the paper is a solid agent-system contribution but not yet a polarization measure.","tokens_in":706,"tokens_out":772,"would_cite":false,"duration_ms":34941,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that group polarization on social media is best measured as a graph of intergroup sentiments, built by LLM agents, and summarized by a single Community Opposition Index.","keywords":["multi-agent system","group polarization","large language models","community sentiment network","community opposition index","stance detection","sentiment analysis","social media analysis"],"falsifier":"Run the full pipeline on a comment corpus where an independent expert panel has ranked subsets by polarization; if the COI ordering diverges from the panel ordering, the index does not capture what humans mean by group polarization. A second check is to rerun the stochastic imputation of missing stances with different random seeds and see whether the resulting COI values are stable.","tokens_in":11157,"feed_emoji":"🕸️","tokens_out":6105,"duration_ms":61645,"temperature":0.7,"pith_summary":"This paper proposes a new way to measure group polarization on social media: instead of counting volumes, classifying sentiments, or analyzing social networks in isolation, it builds a directed graph called the Community Sentiment Network (CSN), where nodes are stance-based subgroups and edges carry sentiment scores between them. The paper's central claim is that this graph, constructed by a multi-agent system of large language models, supports a quantitative metric, the Community Opposition Index (COI), that captures both internal cohesion within subgroups and hostility between them, making polarization measurement more usable, accurate, and interpretable than prior approaches. To support the claim, the paper tests the agent pipeline on zero-shot stance detection across three benchmark datasets and reports substantial gains over existing methods, especially on the VAST dataset. A sympathetic reader would care because the approach promises a practical, explainable tool for tracking how online discussion fragments over time.","feed_headline":"One graph turns comments into a group-polarization score","feed_subtitle":"A community opposition index weighs subgroup hostility and cohesion from social media texts.","key_machinery":"The load-bearing machinery is the Community Sentiment Network (CSN), a directed cyclic graph with sentiment-labeled edges, and the multi-agent LLM pipeline that builds it. The pipeline runs in three stages: Background Mining (a Domain Specialist and Subgroup Exploration Expert infer the event context and candidate subgroups), Semantic Analysis (Social Media Veteran, Linguistic Expert, and Sentiment Analysis Expert decode slang, sarcasm, and rhetoric), and Polarization Assessment (a Polarization Assessor outputs per-comment triplets of stance, sentiment score, and target subgroup). The triplets are aggregated into an adjacency matrix with like-weighted averaging, and stance labels missing from incomplete triplets are imputed stochastically from observed frequencies. From this matrix, the Community Opposition Index (COI) computes a single polarization number that combines subgroup size, internal cohesion, and intergroup hostility.","core_discovery":"The paper claims that group polarization can be represented as a temporal graph of subgroup sentiments and quantified by a single index. The Community Sentiment Network (CSN) generalizes the two-party 'Sentiment Thermometer' to any number of subgroups: vertices are opinion subgroups, directed edges carry average sentiment (positive or negative) from one subgroup toward another, and self-loops capture internal cohesion. The Community Opposition Index (COI) sums, over all subgroups, the size-weighted product of internal cohesion and negative (oppositional) sentiment directed at other subgroups, formally $\\sum_i \\frac{n_i}{N} t_i \\sum_j (-e_{ij}) \\mathbf{1}_{e_{ij} \\le 0}$. The paper validates the construction pipeline, not the index directly, by showing that its LLM-based multi-agent system achieves strong zero-shot stance detection accuracy on SEM16, P-Stance, and VAST, surpassing prior baselines on several targets.","pith_inferences":["Beyond the paper's validation, a natural next step is to test whether COI ordering matches expert ratings or independent polarization surveys on the same comment corpus, something the paper does not do.","The internal cohesion term $t_i$ is stated but its computation is not specified in the presented equations, so the index's behavior depends on a definition the paper leaves open.","Because Algorithm 2 imputes missing personal stances by sampling from observed frequencies, the index has an inherent stochastic component; a robustness check would report variance across seeds.","The same graph could support derivative metrics beyond COI, such as detecting which subgroups are most isolated or most adversarial over time."],"forward_implications":["If COI is a valid measure, polarization can be tracked over time by computing it on successive time slices of a comment stream, giving a temporal curve of group fragmentation.","The graph representation exposes which pairs of subgroups drive polarization, since the CSN keeps the edge-level hostility scores that COI aggregates.","The multi-agent design can be adapted to new topics without retraining, since the agents operate zero-shot from background mining.","The method downgrades the importance of volume: COI is relative and size-weighted, so large neutral crowds do not mask small hostile factions."],"supporting_citations":[{"why":"Defines the two-group Sentiment Thermometer that CSN extends to multiple subgroups.","marker":"[18]"},{"why":"Supports the premise that out-group hostility drives polarization, which COI formalizes.","marker":"[26]"},{"why":"Establishes affective polarization as intergroup animosity, grounding the design of COI.","marker":"[17]"},{"why":"Provides the SEM16 stance detection dataset used to evaluate the agent system.","marker":"[34]"},{"why":"Provides the P-Stance political dataset for zero-shot stance evaluation.","marker":"[30]"},{"why":"Supplies the VAST zero-shot stance dataset and the TGANet baseline method.","marker":"[1]"},{"why":"The COLA LLM-based agent baseline that the proposed system improves upon.","marker":"[25]"}],"fun_headline_variants":["LLM agents build a graph that scores group polarization","Graph index measures group hostility and cohesion from posts","Community Opposition Index: quantifying polarization via graph","From text to graph: LLM multi-agent system gauges polarization","CSN and COI: a graph approach to measure polarization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if correct per-comment stance, target, and sentiment extraction does not guarantee that the aggregated Community Opposition Index is a valid measure of group polarization, since the paper validates only the extraction stage, not the index itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents build a graph that scores group polarization","Graph index measures group hostility and cohesion from posts","Community Opposition Index: quantifying polarization via graph","From text to graph: LLM multi-agent system gauges polarization","CSN and COI: a graph approach to measure polarization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1436,"prompt_tokens":913,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":529,"tokens_out":523,"duration_ms":6327,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:49:31.384722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a comment corpus where an independent expert panel has ranked subsets by polarization; if the COI ordering diverges from the panel ordering, the index does not capture what humans mean by group polarization. A second check is to rerun the stochastic imputation of missing stances with different random seeds and see whether the resulting COI values are stable.","supporting_citations":[{"cited_title":"Public Opinion Quarterly76(3), 405–431 (2012) 3, 5, 10","cited_arxiv_id":null,"evidence_quote":"Defines the two-group Sentiment Thermometer that CSN extends to multiple subgroups."},{"cited_title":"The Journal of Politics 79(2), 485–501 (2017) 2, 3, 10","cited_arxiv_id":null,"evidence_quote":"Supports the premise that out-group hostility drives polarization, which COI formalizes."},{"cited_title":"Annual Review of Political Science 22(1), 129–146 (2019) 2, 3, 10","cited_arxiv_id":null,"evidence_quote":"Establishes affective polarization as intergroup animosity, grounding the design of COI."}],"review_version":1}