{"id":"76593fd6-a659-4831-b9af-83cd53ce31ee","arxiv_id":"2504.16947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SCRAG combines historical response retrieval, ideological clustering, and external news retrieval with LLMs to generate diverse predicted replies to social media posts.","lead":"This paper introduces SCRAG, a system that forecasts how social media communities will react to a post by retrieving past responses and current news, then prompting an LLM to write replies from each detected ideological group. It reports improved accuracy over direct and few-shot prompting across six Twitter/X scenarios, but the evaluation relies on automated LLM metrics that may favor the system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cluster Matching/Coverage metrics are partly circular: the same UMAP+HDBSCAN pipeline that supplies SCRAG's generation examples (Sec. III-B3) and the quota rule (Eq. 4) also score its outputs, so the coverage gains may reflect prompt construction rather than forecast quality.","rationale":"The paper presents a coherent modular RAG system, and the authors provide code and data plus an honest ablation study. The concern is not about internal consistency but about whether the principal empirical evidence distinguishes SCRAG from a much simpler procedure that uses the same clustering structure. The reader's weakest assumption and my load-bearing concern coincide; I therefore agree with the CONDITIONAL verdict. The cluster-sample control is a cheap, decisive experiment: if it matches SCRAG, the coverage and matching metrics are not measuring generation quality; if it does not, the concern is resolved in SCRAG's favor. I would not move the verdict to REJECT, because the non-circular metrics (Emotion JSD, discrimination score) and the qualitative example provide some evidence, and the confound is testable. But I would not accept unconditionally without the control or equivalent independent scoring.","tokens_in":14215,"tokens_out":5102,"duration_ms":53048,"concrete_test":"Implement a sampling control: for each test post, run SCRAG's retrieval and clustering stages, then generate the forecast set by randomly sampling one historical response from each cluster in proportion to cluster size, with no LLM rewriting. Score this control exactly as SCRAG is scored. If the control achieves Cluster Coverage within a few points of SCRAG's reported values, the coverage metric is dominated by the retrieval/clustering/quota design rather than by generation quality; the central claim would then need independent human or metric validation to stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the automatic evaluation metrics are unbiased indicators of forecast quality. For the two cluster-based metrics, that assumption is not secure. SCRAG's generation module is built from the output of Section III-B3 clustering: retrieved historical responses are grouped by combined text and ideological embeddings, and Eq. (4) allocates the M forecasts in proportion to cluster sizes. The evaluator then clusters the real withheld responses with the same Section III-B3 procedure (same UMAP + HDBSCAN + ideological embeddings) and measures whether the generated responses fall into those clusters and cover them. A baseline that simply sampled one historical response per cluster in proportion to cluster size would be expected to score highly on Cluster Coverage, because the generator's conditioning clusters and the evaluator's ground-truth clusters live in the same embedding geometry. Thus the reported 11.8% coverage improvement, and part of the 4.1% matching improvement, may be an artifact of giving SCRAG direct access to the clustering structure rather than evidence that it better forecasts the specific new post. The other two metrics are less circular but still rely on LLM judges from the same model family as the generator, with no validation against human ratings and no error bars or significance tests anywhere in Tables II and III. Section IV-A2 candidly states the metrics were chosen 'due to the high volume of generated texts and budget constraints,' but that does not remove the confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SCRAG, a retrieval-augmented generation framework for forecasting community responses to social media posts. Given a new post, SCRAG retrieves semantically and ideologically similar historical posts, clusters the retrieved responses via UMAP+HDBSCAN on combined text and ideological embeddings, retrieves news articles and knowledge-graph triplets via SPLADE, and prompts an LLM to generate M responses with a per-cluster quota proportional to cluster size (Eq. 4). The framework is evaluated on six X/Twitter scenarios (Russophobia, US2020, COVID, UK, US2024, Gaza) against direct and few-shot prompting baselines, with four automatic metrics: emotion JSD, LLM discrimination score, cluster matching, and cluster coverage. The paper reports average improvements of 10.2%, 1.5%, 4.1%, and 11.8% over baselines, respectively, and includes ablations and a qualitative example. The central claim is that SCRAG produces more realistic, diverse, and ideologically distributed forecasts than the baselines.","tokens_in":14476,"tokens_out":4431,"duration_ms":37706,"significance":"If the empirical claims were fully substantiated, SCRAG would be a useful modular contribution to social-media response forecasting, demonstrating that grounding generation in community-aware historical retrieval and external knowledge improves LLM forecasts. Strengths of the paper include the release of code/data, the use of multiple embedding models and LLMs, the temporal masking of test data, and the ablation studies. The main weakness is that the two cluster-based metrics share the clustering procedure with the generation module, and the LLM-based metrics are not validated against human judgments; these issues bear directly on the paper's headline improvements. The significance of the work is therefore contingent on strengthening the evaluation.","major_comments":[{"comment":"Cluster Matching Percentage and Cluster Coverage Percentage are computed with the same combined embedding and HDBSCAN clustering procedure that SCRAG uses to group retrieved historical responses and to allocate generation quota via Eq. (4). Because the generator's conditioning clusters and the evaluator's ground-truth clusters live in the same embedding geometry, these two metrics are favorably biased toward SCRAG by construction; a baseline that samples responses per cluster from the same clustering would also score highly on coverage. The authors should either evaluate with an independent clustering pipeline (e.g., different embeddings, a different clustering algorithm, or a held-out set of clusters) or replace these metrics with human-annotated diversity/relevance judgments.","section":"Section IV-A2 and Section III-B3"},{"comment":"All evaluation results are reported as averages over 30 test posts without error bars, confidence intervals, or significance tests. Given the small number of test posts and the small absolute differences in some metrics (e.g., a 1.5% average improvement in LLM discrimination score), it is not possible to determine whether the reported gains are statistically reliable. Please report per-scenario variance, paired significance tests (e.g., Wilcoxon signed-rank) across the 30 posts, and win rates across scenario-metric pairs.","section":"Tables II and III"},{"comment":"The LLM-based Emotion JSD and LLM Discrimination Score are used as proxies for human judgments of emotional faithfulness and realism, but they are not validated against any human ratings. The paper justifies automatic metrics by budget constraints, yet without a small human-validation study (or evidence that these LLM judges have been previously validated for this task) the construct validity of these metrics is unestablished. Note also that the judges include models from the same family as the generator (e.g., Llama3.3-70B), which may further bias comparisons.","section":"Section IV-A2"},{"comment":"Despite the summary claim of improvements, a close reading of Table III shows that SCRAG underperforms the fewshot baseline in several scenario-metric pairs, e.g., Qwen2.5-32B on Russophobia Cluster Matching (58.15 vs. 63.70), Gemma2-9B on Russophobia LLM Discrimination Score (7.948 vs. 8.167), and Mistral-Large on UK Cluster Coverage (63.17 vs. 69.83). The paper needs a systematic win/loss summary across all comparisons and a discussion of these inconsistent cases rather than only reporting the average.","section":"Table III"}],"minor_comments":[{"comment":"The manuscript contains duplicated content: Fig. 2 appears twice (once in Section III-D and once at the start of Section IV), Table I appears twice, and there are redundant duplicated paragraphs in Section IV. Please remove duplicates.","section":"General"},{"comment":"The header 'V oyageAI' contains an extra space; it should be 'VoyageAI'.","section":"Table II"},{"comment":"Equation numbering is inconsistent: the quota allocation equation is labeled Eq. (4) in one place and Eq. (1) in another. Please renumber all equations consistently.","section":"Section III-D"},{"comment":"The parenthetical numbers in scenario headers (e.g., 'Rus (72)') are not explained in the table captions; please define them in the text (e.g., number of clusters used for the cluster-matching evaluation) or make the caption self-contained.","section":"Tables II and III"},{"comment":"The code link is provided, but the paper says 'code and data' are available without specifying the license or conditions for data sharing; please clarify.","section":"Section IV-A1"}],"recommendation":"major_revision","confidential_remarks":"The main concern raised in the stress-test note—circularity of the cluster metrics—is legitimate and should be the focus of revision. I recommend major revision rather than rejection because the central idea is sound and the evaluation can be strengthened with additional experiments and analyses. The absence of significance testing is particularly concerning for a paper whose central claim is a quantitative improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nHere is my read on SCRAG. The genuinely new thing is the integration: retrieve historical responses from the target community, cluster them with text plus ideological embeddings, and condition an LLM on those clusters plus news and knowledge-graph context to forecast responses to a new post. The pipeline is modular, clearly described, and tested across six scenarios with several embedding models and LLMs. The authors also ship code and data, run ablations, and the qualitative example in Table IV does show ideology-aware output. That is real engineering value.\n\nThe problem is evaluation. The two cluster metrics — matching and coverage — use the same UMAP+HDBSCAN pipeline in Section III-B3 that selects generation examples, and Eq. (4) allocates the response quota in proportion to cluster sizes. So SCRAG is effectively being scored on whether it reproduces the cluster structure it was handed. A baseline that sampled one historical response per cluster in the same proportions would score well on coverage without forecasting the new post at all. That makes the 11.8% coverage improvement, and part of the 4.1% matching improvement, non-identifying. The stress-test note is right about this.\n\nThe other two metrics are less circular but not clean. Emotion JSD uses LLM-extracted emotions; the LLM Discrimination Score uses an LLM judge. Neither is validated against human raters, and no error bars or significance tests appear anywhere in Tables II and III. The best improvement on discrimination score is about 1.5%, which is tiny. The abstract's \"over 10% improvements\" is really one metric (emotion JSD) plus coverage; the other two are modest. Section IV-A2 openly says automatic metrics were chosen for budget reasons, which is honest but does not remove the confound.\n\nSo where does that leave the paper? The framework is plausible and likely useful, and the direction is sound. The central claim — that SCRAG forecasts more accurately than baselines — is not established by the evidence as presented. But this is fixable. Human evaluation on a sample, external cluster baselines, and significance testing would clarify whether the gains are real. I would not desk-reject this. I would send it to peer review and ask for a revised evaluation.\n\nAlso flag minor editorial issues: duplicated paragraphs and tables in the manuscript (e.g., Section IV-A and Table I appear twice) need cleanup.\n\nWho is it for: people building response-forecasting or social-media RAG systems. It will not reshape the field, but it is a solid application paper. I would not cite it over the next year; I would wait for the evaluation to firm up. I would bring it to reading group as an example of metric circularity in LLM evaluation. Verdict: serious thinker, yes; accept for review, yes.","headline":"Useful modular RAG framework for forecasting community responses, but the headline numbers rest on partly circular cluster metrics and no human validation; worth reviewing seriously, not accepting at face value.","tokens_in":15049,"tokens_out":2314,"would_cite":false,"duration_ms":22182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular retrieval-augmented generation framework that conditions an LLM on a community's historical responses and external news can forecast social media reactions more accurately, realistically, and diversely than prompting alone.","keywords":["social media response forecasting","retrieval-augmented generation","social computing","ideological embedding","community response prediction","large language models","knowledge graph","emotion distribution"],"falsifier":"Recompute cluster matching and coverage with a held-out clustering procedure that uses a different embedding model and a different clustering algorithm than the one SCRAG uses for retrieval and example selection, and check whether the reported average improvements persist.","tokens_in":13967,"feed_emoji":"💬","tokens_out":11295,"duration_ms":93880,"temperature":0.7,"pith_summary":"SCRAG is a framework for forecasting how a community on X (formerly Twitter) will react to a real or hypothetical post, producing individual predicted replies rather than a single aggregated summary. Its central claim is that grounding an LLM in retrieval-augmented generation—historical responses from the target community plus current news and knowledge-graph relations—yields forecasts that match real reactions better in emotional tone, realism, and ideological diversity. Across six scenarios and multiple embedding models and LLMs, the paper reports average improvements of 10.2% in emotion JSD, 1.5% in LLM discrimination score, 4.1% in cluster matching, and 11.8% in cluster coverage over direct and few-shot prompting baselines. If the claim holds, the practical payoff is that public relations teams, public figures, and crisis managers could preview how a message would land before posting, including for emerging events that historical data alone does not cover.","feed_headline":"LLMs grounded in past posts forecast community reactions 10% better","feed_subtitle":"By retrieving a community's past replies and live news, the framework cuts emotion divergence 10.2% and lifts ideological coverage 11.8%.","key_machinery":"The load-bearing mechanism is the community-aware historical response retriever combined with an external knowledge module. Historical posts are first augmented with their conversational context and embedded twice: once semantically and once ideologically, where the ideological embedding comes from a variational graph autoencoder over the user–post interaction graph. The two representations are combined after UMAP dimension reduction and clustered with HDBSCAN, so each cluster stands in for a community with a shared stance. A maximal-marginal-relevance-style selection picks representative responses per cluster, and a SPLADE sparse lexical-and-expansion retriever pulls current news snippets and knowledge-graph triplets. The generation step allocates the forecast quota by cluster size, $M_k \\propto |C_k| \\cdot M$, and prompts the LLM with each cluster's examples, ideology summary, and external context; this joint conditioning is what forces the output to be diverse, emotionally aligned, and grounded in current facts.","core_discovery":"The paper's discovery is that a social-computing-aware retrieval loop can turn a static LLM into a community response forecaster. SCRAG first augments each historical post with its reply-chain context, then embeds these augmented posts with both a text embedding model and an ideological embedding learned from the user–post interaction graph. For a new post, it retrieves similar past posts, gathers their real responses, and clusters those responses with UMAP followed by HDBSCAN on the combined embeddings, so each cluster represents a community with a shared stance. Representative responses per cluster are selected with a maximal-marginal-relevance-style diversity criterion, while a sparse SPLADE retriever separately fetches recent news snippets and knowledge-graph relation triplets. The LLM is then prompted once per cluster with the cluster's representative responses, a summary of its ideology, and the external context, and the number of responses per cluster is allocated in proportion to cluster size. The paper reports that this pipeline outperforms direct and few-shot prompting across six test scenarios, with the largest gains in emotional fidelity and cluster coverage, and its ablations show that the ideological embedding, sparse news retrieval, and knowledge-graph relations each contribute to the result.","pith_inferences":["Editorial extension: because cluster matching and coverage are computed with the same embedding and clustering machinery SCRAG uses for retrieval, an independent human-rating study or a held-out clustering procedure would separate genuine forecast quality from alignment with the framework's own cluster geometry.","Editorial extension: the ideology representation is two-sided (pro/anti), so applying SCRAG to multi-polar debates would require extending the belief-representation component beyond a single axis.","Editorial extension: comparing the predicted per-cluster response counts against actual reply counts per stance in held-out responses would give a direct test of whether the quota-allocation heuristic also predicts community activity levels."],"forward_implications":["SCRAG can generate forecasts for hypothetical posts, so communicators could compare how alternative phrasings would be received by different ideological clusters before publishing.","Because the external-knowledge module supplies fresh news and entity relations, the same framework can forecast reactions to new events that are not represented in the historical response database, as the two adaptability-testing scenarios demonstrate.","The quota allocation proportional to cluster size makes the framework an activity forecaster as well as a content forecaster, predicting how much of the response volume each ideological community will contribute.","The framework's modularity across embedding models and LLMs means its gains are not tied to one proprietary model, supporting deployment under different resource constraints."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the base paradigm SCRAG extends to community response forecasting.","marker":"[5]"},{"why":"Supplies the continuously updated global news articles used as external knowledge for news retrieval.","marker":"[6]"},{"why":"Provides the variational graph autoencoder method for ideological embeddings of users and posts used in combined clustering.","marker":"[7]"},{"why":"Provides the disentangled belief representation learning used to compute ideological embeddings on polarized social graphs.","marker":"[8]"},{"why":"Supplies UMAP, used to reduce semantic embeddings before concatenating them with ideological embeddings.","marker":"[9]"},{"why":"Supplies HDBSCAN, the density-based clustering algorithm that discovers community clusters in retrieved responses.","marker":"[10]"},{"why":"Supplies SPLADE, the sparse lexical-and-expansion retriever used to fetch news snippets and knowledge-graph triplets.","marker":"[12]"},{"why":"Provides a public dataset incorporated into the Russophobia historical response database.","marker":"[13]"},{"why":"Provides a public dataset incorporated into the US Election 2020 historical response database.","marker":"[14]"},{"why":"Provides a public dataset incorporated into the COVID historical response database.","marker":"[15]"}],"fun_headline_variants":["Retrieving community history and news helps LLMs forecast reactions","Past replies plus live news let LLMs predict community responses","SCRAG boosts community response forecasting via social-computing retrieval","Grounding LLMs in past posts and news improves response forecasts 10%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result rests on the assumption that the automatic evaluation metrics—especially cluster matching and cluster coverage, which use the same embedding model and HDBSCAN clustering procedure that SCRAG uses to select its generation examples—measure forecast quality fairly rather than favoring SCRAG by construction.","fun_headline_variants_meta":{"raw":{"variants":["Retrieving community history and news helps LLMs forecast reactions","Past replies plus live news let LLMs predict community responses","SCRAG boosts community response forecasting via social-computing retrieval","Grounding LLMs in past posts and news improves response forecasts 10%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3759,"prompt_tokens":1061,"completion_tokens":2698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":2625}},"tokens_in":677,"tokens_out":2698,"duration_ms":18027,"temperature":1.0,"reasoning_tokens":2625,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:01:05.639763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute cluster matching and coverage with a held-out clustering procedure that uses a different embedding model and a different clustering algorithm than the one SCRAG uses for retrieval and example selection, and check whether the reported average improvements persist.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the base paradigm SCRAG extends to community response forecasting."},{"cited_title":"The gdelt project","cited_arxiv_id":null,"evidence_quote":"Supplies the continuously updated global news articles used as external knowledge for news retrieval."},{"cited_title":"Unsupervised belief representation learning with information-theoretic variational graph auto-encoders,","cited_arxiv_id":null,"evidence_quote":"Provides the variational graph autoencoder method for ideological embeddings of users and posts used in combined clustering."},{"cited_title":"Large language model-guided disentangled belief representation learning on polarized social graphs,","cited_arxiv_id":null,"evidence_quote":"Provides the disentangled belief representation learning used to compute ideological embeddings on polarized social graphs."},{"cited_title":"Splade: Sparse lexical and expansion model for first stage ranking,","cited_arxiv_id":null,"evidence_quote":"Supplies SPLADE, the sparse lexical-and-expansion retriever used to fetch news snippets and knowledge-graph triplets."},{"cited_title":"Tweets in time of conflict: A public dataset tracking the twitter discourse on the war between ukraine and russia,","cited_arxiv_id":null,"evidence_quote":"Provides a public dataset incorporated into the Russophobia historical response database."},{"cited_title":"# election2020: the first public twitter dataset on the 2020 us presidential election,","cited_arxiv_id":null,"evidence_quote":"Provides a public dataset incorporated into the US Election 2020 historical response database."},{"cited_title":"Tracking social media discourse about the covid-19 pandemic: Development of a public coronavirus twitter data set,","cited_arxiv_id":null,"evidence_quote":"Provides a public dataset incorporated into the COVID historical response database."}],"review_version":1}