{"id":"cbf7e5e0-0a58-47d3-8bc5-d61895c1157e","arxiv_id":"2411.13681","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A large meta-analysis of OSN research, built on the Minerva-OSN dataset of 13,842 papers, shows research concentrates on Twitter and a few topics while under-covering platforms popular among younger users.","lead":"This paper builds a dataset of 13,842 online social network research papers and analyzes which platforms, topics, and countries dominate the field, then surveys 50 experts about data access challenges. It gives researchers, platform owners, and policymakers a quantitative picture of how OSN research is concentrated and why it may be misaligned with real-world platform popularity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sampling frame (135 CS venues plus abstract must name a platform) may bias the core statistics; a broader-coverage validation is needed before accepting that OSN research is unrepresentative.","rationale":"The paper's central contribution is a quantified, holistic description of OSN research. That description inherits every property of the sample construction. I find two mechanisms by which the sample could be non-representative: venue selection (135 largely CS venues) and the abstract-name-mention heuristic. Both are acknowledged in the text, but the acknowledgment does not bound the magnitude of the resulting bias. The manual inspection step addresses false positives only, not false negatives, so the direction of bias is unknown. This is not an accusation of sloppiness; the authors transparently document their choices and limitations. However, the claims 'only 91 of 296 OSN investigated', 'Twitter predominant since 2012', and 'OSN research is not representative' are strong and would be actionable for funders and policymakers. Before such claims are accepted, the sampling frame should be validated against a broader universe that includes non-CS venues and papers that do not name a platform in the abstract. My proposed test does exactly this. The reader's CONDITIONAL verdict is appropriate; I do not recommend changing it, but the condition should be the validation I describe rather than merely 'check the code'. If the test passes, the paper's conclusions are substantially strengthened; if it fails, the central narrative would need to be reframed as a statement about a specific CS-heavy segment of the literature.","tokens_in":24309,"tokens_out":4310,"duration_ms":45329,"concrete_test":"Query Scopus or OpenAlex for all 2006-2023 papers with 'social media' OR 'online social network' in title/abstract, restricted to 50 non-CS journals (e.g., Journal of Medical Internet Research, Cyberpsychology Behavior and Social Networking, New Media & Society, Journal of Business Research, American Behavioral Scientist) in addition to the 135 venues. From the union, draw a stratified random sample of 500 papers and have two independent annotators label (a) whether the paper is OSN research and (b) which OSN it primarily studies. Compare the resulting platform distribution and number of OSN covered with Minerva-OSN (e.g., Twitter share of papers and the 91/296 ratio). If Twitter's share drops by >10 percentage points or the number of studied OSN rises by >20%, the central claims about nonrepresentativeness and Twitter predominance are sampling artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Minerva-OSN is built from (i) 135 venues selected from Google Scholar's top CS subcategory venues plus LNCS and ACM ICPS, and (ii) a heuristic keeping a paper only if its abstract mentions at least one of 296 OSN names ('Venues and candidate papers'; 'Filtering and Validation'). Both decisions are load-bearing for every descriptive claim in RQ1-RQ5. The venue list is dominated by computer-science conferences and journals; health, psychology, communication, and management venues are absent, although these fields publish substantial OSN research and often study different platforms (e.g., Instagram, Snapchat, TikTok). The abstract-mention rule creates a second, compounding bias: papers that refer to 'social media' or 'online communities' without naming a platform are excluded, as the authors acknowledge in Limitations ('our heuristic may have not captured papers that did not mention any specific OSN in the abstract'). The manual validation only removes false positives (up to 50 papers per OSN) and never quantifies false negatives. Because the 13,842-paper dataset is the sole evidence base for 'Twitter has been predominant since 2012', 'only 91 of 296 OSN have been investigated', and 'OSN research is not representative of the real world anymore', an unmeasured systematic gap in the frame could overturn or materially alter these headlines. The paper is internally consistent and carefully caveated, but the external validity of the sampling frame is the load-bearing condition.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the most substantial attempt yet at a quantitative meta-analysis of OSN research, and the Minerva-OSN dataset is a real resource. 13,842 papers, 135 venues, public code and data. They quantify things everyone suspected—Twitter dominance, topic concentration, data access problems—and the survey, despite its small n, adds useful color. If you work on OSN research, you will likely cite this.\n\nThe methodological core is a two-stage filter: venues limited to 135 CS-oriented venues from Google Scholar top lists plus LNCS and ACM proceedings, then papers kept only if the abstract names one of 296 OSN. Both decisions are acknowledged, and the venue list is indeed CS-heavy. I don't think this is fatal, but it is load-bearing for the headline claim that 'OSN research is not representative of the real world anymore.' The claim as stated is about all OSN research; the evidence is about CS-venue research that mentions a platform by name. Health, psychology, communication, and management venues are largely absent, and those fields study different platforms. A paper on 'social media use and depression' that never says 'Facebook' in the abstract is invisible here. So the absolute counts and the 91/296 figure are lower bounds with unknown error. The qualitative pattern—Twitter overstudied, popular newer platforms understudied—is almost certainly robust; prior single-platform reviews point the same way. But the precision of the numbers, and the nonrepresentativeness claim, need either a broader sample or more careful caveating.\n\nOther soft spots are minor. The expert survey has a 2% response rate and is Europe-senior-Twitter heavy; they say so. The GDPR discussion is explicitly labeled conjecture, which is the right call. The topic model validation is genuinely thoughtful—three independent reviewers, 72.5% agreement with the model—and better than most work in this area.\n\nBottom line: this deserves serious peer review. The right referee will push on the sampling frame, and the authors may need to reframe some claims or run a validation set from non-CS venues. But the dataset alone justifies the paper, and the meta-analytic approach is sound enough to build on.","headline":"First holistic OSN meta-analysis with a genuinely useful public dataset; sampling-frame bias is real but the qualitative findings are likely robust — send to review.","tokens_in":25122,"tokens_out":2500,"would_cite":true,"duration_ms":24611,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the only holistic dataset of OSN research reveals a field concentrated on Twitter and out of step with real-world platform use.","keywords":["Online Social Networks","Minerva-OSN","Meta-analysis","Topic modeling","Research trends","Data access policies","Twitter prevalence","Expert survey"],"falsifier":"Re-run the pipeline on a sample of Scopus-indexed health and psychology venues not in the 135-venue list, and on abstracts that mention 'social media' without naming a platform; if the Twitter share drops substantially or the number of studied OSN rises well above 91, the paper's descriptive claims are artifacts of the selection frame.","tokens_in":24070,"feed_emoji":"📊","tokens_out":5852,"duration_ms":55251,"temperature":0.7,"pith_summary":"This paper tries to establish, for the first time, a holistic quantitative picture of all academic research on online social networks (OSNs) since 2006. It builds a public dataset, Minerva-OSN, of 13,842 peer-reviewed papers drawn from over a million candidates, and argues that the literature is lopsided: only 91 of 296 platforms have ever been studied, Twitter is the dominant subject since 2012 despite its declining popularity, and research output began dropping around 2018. The paper also argues that data-access policies of OSN owners are a root cause, and supports this with a manual review of eight platforms' APIs and a survey of 50 experienced researchers. If these claims hold, the community has the first evidence that OSN research is not representative of the platforms that people actually use, and a released dataset to test and extend that conclusion.","feed_headline":"Only 91 of 296 social networks have been studied","feed_subtitle":"A 13,842-paper meta-analysis shows research skews to Twitter while TikTok and BeReal stay barely studied","key_machinery":"Minerva-OSN, a curated dataset of 13,842 OSN papers, is the load-bearing object. Its construction has three stages: selecting 135 venues (top Google Scholar venues plus Lecture Notes in Computer Science and ACM proceedings), screening over one million abstracts with a heuristic that keeps a paper only if the abstract names at least one of 296 OSNs, and manual checks for homonyms and false positives. On top of this, a BERTopic pipeline, using sentence embeddings, UMAP, and HDBSCAN, assigns 17 topics, with OSN names replaced by a neutral tag to avoid platform bias; a tri-party human review of 170 abstracts validates the topic labels, achieving 72.5% first-choice agreement. The dataset and pipeline carry every subsequent claim about prevalence, topics, authors, and popularity mismatch.","core_discovery":"The paper's central discovery is that the body of OSN research is concentrated on a few platforms, above all Twitter, and that this concentration has detached research from real-world platform usage. Among 296 OSNs considered, only 91 have been investigated; Twitter accounts for 5,248 of 13,842 papers, and it has been the most studied OSN since 2012 even though its website-traffic rank has fallen. Only 82 papers concern TikTok despite it having more than a billion users and a top-10 ranking. The paper connects this skew to data-access policies: platforms with free or cheap APIs, notably Twitter before April 2023, attracted research, while restrictive or expensive access deterred it. A survey of 50 researchers finds that 78% never collaborated with an OSN and that most call data access and reproducibility difficult. The conclusion is that OSN research carries nonrepresentative bias, with consequences for topics tied to younger demographics.","pith_inferences":["If the selection heuristic were widened to abstracts using generic phrases like 'social media' without naming a platform, method-only and cross-platform papers would likely enter the dataset; the reported Twitter share could then shrink, so the dominance number is partly a property of the search rule.","The 2018 decline in US and EEA output could be checked against a matched corpus of non-OSN papers from the same venues; if those also decline, the GDPR explanation is weaker than a general publication trend.","A testable projection arises: the 2023 API paywalls at Twitter and Reddit should push future OSN research toward platforms with open access, such as Mastodon or Wikipedia, and reduce Twitter's share in follow-up versions of Minerva-OSN."],"forward_implications":["Minerva-OSN gives researchers a quantitative baseline to measure whether future OSN research broadens beyond a handful of platforms.","The paper's evidence makes the case that restrictive data-access policies have a measurable effect on what gets studied.","Topical gaps follow from the skew: issues affecting younger demographics are concentrated on platforms like TikTok and Instagram that the literature has yet to cover adequately.","The 17-topic model and its tri-party validation give future literature analyses a reusable procedure instead of an ad hoc manual review."],"supporting_citations":[{"why":"Defines the starting point of OSN research; the paper's time frame begins with this seminal Facebook study.","marker":"(Acquisti and Gross 2006)"},{"why":"Supplies the method for compiling the OSN list and the comparative approach for reviewing data-access policies.","marker":"(Fiesler, Beard, and Keegan 2020)"},{"why":"Provides the prior Twitter-focused survey and the 2006 start date, serving as the baseline for prevalence comparisons.","marker":"(Antonakaki, Fragopoulou, and Ioannidis 2021)"},{"why":"Introduces BERTopic, the topic-modeling method used to assign the 17 topics across the dataset.","marker":"(Grootendorst 2022)"},{"why":"Supplies LLAMA-2, the model used to label the topics automatically.","marker":"(Touvron et al. 2023)"},{"why":"Provides the systematic literature review guidelines that structure the dataset construction.","marker":"(Booth et al. 2021)"},{"why":"A single-OSN Facebook review used as contrast to the paper's holistic scope.","marker":"(Wilson, Gosling, and Graham 2012)"},{"why":"A single-topic SoK on content moderation, used to show that prior reviews did not span the whole OSN field.","marker":"(Singhal et al. 2023)"}],"fun_headline_variants":["Social network research ignores 69% of platforms, review finds","Twitter dominates 13,842 social network papers, TikTok barely seen","Meta-analysis: only 91 of 296 social networks studied","Data access skews social network research to Twitter","Social network research has a Twitter problem, 13,842 papers show"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis depends on the selection rule that a paper is about an OSN only if its abstract names at least one OSN; papers that study online social networks without spelling out a platform, and venues outside the 135 chosen, are invisible to the dataset.","fun_headline_variants_meta":{"raw":{"variants":["Social network research ignores 69% of platforms, review finds","Twitter dominates 13,842 social network papers, TikTok barely seen","Meta-analysis: only 91 of 296 social networks studied","Data access skews social network research to Twitter","Social network research has a Twitter problem, 13,842 papers show"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001042,"raw_usage":{"total_tokens":4437,"prompt_tokens":1055,"completion_tokens":3382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":3296}},"tokens_in":671,"tokens_out":3382,"duration_ms":26372,"temperature":1.0,"reasoning_tokens":3296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:59:02.157568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on a sample of Scopus-indexed health and psychology venues not in the 135-venue list, and on abstracts that mention 'social media' without naming a platform; if the Twitter share drops substantially or the number of studied OSN rises well above 91, the paper's descriptive claims are artifacts of the selection frame.","supporting_citations":[],"review_version":1}