{"id":"93400893-5e77-4a16-a289-dabf20b7a6f8","arxiv_id":"2505.01754","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A comparative media-bias pipeline on 37 news outlets using hierarchical topics, sentiment, entities, and LLM ontologies was tested, with clear signals from titles and entities but failures in body sentiment and larger ontology comparisons.","lead":"This paper describes a modular NLP pipeline for measuring media bias by comparing how 37 international newspapers cover the same political topics. It combines topic modeling, sentiment analysis, and GPT-4-built ontologies, and tests the approach on three case studies with mixed results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entity-level count deviations in the media bias spectrum are unnormalized; raw mention counts may reflect outlet size rather than media bias, so the central effectiveness claim is not yet supported.","rationale":"The reader identified the lack of external ground truth as the weakest assumption. I agree that is a problem, but I found a more concrete, internal flaw that independently threatens the effectiveness claim: the entity-level count axis of the media bias spectrum is not normalized by outlet size, despite the paper's own principle that relative values compensate for uneven publication rates. Because the corpus is highly imbalanced, raw entity counts conflate scale with emphasis. This is not merely a disagreement with consensus; it is an internal inconsistency in how the headline instrument is defined. A re-analysis with normalized rates could settle it. The absence of uncertainty quantification compounds the issue: multiple newspapers appear in the spectra with one article, so the spread in sentiment deviation may be noise. The paper is honest about the failed components, which supports a conditional rather than unconditional conclusion; my concern strengthens the conditions but does not change the reader's verdict.","tokens_in":32942,"tokens_out":5127,"duration_ms":53906,"concrete_test":"Recompute the entity-level media bias spectra in Tables A3, A4, B7, B8, and C10 using the entity mention rate per newspaper (entity count divided by that newspaper's total article count) instead of raw count. Compare the rank order and sign of count deviations before and after normalization; if material reordering occurs, the y-axis reflects outlet size. As a second check, run a bootstrap or exact binomial test on the title-sentiment deviations for newspapers with one, two, or three articles in t138 and t751; if the 95% intervals include zero, those spectrum positions do not support the claimed bias differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5.2: \"successfully analyze the potential media bias in event selection\") depends on the media bias spectrum. For articles, Eq. (1) normalizes coverage by each newspaper's total article count, but for entities Section 3.5 defines the y-axis as the deviation of the raw number of entity mentions from the mean raw count, with no normalization by newspaper size. In the dataset (Section 4.1) per-newspaper article counts range from 91 (CommonWealth Beacon) to 10,230 (The Hindu), so raw entity counts in Tables A3-A4, B7-B8, and C10 are dominated by outlet output. For example, the \"Israel\" entity spectrum in C10 gives count deviations of +2,468 for Jerusalem Post and +2,422 for NDTV, both high-output outlets, while a normalized rate could reorder these. Additionally, many sentiment deviations are computed from one or two articles (e.g., CBC, Mothership, France24 in t138), with no confidence intervals or significance tests, so the reported differences may be sampling noise. The paper itself concedes the body-text sentiment component was useless and the second ontology case failed to reveal commission/omission bias (Section 5.1), yet the abstract's generalized effectiveness claim is retained. Without ground truth or uncertainty bounds, mean-deviation scores are not sufficient to establish media bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Jähde, Weber, and Buchkremer present a modular NLP pipeline for comparative media-bias analysis. The pipeline combines BERTopic-based hierarchical topic modeling with RoBERTa and spaCy sentiment analysis at the title, body-text, and entity levels, and GPT-4-based ontology learning with automatic consistency checks. Outlets are compared through a 'media bias spectrum' showing each newspaper's deviation from the cross-outlet mean for coverage share and sentiment. The method is applied to 40,308 English RSS articles from 37 outlets in three case studies: the Supernova festival attack (t138), the 2023 Belt and Road Forum and Putin's visit (t751), and a cross-topic study. The paper reports success for event-selection bias and for labeling/word-choice bias measured through title and entity sentiment; it concedes that spaCy body-text sentiment was not informative and that ontology-based commission/omission analysis worked in the first case study but not in the second.","tokens_in":33213,"tokens_out":6696,"duration_ms":65835,"significance":"The paper has several genuine strengths. The comparative, deviation-from-mean design is transparent and reproducible in principle; Eqs. (1)–(5) are definitions rather than fitted quantities, so the bias scores do not smuggle in a fitted target. The ontology-consistency checks (object-class, object-object, object-relation) provide a concrete quality gate for LLM outputs, and the case studies include manual spot-checks that give the reader a sense of where the method works. If the effectiveness claim were fully supported, the 'media bias spectrum' would be a practical, scalable tool for comparing outlets. However, the evidence as presented does not yet support the generalized claim: entity-level count deviations are unnormalized, small-sample sentiment differences are reported without uncertainty, and the methodology's own results in Section 5.1 limit the validated scope to title/entity sentiment and some event-selection comparisons. The central claim is therefore defensible only after the measurement and validation gaps are addressed.","major_comments":[{"comment":"The y-axis for the entity-level media bias spectrum is the deviation of the raw number of entity mentions from the mean raw count, with no normalization by outlet size. Because per-outlet article counts range from 91 (CommonWealth Beacon) to 10,230 (The Hindu) in Section 4.1, raw mention counts largely reflect outlet output rather than bias. In Table C10, for example, Jerusalem Post (+2,468) and NDTV (+2,422) have the largest positive count deviations partly because they are the highest-output outlets in the dataset. The claim in Section 5.2 that the methodology 'successfully analyze[s]' event-selection bias depends on this axis. Please re-compute entity-level spectra with rates (e.g., mentions per 1,000 articles, or a regression controlling for outlet article volume) and report whether the qualitative conclusions change.","section":"Section 3.5, entity-axis definition; Tables A3–A4, B7–B8, C10"},{"comment":"Many sentiment deviations are computed from a single article (e.g., CBC, Mothership, France24, Al Jazeera, El Pais, Straits Times, Japan Times, and BBC in t138; Bangkokpost, Mothership, El Pais, Journal, and Korea Herald in t751), and the paper reports no confidence intervals or significance tests. A deviation of -0.59 from one article cannot be distinguished from sampling noise. Please add uncertainty quantification, such as bootstrap intervals or a minimum-count threshold with suppressed or flagged points, and re-state the corresponding conclusions in Sections 5.1–5.2.","section":"Section 4.3, Table A1; Section 4.4, Table B5"},{"comment":"The generalized effectiveness claim is broader than the evidence in the paper itself. Section 5.1 states that the spaCy body-text sentiment analysis 'was found to hold no valuable result' and that for t751 'the proposed methodology failed to provide insights into omission and commission bias'; Section 5.2 concedes that commission and omission 'could not be consistently and successfully analyzed.' The abstract nonetheless says the methodology 'examines ... commission and omission biases' and 'demonstrate[s] the methodology's effectiveness in identifying biases across news sources.' Please narrow the abstract and conclusion to the components that survived the case studies (title and entity-level sentiment; event-selection comparisons), and present ontology-based commission/omission analysis as preliminary.","section":"Abstract and Section 5.2"},{"comment":"The bias scores are deviations from the mean of the included newspapers, and the paper explicitly rejects a zero-bias baseline. This is a defensible design choice, but the central claim that the scores identify media bias requires some evidence that the cross-outlet mean is a meaningful reference. As it stands, the method is calibrated to the selected corpus; the same scores would change if a different set of outlets were included. A concrete validation would be to compare a small set of computed deviations against independent bias judgments (e.g., human annotation or established outlet ratings such as AllSides or Ad Fontes) for the same topics, and to report agreement. Without such a check, the wording 'successfully analyze the potential media bias' in Section 5.2 remains an assumption about the reference distribution rather than a demonstrated result.","section":"Section 2.1 and Eqs. (2), (5)"}],"minor_comments":[{"comment":"There is a stray '].' immediately following the sentence that ends with '[60].' Please remove it.","section":"Section 3.3"},{"comment":"'Statistical interference' should be 'statistical inference,' and 'assorted communities' in Section 4.4 should be 'assortative communities.'","section":"Section 3.5 and Sections 4.3–4.4"},{"comment":"The phrase '150 characters suitable of the target' is ungrammatical; it should read 'to the right of the target' or 'after the target.'","section":"Section 4.2"},{"comment":"The summation notation \\sum_{i=0}^{|a(N_t)|} does not define the indexing variable; please write the average more explicitly or define i and the bounds.","section":"Section 3.5, Eq. (3)"},{"comment":"The appendix heading 'Thirt Beld and Road Forum' contains a typo; it should be 'Third Belt and Road Forum.'","section":"Appendix B"},{"comment":"'Mispresented' should be 'misrepresented' in the discussion of the Moscow Times title.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, and the empirical components are described in enough detail to be reproduced. My main concern is not integrity but scope: the abstract and Section 5.2 claim more than the results support, and the entity-count normalization issue is mechanical to fix. I would encourage the editor to require a revision that either normalizes entity counts, adds uncertainty bounds, or narrows the claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this one. First, the novelty is at the integration level: nobody else has combined hierarchical BERTopic, RoBERTa title and entity sentiment, and GPT-4 ontology learning into a single pipeline for event-selection, labeling, and commission/omission bias. Second, the abstract's claim that the methodology's effectiveness is demonstrated is not supported by the paper's own evidence. The paper is honest about much of this in its limitations section, but the headline claim outruns the results.\n\nWhat's good: the pipeline is clearly described and reproducible in principle. The case-study narratives are grounded in manual reading; for example, the Jerusalem Post's positive title sentiment in t138 is verified by actual headlines. The consistency checks for GPT-4 ontology extraction (object-class, object-object, object-relation) are a sensible response to hallucination. The comparison with prior work is fair, and the limitations section is unusually candid. It admits spaCy body-text sentiment was useless, that the second ontology case failed to reveal commission/omission bias, and that per-newspaper article counts range from 91 to 10,230.\n\nThe main problem is that \"effectiveness\" is asserted rather than demonstrated. There's no ground truth, no baseline, no confidence intervals. Many sentiment deviations are computed from one or two articles (CBC, Mothership, France24 in t138). The t751 ontology analysis explicitly failed, and the cross-topic analysis is just title sentiment deviations with no significance testing. The stress-test note is on target: the entity-level y-axis in the media bias spectrum uses raw mention counts, not rates. With The Hindu at 10,230 articles and CommonWealth Beacon at 91, raw counts mostly reflect outlet size. Table C10 shows Jerusalem Post at +2,468 and NDTV at +2,422 for \"Israel\" — those are high-output outlets, not necessarily biased ones. Normalizing entity counts by newspaper total output could easily reorder those results.\n\nWho this is for: people working on media bias detection who want a menu of off-the-shelf components and a useful visualization scheme. It is not a validated instrument yet. I would send it to peer review, but with a request for major revision: normalize entity counts, add uncertainty-aware scores, release code and data, and tone down the abstract so it matches what the case studies actually establish.","headline":"Integration-level novelty with a real validation gap: the entity-level spectrum uses raw counts and the abstract overclaims what the case studies actually show.","tokens_in":33720,"tokens_out":2211,"would_cite":false,"duration_ms":23902,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a scalable, comparison-based pipeline that detects media bias in event selection and labeling and word choice by measuring each news outlet's deviation from the cross-outlet average, using topic modeling, sentiment…","keywords":["media bias","large language models","topic modeling","sentiment analysis","ontology learning","event selection bias","labeling and word choice","media bias spectrum"],"falsifier":"Take one newspaper's articles on a single event, compute its sentiment and coverage deviations against two different sets of peer outlets from the same dates, and check whether its bias scores move substantially; if they do, the measured bias is a property of the outlet pool rather than of the reporting.","tokens_in":32735,"feed_emoji":"📰","tokens_out":3875,"duration_ms":37910,"temperature":0.7,"pith_summary":"This paper argues that media bias can be studied without any zero-bias baseline: instead, bias is defined relationally, as a news outlet's deviation from the average behavior of the other outlets covering the same topic in the same period. The authors build a pipeline that groups articles with hierarchical topic modeling, scores sentiment in titles, text bodies, and toward named entities, and extracts article-level ontologies with a large language model. Three case studies on the October 2023 Hamas attack, the 2023 Belt and Road Forum, and the full dataset demonstrate that event selection bias and labeling/word choice bias show up as measurable, visualizable differences across outlets. The paper claims the method succeeds for those two bias forms, while commission and omission analysis through ontologies works in the smaller case study but degrades as topic size grows.","feed_headline":"Media bias measured as deviation from the pack, not truth","feed_subtitle":"Topic, sentiment, and ontology steps rank how outlets over- or under-cover events and slant toward entities.","key_machinery":"The carrying object is the media bias spectrum, a two-axis scatter plot in which the x-axis is an outlet's sentiment deviation from the cross-outlet mean and the y-axis is its coverage-share deviation from the same mean, with point size encoding the underlying article or mention count. This spectrum is fed by a hierarchy of topic clusters from BERTopic, target-dependent sentiment scores from a RoBERTa news model, and GPT-4-generated ontologies with automatic consistency checks. The spectrum makes the paper's relational definition of bias operational: the mean is always zero, and every outlet is plotted as a departure from that zero.","core_discovery":"The central claim is that comparing outlets against the mean of all outlets—rather than against an external truth or ideological label—yields a scalable, minimally biased way to surface media bias. Concretely, the coverage share per outlet per topic, compared with the cross-outlet mean, exposes event selection bias, while the mean sentiment per outlet, also compared with the cross-outlet mean at title, body, and entity levels, exposes labeling and word choice bias. The paper further claims that comparing the objects and relations in each outlet's ontology can expose commission and omission, and that this worked in one small topic but did not hold up when the corpus grew larger.","pith_inferences":["A direct consequence of the relational definition is that an outlet's measured bias depends on the chosen pool of comparison outlets: swap the pool and the same articles can show a different deviation, a corollary the paper does not draw out.","A testable extension would be to merge ontology nodes by embedding similarity before comparing graphs, which would likely let commission and omission analysis scale to the large topics where the paper reports failure.","Adding a time axis to the media bias spectrum could turn the method from a static snapshot into a detector of sentiment shifts around single events, an extension the paper lists as future work."],"forward_implications":["Event selection bias becomes measurable without labels: an outlet that consistently sits above the mean on coverage of a topic is giving that topic disproportionate space relative to its peers.","Labeling and word choice bias is captured by title sentiment and entity-level sentiment, while whole-text sentiment is found to be uninformative in the paper's tests.","The same pipeline works at any level of the topic hierarchy, from a single small event to a cross-topic corpus, without extra adaptation steps.","For commission and omission, the method's usefulness is currently limited by unresolved semantic duplicates in the ontology nodes, so its reliable reach is smaller topics."],"supporting_citations":[{"why":"Supplies the six-step news production framework and the three bias forms the paper targets.","marker":"[1]"},{"why":"Provides the comparative approach of contrasting the same topic across newspapers instead of using a zero-bias baseline.","marker":"[11]"},{"why":"Establishes the source-event comparison strategy for event selection bias that the paper adapts through topic modeling.","marker":"[27]"},{"why":"Supplies BERTopic, the hierarchical topic modeling algorithm that organizes articles and enables multi-level comparisons.","marker":"[40]"},{"why":"Provides the NewsMTSC RoBERTa model used for title and target-dependent sentiment analysis.","marker":"[58]"},{"why":"Supplies the BERT-based named entity recognizer used to extract entities for sentiment and ontology analysis.","marker":"[60]"},{"why":"Provides the GPT-4 model whose prompt-based outputs are the source of the article ontologies.","marker":"[81]"},{"why":"Defines the Digital News Report 2023 used to select the newspapers in the case study dataset.","marker":"[98]"}],"fun_headline_variants":["Bias measured vs pack, but only in small corpora","Media bias: deviation from outlet mean","Bias detection via pack comparison","Outlier from pack spells media bias","Pack-based bias metric fragile on big data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a newspaper's systematic difference from the average of the other included newspapers is media bias, not sampling noise, editorial style, or model error, since the paper explicitly rejects any zero-bias reference point and supplies no external ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Bias measured vs pack, but only in small corpora","Media bias: deviation from outlet mean","Bias detection via pack comparison","Outlier from pack spells media bias","Pack-based bias metric fragile on big data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2747,"prompt_tokens":799,"completion_tokens":1948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":1882}},"tokens_in":415,"tokens_out":1948,"duration_ms":15984,"temperature":1.0,"reasoning_tokens":1882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:11:04.711425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one newspaper's articles on a single event, compute its sentiment and coverage deviations against two different sets of peer outlets from the same dates, and check whether its bias scores move substantially; if they do, the measured bias is a property of the outlet pool rather than of the reporting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the comparative approach of contrasting the same topic across newspapers instead of using a zero-bias baseline."},{"cited_title":"Selection Bias in News Coverage: Learning it, Fighting it","cited_arxiv_id":"1904.07536","evidence_quote":"Establishes the source-event comparison strategy for event selection bias that the paper adapts through topic modeling."},{"cited_title":"Technical report (2023)","cited_arxiv_id":null,"evidence_quote":"Defines the Digital News Report 2023 used to select the newspapers in the case study dataset."}],"review_version":1}