{"id":"2f9e5f90-8a61-4841-8ace-c51ba4805a5e","arxiv_id":"2501.00004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Homepage layouts encode editorial prioritization, and pairwise models trained on article size and position can rank news and policy documents by predicted newsworthiness.","lead":"Researchers created NewsHomepages, a dataset of more than 363,000 snapshots of 3,000+ news homepages collected over three years, and trained models that learn which stories editors consider important from the size and position of articles on the page. The models were then used to rank news articles and San Francisco city policies to test whether layout cues reveal \"newsworthiness\" that generalizes beyond a single outlet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transferability of editorial-preference models across outlets is never directly validated; cross-outlet pairwise evaluation is missing.","rationale":"The reader identified transferability as the weakest assumption, and I agree. My stress-test sharpens this into a concrete, testable omission: the paper never checks whether a model trained on one outlet can predict another outlet's layout-based preference labels, even though those labels are available in the dataset. This is a stronger concern than merely lacking a gold truth for newsworthiness, because a direct cross-outlet pairwise evaluation would not require any external editorial ranking; it would use the same operationalization the paper already relies on. If cross-outlet pairwise accuracy is low, the outlet-agreement results in Section 5.1 and the policy-ranking demonstration in Section 5.2 lose their grounding. The dataset contribution and the within-outlet modeling results remain valuable, which is why I do not move the verdict to REJECT. The conditional verdict is appropriate, but it should be explicitly tied to the need for this cross-outlet validation.","tokens_in":14058,"tokens_out":5469,"duration_ms":61773,"concrete_test":"For every pair of outlets (A, B) among the 31 analyzed, evaluate model trained on A on B's held-out test pairs (ground truth = B's layout-derived preference) and compute pairwise accuracy/F1. If cross-outlet accuracy is near chance (e.g., ≤55%) for most pairs, the transfer assumption in Section 6 fails; if it is significantly above chance, the transfer claim is supported. Also repeat for size-only and position-only models separately to determine which cue transfers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6 that editorial priorities can be inferred from homepage layouts and transferred to other domains rests on an untested assumption. Section 5.1 computes Kendall's tau between rankings produced by different outlets' models on a shared article list, but this only measures agreement between models, not whether a model trained on outlet A can predict outlet B's actual layout-based preferences. The dataset contains per-outlet pairwise labels, so cross-outlet pairwise accuracy can be measured directly: apply A's model to B's held-out pairs and compare against B's layout-derived ground truth. The paper does not report this. Section 5.2 applies models to San Francisco policies with no gold truth; the journalist survey is an impression metric, not a ranking-quality metric. Without a direct transfer test, the outlet-agreement analysis could reflect shared text cues (e.g., sensational language or article length) that happen to correlate with layout across outlets rather than the target outlet's priorities, and the policy ranking could be arbitrary. This is load-bearing because both demonstrations and the central claim depend on transfer across outlets and domains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NewsHomepages, a corpus of 3,489 news homepage snapshots collected twice daily over three years, together with a bootstrapped layout-parsing pipeline that trains a Detectron2 bounding-box detector from DOM-derived pseudo-labels. It then trains per-outlet DistilBERT pairwise classifiers to predict relative article prominence from article text, using size-based, position-based, and combined size-and-position labels derived from the layouts, and reports held-out F1 scores for 31 outlets. Two downstream demonstrations are presented: a cross-outlet rank-correlation analysis of newsworthiness judgments, and an application of the models to rank San Francisco Board of Supervisors policies. The paper argues that homepage layouts capture editorial prioritization decisions and that these learned preferences transfer across outlets and into non-news domains. It releases code and data links for the dataset and pipeline.","tokens_in":14259,"tokens_out":4171,"duration_ms":42939,"significance":"If the transferability claims were validated, this would be a genuinely useful contribution: a large, public homepage-layout dataset; a practical deterministic-to-learned parsing bootstrapping procedure; and evidence that textual cues can predict layout-derived prominence within outlets. The dataset release, the pipeline description, and the per-outlet pairwise benchmark are concrete, reproducible assets. The central scientific claim, however, is stronger than the evidence: the paper's own Section 6 concedes that transfer cannot be conclusively measured without a gold standard, yet the two demonstrations are presented as findings. The value of the resource is not in doubt, but the interpretive weight placed on the transfer experiments currently exceeds what the experiments support.","major_comments":[{"comment":"The central transferability claim is not directly tested. Section 5.1 computes Kendall's tau between rankings produced by different outlets' models on shared article lists, which measures agreement between models, not whether a model trained on outlet A can predict outlet B's actual layout-derived pairwise preferences. The dataset contains per-outlet held-out pairwise labels, so a direct cross-outlet evaluation is feasible: train on A, apply to B's held-out pairs, and measure pairwise accuracy/F1 against B's layout-derived ground truth. This missing evaluation is load-bearing because the Section 6 claim that 'editorial priorities and decision-making can be inferred simply by examining the layout decisions made on homepages' and both downstream demonstrations depend on transfer across outlets and domains.","section":"§5.1 and §6"},{"comment":"Task 2 lacks any ground-truth ranking or baseline. The top-10 policy lists are summarized by an LLM, and the 81% journalist-survey figure reports impressions of a tool, not quality of the ranking. No comparison is made to chronological, random, or topic-similarity baselines, and the top-10 cutoff is arbitrary. Since this task is one of the two demonstrations used to support domain transfer, the manuscript should either provide a ranking-quality evaluation (even a small human annotation of policy newsworthiness) or explicitly reframe the result as an uncontrolled case study.","section":"§5.2"},{"comment":"No confidence intervals or significance tests are reported for the per-outlet pairwise classifiers or for the Kendall's tau agreement matrix. This matters because several outlets perform near chance (arstechnica F1=17.5, breitbartnews F1=22.8, foxnews F1=38.6, wsj F1=48.0), and the Figure 2 correlation matrix is built from rankings produced by these noisy models. A specific conclusion such as the Breitbart/Mother Jones agreement should be accompanied by bootstrapped intervals or a permutation test to establish that it is not an artifact of model noise.","section":"Table 4 and Figure 2"}],"minor_comments":[{"comment":"The first bullet contains a typo: 'hompage' should be 'homepage'.","section":"Contributions bullet"},{"comment":"The notation for the article lists is confusing: '{a1, a2, ..., a1000}9 j=1' and '{o}9 j=1' should be written with explicit subscripts and index ranges so that the outer loop over the nine outlets is clear.","section":"§5.1"},{"comment":"The outlet selection criterion 'matching on variables such as topic, length, publication date, and other potential confounders' is not described in enough detail to assess whether the Task 1 agreement analysis is confounded by these variables; please specify the matching procedure and the number of matched articles per outlet.","section":"§5.1 and §5.2"},{"comment":"The definition of the combined size-and-position preference is incomplete: 'particularly focusing on articles that are in the top 10' is a sentence fragment, and the manuscript does not state what 'top 10' refers to or how ties between the size and position criteria are broken.","section":"§4.1"},{"comment":"Figure 1 lists graphics and font as newsworthiness signals, but the pairwise models in Section 4 use only text as input and size/position as labels; the paper should clarify that graphics and font are not modeled in the reported experiments.","section":"Figure 1 and §4.1"},{"comment":"Model names are inconsistent: 'Flan-t5-base' and 'flan-t5-base' are both used, and 'Roberta' is sometimes written with lowercase 'b'; please standardize the names.","section":"Table 3"},{"comment":"The sentence 'in Table 8 we show' is incomplete; it should be finished with the missing object (presumably the outlet identifiers or a description of the dataset breakdown).","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript sits between a dataset/resource paper and a claim about transferable editorial-preference models. The resource itself appears valuable, but the framing should be adjusted to match the evidence. A direct cross-outlet pairwise evaluation and a grounded evaluation of the policy-ranking task would substantially strengthen the paper; without them, the Section 6 conclusions should be presented as hypotheses rather than demonstrated findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead NewsHomepages. The main thing to know: the dataset is the contribution. 363k homepage snapshots across 3k outlets, captured twice daily over three years, with HTML, screenshots, links, and Internet Archive storage. That is a real public resource for anyone studying editorial gatekeeping, media bias, or layout dynamics. The bootstrapped DOM-tree plus Detectron2 bounding-box pipeline is also solid engineering: on the challenge set the trained detector is far better than the deterministic baseline (90% vs 61% card-correct), and on clean pages both are above 96%. They ship code and data, which counts.\n\nThe pairwise preference modeling is a reasonable operationalization. Size and position as proxies for importance are standard in the literature, not novel, but applying it at this scale and showing that distilbert can predict held-out layout choices above 60% F1 for most of 31 outlets is a legitimate result. Breitbart and Fox being low is itself informative, though the authors don't dig into why.\n\nNow soft spots, in order of importance. The transfer claims in Section 5 are the weakest part. The stress-test note is right: cross-outlet transfer is never directly evaluated. Kendall's tau between two outlet models on a third outlet's articles measures agreement between models, not whether A's model predicts B's layout decisions. The data has per-outlet pairwise labels, so a direct test was available and should be run. Without it, \"editorial priorities can be inferred simply by examining layout decisions\" (Section 6) is too strong. The authors do explicitly concede in Section 6 that there is no gold truth for arbitrary lists, and the journalist survey in Task 2 is an impression metric, not a validation of the ranking. So the central claim is not proven; it is a reasonable hypothesis with encouraging but incomplete evidence. I would not call it a fatal flaw because the paper frames the demonstrations as case studies and the dataset stands on its own.\n\nMinor issues: no confidence intervals on the F1 numbers, the pairwise-to-global ranking procedure for sorting 1,000 items is unspecified, and the policy task has no baseline beyond \"impressed journalists.\" Also, the labels are layout-derived, so calling the outputs \"newsworthiness\" is a semantic stretch. That is not circular--the model is not trained on its own outputs--but it is an interpretive leap.\n\nWho is this for? Anyone building news datasets or studying editorial prioritization. It deserves a serious referee. I would send it out, and ask for the cross-outlet transfer experiment and a clearer statement that newsworthiness is an inference from layout, not ground truth.\n\nRecommendation: send to peer review, conditional accept.","headline":"A genuinely useful public dataset and a sensible pairwise model of layout prominence; the transfer demonstrations are exploratory and should be framed as such.","tokens_in":14776,"tokens_out":1716,"would_cite":true,"duration_ms":16200,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Editorial priorities on news homepages can be inferred from layout alone, and the learned signal transfers across outlets and to non-news documents.","keywords":["homepage layouts","information prioritization","newsworthiness","editorial decision-making","pairwise preference models","layout parsing","computational journalism","news dataset"],"falsifier":"Compare the model's inferred ranking against an explicit priority ranking provided by an outlet's editors for the same set of articles from that outlet's homepage; if the ranked lists disagree substantially, for instance Kendall's $\\tau$ near zero on a held-out sample where editors state their true order, the claim that layout reveals editorial priorities would be falsified. The paper itself notes that no such gold truth currently exists.","tokens_in":13864,"feed_emoji":"📰","tokens_out":10263,"duration_ms":80077,"temperature":0.7,"pith_summary":"This paper argues that the layout of a news homepage—where an article appears, how large it is, and whether it carries images—encodes the editorial judgment of how newsworthy that article is. To support this, it introduces NewsHomepages, a dataset of over 3,000 news homepages captured twice daily for three years, and trains pairwise text classifiers that predict which of two articles an editor would rank higher based on size and position. The paper shows these models transfer: a model trained on one outlet can sort another outlet's articles, and can rank non-news documents such as San Francisco city council policies by their newsworthiness. If the argument holds, homepage layouts become a large-scale, continuously updated source of editorial priority labels for journalism tools and for studying how news organizations allocate attention.","feed_headline":"Homepage layouts expose the newsroom's priority list","feed_subtitle":"A dataset of over 3,000 homepages plus text models recovers editorial newsworthiness and ranks city policies.","key_machinery":"The machinery is a pipeline that converts homepage screenshots into pairwise training signals. First, a deterministic DOM-Tree algorithm extracts candidate article bounding boxes from the HTML; a Detectron2 object detector (ResNet-101 backbone with a feature pyramid network) is then trained on the filtered output to robustly find article cards in the 363,000 snapshots. Second, each article is reduced to a text sequence, and a distilbert-base-uncased Transformer is trained to classify pairs $(a_1, a_2)$ as preferring $a_1$ over $a_2$ according to three criteria: size, position, and the combination size $\\times$ position. The key move is the pairwise formulation, which converts an intractable global placement prediction into a binary classification that captures the site's implicit ranking, with adjacent-article sampling keeping pairs challenging and topically similar.","core_discovery":"The central discovery is that editorial priorities and decision-making can be inferred simply by examining the layout decisions made on homepages. Working from the visual cues editors manipulate—position (top-left is more prominent), surface area (larger is more important), and the presence of graphics—the paper casts homepage construction as a pairwise preference problem: for any two articles on a homepage, a model predicts which one the editor treated as more significant. A compact Transformer (distilbert-base-uncased) trained on these pairwise labels recovers the preference signal with respectable accuracy, most outlets scoring above $F_1 > 0.6$. Two demonstrations show the signal is not merely topical: a right-leaning outlet and a left-leaning outlet, Breitbart and Mother Jones, exhibit among the highest newsworthiness agreement despite being topically distant, and models trained on homepages can rank a decade of San Francisco policies to surface newsworthy leads, an application 81% of consulted journalists said they would consider using.","pith_inferences":["If this transferability holds beyond news, curated interfaces such as social media feeds, streaming homepages, and app stores could be read the same way: their layouts are implicit priority statements that models could decode.","The Breitbart–Mother Jones agreement suggests a testable hypothesis that editorial priority norms are shared across the professional field more than ideology; other ideologically opposed pairs could be checked with the released models.","Because the dataset spans 18 languages but validation covers only English-language U.S. outlets, a natural extension is to test cross-lingual transfer; failure would indicate that layout-based newsworthiness is culturally specific.","A practical extension is a real-time newsworthiness score for arbitrary text, such as monitoring city council minutes, regulatory filings, or press releases, by applying the pairwise model to each incoming document."],"forward_implications":["If layout encodes editorial priorities, then every archived homepage becomes a training label, allowing newsworthiness models to be built at scale without manual annotation.","Newsworthiness judgments emerge as a measurable variable orthogonal to topical similarity, opening the way to study how outlets differ in priority even when they cover the same stories.","The pairwise preference formulation can be reused to rank any corpus, such as government documents, press releases, or social media posts, by implicit news value, supporting journalistic lead-finding.","Accuracy on held-out homepage pairs indicates that editorial layout policies are predictable enough to model, so the same approach could be extended to track how priorities shift over time or across sections."],"supporting_citations":[{"why":"Supplies eye-tracking evidence that top-left page areas draw the most attention, grounding the positional preference signal.","marker":"Nielsen and Pernice, 2009"},{"why":"Establishes the print-design principle that larger space signals story importance, grounding the size preference.","marker":"García, 1987"},{"why":"Relates the presence of news factors to upfront position and story length, connecting layout to newsworthiness.","marker":"Boukes et al., 2022"},{"why":"Provides the self-training/bootstrapping methodology used to generate clean training data for the layout parser.","marker":"Amini et al., 2022"},{"why":"Supplies Detectron2, the object-detection model trained to find article bounding boxes in homepage screenshots.","marker":"Wu et al., 2019"},{"why":"Provides SBERT sentence embeddings used as the baseline to show newsworthiness differs from topical similarity.","marker":"Reimers and Gurevych, 2019"},{"why":"Contributes the San Francisco Board of Supervisors policy corpus and the newsworthiness-detection task used in the second demonstration.","marker":"Spangher et al., 2023a"},{"why":"Prior evidence that right- and left-leaning outlets can show surprising overlaps, motivating the outlet-agreement analysis.","marker":"Gentzkow and Shapiro, 2010"},{"why":"Articulates that a page presents a cohesive bundle of information, motivating pairwise comparison rather than single-article prediction.","marker":"Tufte, 1990"}],"fun_headline_variants":["Homepage layouts expose editorial priorities","Layout reveals the newsroom's must-read list","Pairwise layout model decodes newsworthiness","From homepages to policy: inferring importance","Dataset shows layout encodes news significance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the editorial priority pattern learned from one outlet's homepage transfers to other domains, whether another outlet's articles or non-news texts like city policies, and the paper concedes that without a gold-standard ranking from editors this transfer cannot be conclusively measured.","fun_headline_variants_meta":{"raw":{"variants":["Homepage layouts expose editorial priorities","Layout reveals the newsroom's must-read list","Pairwise layout model decodes newsworthiness","From homepages to policy: inferring importance","Dataset shows layout encodes news significance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2427,"prompt_tokens":862,"completion_tokens":1565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1500}},"tokens_in":478,"tokens_out":1565,"duration_ms":11325,"temperature":1.0,"reasoning_tokens":1500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:54:14.586924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the model's inferred ranking against an explicit priority ranking provided by an outlet's editors for the same set of articles from that outlet's homepage; if the ranked lists disagree substantially, for instance Kendall's $\\tau$ near zero on a held-out sample where editors state their true order, the claim that layout reveals editorial priorities would be falsified. The paper itself notes that no such gold truth currently exists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies eye-tracking evidence that top-left page areas draw the most attention, grounding the positional preference signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Relates the presence of news factors to upfront position and story length, connecting layout to newsworthiness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evidence that right- and left-leaning outlets can show surprising overlaps, motivating the outlet-agreement analysis."}],"review_version":1}