{"id":"32257c9a-979a-4e33-a848-0815c53ca071","arxiv_id":"2507.01170","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using embeddings and Wikidata linking, the authors find a small geographic shift in the Nordisk familjebok's entries between its first and second editions, away from Europe and toward the rest of the world.","lead":"This paper builds a pipeline to segment, classify, and match entries between the first and second editions of a historical Swedish encyclopedia, then links geographic entries to Wikidata. The authors report a shift in geographic focus away from Europe toward North America, Africa, Asia, Australia, and northern Scandinavia between the editions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The geographic shift may be an artifact of geographically biased Wikidata linking, not an editorial change; the paper's own Section 5.5 acknowledges only about half of locations linked within 25 km.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the entire distributional comparison depends on linking errors being geographically neutral, and the paper provides only an unsupported plausibility statement in Section 5.5. The effect size is described as 'small,' while the second-edition recall is low enough (0.40 within 25 km) that the missing 60% could dominate the signal. This is not an external-consensus disagreement; it is a correctness risk internal to the pipeline. The paper has genuine independent value in its released code and data, and its matching/classification results are plausible, so the appropriate verdict remains conditional: the historical interpretation should not be accepted until linking bias is quantified and, if necessary, corrected. My read does not change the reader's verdict; it reinforces it. The concrete test I propose is one check that would settle whether the concern lands: a stratified manual comparison of linked versus unlinked entries, or an inverse-propensity reweighting of the continent distribution.","tokens_in":9454,"tokens_out":2421,"duration_ms":33999,"concrete_test":"Take a stratified random sample of 200 location-classified entries per edition that failed Wikidata linking, manually determine their continent from the entry text, and compare the continent distribution of failed links to that of successful links. Additionally compute the reported continent-shift after re-weighting each linked entry by the inverse linking rate per continent (or by per-continent failure rate). If failed links are enriched in North America, Africa, Asia, Australia, or northern Scandinavia in the second edition, the reported shift is not identifiable from the current pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a distributional shift in geographic focus between editions. That claim rests on the assumption, stated in Section 5.5, that errors and missing links in Wikidata linking are roughly geographically unbiased: 'it seems reasonable to assume that the overall distribution of locations remains roughly the same.' Table 3 shows this assumption is load-bearing and unverified: second-edition Wikidata linking has recall of only 0.16 for exact QID match and 0.40 for within-25-km match. If the 60% of unlinked or incorrectly linked second-edition entries are not uniformly distributed across continents, the observed proportional decline of Europe and rise of North America, Africa, Asia, Australia, and northern Scandinavia could be produced entirely by differential linking success. The query design also plausibly creates such bias: only the first five Wikidata API results are considered, Swedish Wikipedia text is preferred for matching, and the q-to-k spelling reform mentioned in Section 5.5 means some older Swedish toponyms do not resolve, while non-European places with sparser Swedish Wikipedia coverage are less likely to be retrieved at all. The paper does not compare the continent distribution of linked versus unlinked location entries, so the assumption is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a computational pipeline for comparing the first two editions of the Swedish encyclopedia Nordisk familjebok (1876–1899 and 1904–1926). The authors scrape OCR text from Project Runeberg, segment it into entries, filter out cross-references, classify entries as locations using a KB-BERT-based classifier, match location entries across the two editions with sentence embeddings and a cosine threshold, and link them to Wikidata items with coordinates. The resulting linked location sets are then used to compare continental distributions between editions. The paper reports a 'small but significant shift' in geographic focus away from Europe and toward North America, Africa, Asia, Australia, and northern Scandinavia, which they interpret as reflecting the influence of the First World War and global industrialization. The pipeline is evaluated on validation sets of 25–200 entries, with the weakest component being Wikidata linking (within-25-km F1 of 0.69 for the first edition and 0.54 for the second).","tokens_in":9696,"tokens_out":2955,"duration_ms":34564,"significance":"If the claimed geographic shift is real, this would be a valuable quantitative contribution to digital humanities and historical encyclopedia research, and the released code and data would enable replication and extension. The paper's main strength is the transparent description of a full pipeline, from raw OCR to interpretable geographic trends, with component-wise evaluation and public GitHub resources. However, the central interpretive claim rests on an unverified and, on its face, questionable assumption about the geographical neutrality of Wikidata linking errors. The paper explicitly acknowledges its small evaluation sets and the poor linking quality but still uses the word 'significant' without any statistical test. Because the headline result depends on the distribution of linked locations, and because that distribution could be systematically distorted by differential linking success across continents, the conclusion is not currently supported by the evidence presented. The paper is more convincing as a description of a reusable pipeline than as a demonstration of a historical shift.","major_comments":[{"comment":"The central claim of a geographic shift between editions depends on the assumption, stated in Section 5.5, that Wikidata linking errors are approximately uniformly distributed geographically: 'it seems reasonable to assume that the overall distribution of locations remains roughly the same.' Table 3 shows that only 40–64% of locations are linked within 25 km of the correct coordinates, so a majority of links are either missing or wrong at kilometer scale. If unlinked or incorrectly linked entries are not uniformly distributed across continents, the observed proportional decline of Europe and rise of other regions could be entirely an artifact of differential linking success. The paper does not compare the continent distribution of successfully linked versus unlinked/incorrectly linked entries, nor does it provide any sensitivity analysis. This assumption is load-bearing, and the current text provides no evidence for it.","section":"Section 5.5 and Table 3"},{"comment":"The abstract and Section 6.2 use the word 'significant' in a statistical sense ('a small but significant shift'), but the paper reports no confidence intervals, hypothesis tests, or effect-size measures. The location counts per continent are derived from a pipeline with substantial component errors (e.g., location-classifier F1 ≈ 0.90, entry-matching F1 ≈ 0.83, Wikidata linking within-25-km F1 between 0.54 and 0.69). To support the word 'significant', the authors should compare the two editions with a formal test on the continent proportions, or at least provide error bars that propagate the estimated component uncertainties. Without this, the statement overreaches the evidence.","section":"Abstract and Section 6.2"},{"comment":"The matching threshold (cosine 0.9) and the Wikidata-linking threshold (cosine 0.6) are fitted on validation sets of 200 and 25 entries per edition, respectively. The matched and linked sets that feed the geographic analysis are therefore not independent of these threshold choices. The paper does not assess how sensitive the continental distributions in Figure 2 are to plausible variations of these thresholds. For example, raising or lowering the linking threshold by 0.05 could change which of the first five Wikidata candidates is accepted, and that could change the continent counts disproportionately for sparsely covered regions. A robustness analysis varying both thresholds and reporting the resulting shift would be needed to show that the observed trend is not an artifact of threshold fitting on small sets.","section":"Sections 4.2 and 4.3.2"},{"comment":"The evaluation of entry matching, which is a prerequisite for the removed/added-entry analysis, is based on a validation set used to select the threshold, and the reported F1 of 0.83 is close to the headword baseline of 0.76. The paper states that the improvement is 'not significant,' and no statistical test is provided even there. Given that the pipeline's downstream geographic analysis depends on the set of matched location entries, the small evaluation sets and the lack of significance testing undermine the precision of any claim about which entries were 'added' or 'removed' between editions. The authors acknowledge this in the Limitations section, but the abstract does not reflect this uncertainty.","section":"Table 3 and Section 5.4"}],"minor_comments":[{"comment":"The relative Levenshtein threshold of 0.15 was set through 'manual testing' without describing the test set or the criterion used; please state how many examples were examined and what error rate was deemed acceptable.","section":"Section 3.2.2"},{"comment":"The location classifier is trained on 200 manually annotated entries, but the annotation procedure (e.g., who annotated, whether disagreements were resolved, inter-annotator agreement) is not described; this would help readers judge the quality of the ground truth.","section":"Section 4.1"},{"comment":"The table footnotes are inconsistent: the 'Entry classifier' row reports a validation set of 4 samples, which is too small to be meaningful, and the row is marked with '*' for both editions without explaining that the same model was evaluated only on first-edition data.","section":"Table 3"},{"comment":"The caption says 'percentage unit increase' but the figure shows only a list of countries without numerical values; define the metric and provide the underlying numbers, otherwise the reader cannot assess the magnitude of the change.","section":"Figure 2c"},{"comment":"The sentence 'albeit not significantly' appears without a supporting statistical test; if the authors intend a statistical claim, they should report the test, and if not, the phrase should be removed.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"This paper is a workshop-style pipeline description with a broader historical claim attached. The pipeline engineering and public release of code/data are commendable, but the headline result ('significant shift') is overstated relative to the evidence. The authors could potentially fix the issue with a targeted additional analysis: compare the continental distribution of linked versus unlinked entries, provide confidence intervals or a permutation test, and add a threshold sensitivity analysis. If these additions are made, the paper could become publishable. As it stands, the load-bearing assumption in Section 5.5 is untested and the word 'significant' is unjustified. I would also suggest the authors consider lowering the strength of the historical interpretation until the linking bias is ruled out."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a useful artifact: the authors release a pipeline and dataset for segmenting, matching, and linking entries from two editions of Nordisk familjebok, and the first-edition segmentation is genuinely new. Second, the headline historical claim is not established. The abstract says 'small but significant shift' but there is no significance test anywhere, and the Wikidata linking step, which produces the geographic distributions, has within-25-km recall of 0.40 for the second edition. The paper's own Section 5.5 admits only about half the locations are linked correctly at that tolerance, then says 'it seems reasonable to assume that the overall distribution of locations remains roughly the same.' That assumption is load-bearing and unverified; if retrieval works better for European places, the observed shift away from Europe and toward North America, Africa, Asia, Australia, and northern Scandinavia could be an artifact.\n\nWhat the paper does well: the pipeline is reproducible, code and data are on GitHub, and the evaluation is transparent even when the numbers are modest. Entry matching beats a headword baseline. The authors are honest about small validation sets (25-100 entries) and about model transfer from modern Swedish. The segmentation and cross-reference handling are sensible and clearly described.\n\nSoft spots, in proportion: the small evaluation sets are a real limitation but not fatal; the paper acknowledges them. The threshold fitting to validation data is a mild circularity, common in this kind of work. The bigger problem is linking bias. There is no comparison of linked versus unlinked entries by continent, which would be the natural check. The search design (first five API results, Swedish Wikipedia text preference, q-to-k spelling issues) plausibly biases against non-European places. Without a confidence interval or a bias analysis, the 'significant' in the abstract is not justified. The causal interpretation (World War I and 'rise of new powers') reads as confirmation, but the data only shows a raw distributional shift that may be partly a measurement artifact.\n\nWho this is for: people working on historical encyclopedia digitization, cross-edition linking, or Wikidata linking for historical Swedish. They will find the dataset useful. The historical claim should be treated as a hypothesis, not a result.\n\nMy recommendation: send it to peer review, but expect major revision. Ask for a significance test or bootstrap CI on the shift, a linked-versus-unlinked geographic comparison, and a toned-down abstract. The paper deserves referee time because the artifact is valuable and the flaw is fixable.","headline":"Useful, honest pipeline paper whose headline geographic claim lacks significance testing and rests on an unverified assumption of unbiased Wikidata linking.","tokens_in":10226,"tokens_out":2779,"would_cite":true,"duration_ms":26877,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Between its first edition (1876–1899) and second (1904–1926), Nordisk familjebok's geographic entries shifted measurably away from Europe.","keywords":["Nordisk familjebok","encyclopedia entries","entry segmentation","semantic embeddings","Wikidata linking","geographic distribution","historical Swedish text","edition matching"],"falsifier":"Recompute the continent distribution using only the entries that the paper's own 25-kilometer accuracy metric accepts; if the shift away from Europe disappears or reverses in that high-confidence subset, the reported shift is an artifact of geographically biased linking rather than an editorial change.","tokens_in":9229,"feed_emoji":"🌍","tokens_out":12840,"duration_ms":234662,"temperature":0.7,"pith_summary":"The paper asks whether the geographic focus of a nation's most influential encyclopedia changed across editions, and it answers with a measured yes. Using digitized text of Nordisk familjebok's first two editions, the authors segment the raw text into individual entries, classify which entries are locations, match corresponding entries between editions, and link each location entry to coordinates in the Wikidata knowledge graph. They find a small but statistically significant shift in the geographic distribution of entries away from Europe and toward North America, Africa, Asia, Australia, and northern Scandinavia between the 1876–1899 and 1904–1926 editions. This matters because it converts an anecdotal historical narrative — that the First World War and new global powers reshaped Swedish intellectual attention — into a reproducible, quantitative observation.","feed_headline":"A Swedish encyclopedia's world map shifts away from Europe","feed_subtitle":"North America, Africa, Asia, Australia, and northern Scandinavia gain share between the two editions.","key_machinery":"The argument is carried by a four-stage linking pipeline. Entry boundaries are recovered from the digitized OCR text by a three-step segmenter: paragraphs opening with a bold tag, paragraphs whose start matches an index headword with a length-relative Levenshtein distance below 0.15, and otherwise a CLD3-style binary classifier. A location classifier built from the [CLS] hidden states of a Swedish BERT model (KB-BERT) with a logistic head selects geographic entries. The selected entries are embedded with a Swedish sentence-transformer model (KB-SBERT) and matched across editions by greedy nearest-neighbor search in a vector database with a cosine threshold of 0.9. For each entry, the pipeline queries Wikidata, a collaborative knowledge graph, encodes the entry and the top five candidate texts, and keeps the candidate above a cosine threshold of 0.6, whose coordinate property (P625) supplies the latitude and longitude. The continent and country distribution of these coordinates between editions is what the geographic-shift claim is read off.","core_discovery":"This paper's central discovery is that the second edition of Nordisk familjebok (1904–1926) contains a measurably more global geographic entry distribution than the first edition (1876–1899). By classifying roughly 22% of entries as locations, matching the two editions' entries, and linking the locations to Wikidata coordinates, the authors observe a shift in the continental distribution of entries: Europe's share decreases while North America, Africa, Asia, Australia, and northern Scandinavia increase. The paper interprets this shift as confirming the influence of the First World War and the rise of new powers on Swedish reference knowledge, and it notes that because the second edition has more location entries overall, a relative decline for European countries does not require an absolute decline in their coverage.","pith_inferences":["Editorial inference: the paper does not break the shift down by volume within the long second edition, so the same data could date the change more precisely if the shift is concentrated in volumes written during or after the First World War.","Editorial inference: a testable extension would replace continent-level counts with per-article first-mention analysis, which could separate the addition of new non-European articles from the rewriting of existing ones.","Editorial inference: if Wikidata coverage is better for European places, the observed direction of bias would tend to inflate the shift, so measuring linking success on a manually verified stratified sample would show whether the claim survives."],"forward_implications":["The second edition of Nordisk familjebok has a measurably more global distribution of geographic entries than the first, with Europe's share falling and North America, Africa, Asia, Australia, and northern Scandinavia rising.","The matched-entry dataset can support cross-edition search systems, so a query could return content from both editions at once rather than from a single volume.","Because the second edition contains more locations overall, a relative decline in Europe's share does not imply fewer European entries in absolute terms.","The public code and data allow the same linking pipeline to be applied to other historical encyclopedias or to later editions of Nordisk familjebok."],"supporting_citations":[{"why":"Supplies the location-classifier method and the earlier extraction of the second edition whose results this paper extends and compares.","marker":"Ahlin et al. (2024)"},{"why":"Provides the digitized OCR corpus of both editions that all entries and coordinates are derived from.","marker":"Aronsson (2023)"},{"why":"Provides the Swedish BERT model whose hidden states feed the location classifier.","marker":"Malmsten et al. (2020)"},{"why":"Provides the Swedish sentence-transformer model used to embed entries for matching and Wikidata linking.","marker":"Rekathati (2023)"},{"why":"Provides the Sentence-BERT architecture that the sentence-embedding model is based on.","marker":"Reimers and Gurevych (2019)"},{"why":"Provides the edit-distance metric used to match index headwords to entry paragraphs during segmentation.","marker":"Levenshtein (1966)"},{"why":"Provides the BERT pretraining underlying the encoders used in the pipeline.","marker":"Devlin et al. (2019)"}],"fun_headline_variants":["Swedish encyclopedia's geographic focus drifts away from Europe","Encyclopedia's second edition shifts world view toward new powers","Geography of Nordic encyclopedia expands beyond Europe","Swedish encyclopedia tracks post-WWI geographic shift","Europe loses share in Swedish encyclopedia's new edition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The distributional conclusion rests on the assumption that Wikidata linking succeeds or fails about as often for European and non-European places, so the measured shift is not a by-product of easier linking for some continents.","fun_headline_variants_meta":{"raw":{"variants":["Swedish encyclopedia's geographic focus drifts away from Europe","Encyclopedia's second edition shifts world view toward new powers","Geography of Nordic encyclopedia expands beyond Europe","Swedish encyclopedia tracks post-WWI geographic shift","Europe loses share in Swedish encyclopedia's new edition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2744,"prompt_tokens":962,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1709}},"tokens_in":578,"tokens_out":1782,"duration_ms":15206,"temperature":1.0,"reasoning_tokens":1709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:57:52.705887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the continent distribution using only the entries that the paper's own 25-kilometer accuracy metric accepts; if the shift away from Europe disappears or reverses in that high-confidence subset, the reported shift is an artifact of geographically biased linking rather than an editorial change.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the location-classifier method and the earlier extraction of the second edition whose results this paper extends and compares."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the digitized OCR corpus of both editions that all entries and coordinates are derived from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Swedish sentence-transformer model used to embed entries for matching and Wikidata linking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the edit-distance metric used to match index headwords to entry paragraphs during segmentation."}],"review_version":1}