{"id":"c2887082-8a34-4de7-abf5-93913b96cca0","arxiv_id":"2505.02352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Bias patterns in Wikidata occupation link prediction cluster 21 countries into Global North and Global South groups across four embedding methods.","lead":"This paper audits how often AI models trained on Wikidata correctly predict people's occupations by gender and age, and reports that the bias patterns differ by country in a way that mirrors the global economic divide. The authors call this a transparent separation of the Global North from the Global South, and they observe it across four different embedding algorithms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unnormalized occupation-count vectors in Sec. 6.2 let Wikidata coverage volume, not bias composition, drive the Global North/South clustering.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing weakness: the unnormalized geography-level sum in Sec. 6.2. My independent reading of the paper confirms that this is the point where the central claim is least secure. The paper's own text notes that large-coverage geographies contain occupations in almost all categories, and the clustering input is a raw sum of indicator vectors, so magnitude effects are a direct and plausible alternative explanation for the Global North/South split. The proposed normalization and total-count control would settle whether the partition reflects bias composition or coverage. Since the reader already assigned CONDITIONAL for this reason, my verdict recommendation is UNCHANGED: no change to the conditional verdict is needed, but the concern remains the key condition to be addressed by the authors. The contradictory statements about data release (Section 1 says all code/data are available; Section 7 says release upon acceptance) are an additional reproducibility issue but are secondary to the statistical-confounding concern.","tokens_in":16677,"tokens_out":3205,"duration_ms":36948,"concrete_test":"Re-run the Sec. 6.2 pipeline with normalized geography-level vectors: divide each of the five counts by the total number of occupations for that geography (or use row-wise L2 normalization or per-category z-scores), then repeat the hierarchical spectral clustering for each of TransE, DistMult, CompGCN, and GeKC. As a control, cluster the 21 geographies using only the scalar total occupation count per geography as the feature. If the normalized vectors no longer separate into Global North/South clusters, or if the total-count-only control reproduces the original partition, the headline claim is an artifact of Wikidata coverage volume rather than of bias composition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the geography-level representation in Sec. 6.2. Each occupation contributes a 5-dimensional indicator of which bias conditions (eqs. 6-10) it satisfies, and these vectors are summed over occupations to get a geography-level count vector. This unnormalized sum conflates two sources of variation: the qualitative mix of biased vs. neutral occupations and the sheer number of occupations Wikidata contains for that geography. Heavily curated countries, mostly in the Global North, have more entities and occupations in every category; the paper itself observes that larger-geography datasets contain occupations in almost all categories while smaller geographies such as Arabia and Israel do not. Spectral clustering on raw count vectors is dominated by vector magnitude, so a high-count cluster vs. low-count cluster split will track a rough Global North/South division even if every country had identical per-occupation bias probabilities. The Table 3 economic indicators do not remove the confound because GDP, HDI, and Wikidata coverage are correlated. Until normalization or a coverage control is provided, the headline partition is not established as reflecting bias composition rather than data volume.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AuditLP, a framework that measures gender and age bias in link prediction over Wikidata triples for 21 geographies using four knowledge-graph embedding models (TransE, DistMult, CompGCN, GeKC). Occupations are labeled male/female-biased, young/old-biased, or neutral from TPR/FPR differences, and each geography is represented by a five-dimensional count vector summing these labels across occupations. Spectral clustering of these vectors is claimed to reproduce a Global North / Global South partition, supported by comparisons with country-level economic and cultural indicators and by lists of occupations with opposite bias between clusters. The paper also presents a new geography-specific Wikidata dataset and states that code and data are available.","tokens_in":16903,"tokens_out":4278,"duration_ms":55975,"significance":"If the central claim holds, the paper would make a notable contribution: it scales bias auditing of knowledge-graph link prediction to 21 geographies, shows consistency across four quite different embedding models, and connects algorithmic bias patterns to macro-level socioeconomic divisions. The authors' effort to validate clusters with external country attributes is a positive feature, as is the stated intent to release code and data. However, the load-bearing geographical representation is an unnormalized count vector, so the headline Global North / Global South separation is not currently established as reflecting bias composition rather than Wikidata coverage volume. The result is potentially interesting, but it needs substantial additional analysis before it can be accepted.","major_comments":[{"comment":"The geography-level representation is defined as the sum of occupation-level five-dimensional indicator vectors over all occupations in that geography. Because the entries are counts rather than proportions, geographies with more occupations have larger vector magnitudes in every category, and spectral clustering on raw count vectors is dominated by this magnitude difference. A high-coverage versus low-coverage split can therefore look like a Global North / Global South partition even if per-occupation bias probabilities were identical across countries. The paper itself notes in Section 6.1 that larger-geography datasets contain occupations in almost all categories while smaller geographies such as Arabia and Israel do not. The authors should normalize the geography vectors (for example, by the number of occupations or by L2 norm) and/or include a coverage-matched control, and then show that the GN/GS clustering still emerges. The country attributes in Table 3 do not resolve this confound, since GDP, HDI, and Wikidata curation volume are correlated.","section":"6.2"},{"comment":"The thresholds t1 and t2 used to label occupations as biased or neutral are derived from the mean and standard deviation of the same TPR/FPR differences that are subsequently used to produce the labels, and the neutrality cutoff of 0.01 is fixed. This means the number of biased occupations is determined by the variance of the data by construction, and the labels are not accompanied by significance tests or confidence intervals. Please report the actual threshold values for each geography and model, and provide a sensitivity analysis over these thresholds; otherwise the occupation labels, and hence the geography vectors that drive the clustering, are not shown to be robust.","section":"5, Eqs. (6)-(10)"},{"comment":"The claimed agreement with the Global North / Global South partition is not quantified, and several clusters do not align cleanly with that partition. For example, in the TransE gender result, C1 includes Argentina, Russia, and Israel while C2 includes Mexico, New Zealand, and South Korea; in the CompGCN gender result, C1 includes Brazil and C2 includes several countries usually classified as Global North. The statement that clusters are 'broadly consistent' appears to be a post-hoc reading. Please report a quantitative agreement measure (e.g., adjusted Rand index or cluster purity) against a reference GN/GS labeling, and compare with a null model using permuted labels or count-only features.","section":"6.2, Table 2"},{"comment":"The quantitative evidence consists of cluster-level averages of six country attributes, but no within-cluster variance, standard errors, or statistical tests are provided. The text uses phrases such as 'significantly different' without supporting inference. At minimum, report the per-country values or provide confidence intervals and a test (e.g., a permutation test) for the difference between the GN and GS clusters on each attribute.","section":"6.2, Table 3"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'DistMuslt' should read 'DistMult'.","section":"1"},{"comment":"The framework is described as a 'noble framework'; 'novel' is the intended word in this context.","section":"1"},{"comment":"The data availability statement is inconsistent: Section 1 says all code and data are made available, while Section 7 says the full dataset and code will be released upon acceptance. Please clarify which is the case.","section":"1 and 7"},{"comment":"Figure 1 is referenced as showing entity and occupation counts, but the counts are not discussed numerically in the text and the figure is not included in the manuscript text; please add a clear description and ensure the figure is readable.","section":"3, Figure 1"},{"comment":"The edge-hiding step uses a single random 50% split while maintaining the male-female ratio, but no random seeds or multiple runs are reported. Some indication of variance across splits would help establish that the classification results are stable.","section":"5"},{"comment":"In the age rows of Table 1, the text appears to duplicate TPR_yn and FPR_ol instead of listing TPR_yn, TPR_ol, FPR_yn, and FPR_ol; please correct the notation.","section":"6.1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core confound identified in the stress-test note is real and central: the unnormalized count vectors in Section 6.2 can produce a coverage-based partition that mimics the Global North / Global South split. The revision should focus on normalization and null-model controls. I would also recommend asking the authors to tone down the 'first in the literature' claim, since the paper does not provide a systematic comparison with prior bias-audit work in this space."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the scope: 21 geographies, four structurally different embedding families, and a hidden-edge link prediction setup with fairness metrics applied per occupation. That is real work, and a region-tagged dataset of this size could be useful to the field. The paper is also honest enough to note that rich countries have occupations in almost all categories while smaller ones don't. I believe them. The problem is that their own observation sinks the headline result.\n\nThe load-bearing step is Section 6.2. Each occupation gets a 5-dimensional indicator vector for which bias conditions it satisfies, and then these are summed per geography. That unnormalized sum conflates two things: the qualitative mix of biased versus neutral occupations, and the sheer number of occupations Wikidata has for that country. Spectral clustering on raw counts is dominated by vector magnitude. So the resulting clusters — the US, France, Germany on one side, Arabia, Nigeria, India on the other — may simply be \"lots of Wikidata coverage\" versus \"less coverage.\" The Table 3 economic indicators do not remove the confound because GDP, HDI, and Wikidata curation effort are correlated. I'm not saying there is no real bias geography in Wikidata; I'm saying this analysis does not establish it.\n\nThere are smaller issues. The thresholds t1 and t2 are computed as mean minus/plus sigma of the very same TPR/FPR differences they are used to label. That is fitting the threshold to the data without significance testing or confidence intervals. The 0.01 neutrality cutoff is arbitrary. And the availability statement is contradictory: the abstract and footnote say all code and data are available, while the conclusion says they will release upon acceptance. That needs cleanup.\n\nCredit where it's due: the experimental pipeline is carefully described, the choice of four KGE families is sensible, and the idea of validating clusters against external country attributes is good, even if the validation is undercut by the confound. This is not a desk-reject paper; it deserves a serious referee. But the central claim is not established until the geography vectors are normalized — say, by proportion of biased occupations or with a per-occupation control — and until the clustering is shown to survive on a coverage-matched sample. I would send it back for major revision, not accept it as is. For my own work I would not cite the headline finding yet, though I might cite the dataset if it becomes available and is better documented.","headline":"A valuable 21-geography bias audit undermined by an unnormalized aggregation step that likely makes the Global North/South split an artifact of Wikidata coverage volume rather than bias composition.","tokens_in":17401,"tokens_out":2058,"would_cite":false,"duration_ms":26350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bias patterns in Wikidata occupation predictions separate the Global North from the Global South.","keywords":["knowledge graph","link prediction","Wikidata","social bias","fairness","gender bias","age bias","Global North–Global South divide"],"falsifier":"Recompute each geography's five-dimensional vector after dividing every entry by the country's total number of occupations (or number of human entities), then re-run the spectral clustering. If the clean Global North/Global South split does not survive this normalization, the reported partition is an artifact of data volume and curation effort rather than of qualitative bias differences.","tokens_in":16502,"feed_emoji":"🌐","tokens_out":5996,"duration_ms":63909,"temperature":0.7,"pith_summary":"This paper claims that social biases in link prediction on Wikidata are not distributed randomly across the world: when occupations are labelled as male- or female-biased (and young- or old-biased) by standard fairness metrics, the per-country pattern of labels clusters into two groups that closely track the familiar economic and cultural divide between the Global North and the Global South. The authors build a framework, AuditLP, that hides a person's occupation from the graph, learns entity embeddings with four different knowledge-graph embedding algorithms, and uses a classifier to predict the hidden occupation links. The framework then classifies each occupation in each of 21 geographies into one of three bias categories for gender and for age. Aggregating these labels into a five-dimensional per-geography vector and running spectral clustering yields the North–South split, and the split appears for all four algorithms. If the claim holds, it means that the structural bias of a widely used open knowledge graph carries a fingerprint of global socio-economic divisions into downstream AI systems.","feed_headline":"Bias patterns in Wikidata split Global North from Global South","feed_subtitle":"The same two-way split appears under four different embedding models, so it is structural, not an algorithm quirk.","key_machinery":"The machinery has three linked parts. First, AuditLP: hide the occupation links of half the humans (stratified by gender), train a knowledge-graph embedding model, and feed head–relation–tail concatenations of the learned vectors into an MLP classifier to score whether each hidden link exists. Second, the labelling rule: using equal-opportunity and equalized-odds fairness criteria, each occupation is assigned to one of three categories according to differences between male and female (or young and old) true-positive and false-positive rates, with the threshold taken as the mean minus the standard deviation of the rate differences. Third, the geographic signature: each of the 21 countries is represented by a five-dimensional vector whose entries count how many of its occupations fall into the bias categories; these vectors are then clustered by hierarchical spectral clustering, and the resulting clusters are compared with global socio-economic indicators. The decisive move is that the same two-cluster separation emerges from four structurally different embedding models, which the paper takes as evidence that the pattern is a property of the data and task, not of a single learning algorithm.","core_discovery":"The central discovery is that the bias profile of Wikidata's knowledge representations contains a global geographic signal: the vector of how many occupations fall into each of the five bias categories reproduces the Global North/Global South partition. The paper demonstrates this for two sensitive attributes, gender and age, across 21 geographies and four embedding models, and checks the clusters against country-level attributes such as GDP per capita, Human Development Index, Gender Gap Index, Gini coefficient, individualism, and cultural distance from the United States. It further shows that some occupations are oppositely biased in the two blocs: intellectual and white-collar occupations tend to be male-biased in the Global North and female-biased in the Global South, while physically demanding sports occupations show the reverse pattern. The paper interprets this as evidence that the bias in the knowledge graph and its embeddings is not merely algorithmic noise but mirrors real socio-economic differences in who gets recorded and how occupations are gendered.","pith_inferences":["A direct test of the authors' interpretation would be to replace the raw five-dimensional count vectors with normalized vectors (for example, dividing by the number of occupations in each geography); if the clean North–South split does not survive normalization, the partition may be driven by data volume and curation effort rather than by qualitative differences in bias.","If the pattern generalizes beyond gender and age to race, ethnicity, or religion, then debiasing methods for knowledge-graph completion may need region-specific objectives rather than a single global fairness constraint.","Adding more geographies not in the current set, such as China, Southeast Asia, or Central Asia, would stress-test the universality of the claimed partition: the claim would gain force if those countries fall into the predicted clusters and lose force if they do not.","The authors' aggregate-level results show only slight overall differences between male and female true-positive rates, which suggests that an analysis that stops at global averages would miss the strong, opposite, and geographically structured biases the paper finds."],"forward_implications":["Because the split appears for all four embedding model families, any of these families used for knowledge-graph completion will propagate a geographically structured gender or age bias into downstream applications such as question answering and language-model pretraining.","Occupations labelled male-biased in the Global North can be female-biased in the Global South and vice versa, so fairness interventions trained on one region may be wrong for the other.","Country-level indicators such as GDP per capita, Human Development Index, Gender Gap Index, Gini coefficient, individualism, and cultural distance correlate with the clusters, so the bias pattern can serve as a proxy signal of a country's position in the global economic order.","The framework labels occupations as age-biased as well as gender-biased for every geography, and the age-based clusters also follow the North–South division.","Aggregate metrics that average over all countries can obscure the fact that individual countries are biased in opposite directions, so future bias audits should report geography-disaggregated results."],"supporting_citations":[{"why":"Provides the TransE translational embedding model, one of the four knowledge-graph embedding families whose learned representations feed the bias audit.","marker":"[8]"},{"why":"Provides the DistMult semantic-matching embedding model used as a second, structurally different embedding family in the audit.","marker":"[53]"},{"why":"Provides the CompGCN convolutional knowledge-graph embedding model, representing the graph-neural-network family of learned representations.","marker":"[42]"},{"why":"Provides GeKC, the generative-circuits knowledge-graph embedding model, representing the newest family of embedding learners tested.","marker":"[27]"},{"why":"Prior work by the same group showing that diversity of measurements affects bias estimates in Wikidata, motivating the geography-disaggregated audit design.","marker":"[14]"},{"why":"Supplies the American cultural distance measure used as a country-level attribute to validate that the detected clusters align with cultural and economic divisions.","marker":"[29]"},{"why":"Defines the Global North/Global South socio-economic partition that the paper uses as the reference classification for interpreting the spectral clusters.","marker":"[34]"},{"why":"Provides labor-force participation evidence used to support the paper's interpretation of opposite gender biases across the two blocs.","marker":"[35]"}],"fun_headline_variants":["Wikidata bias profile redraws global North-South line","Wikidata's bias map draws a clear North-South divide","Wikidata's social biases mirror global inequality","Knowledge graph bias encodes a North-South world split","Bias in Wikidata embeddings reveals a two-bloc world"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding up per-occupation bias indicators within each country measures qualitative differences in bias, rather than merely reflecting how many occupations and people Wikidata happens to contain for richer, more heavily edited countries.","fun_headline_variants_meta":{"raw":{"variants":["Wikidata bias profile redraws global North-South line","Wikidata's bias map draws a clear North-South divide","Wikidata's social biases mirror global inequality","Knowledge graph bias encodes a North-South world split","Bias in Wikidata embeddings reveals a two-bloc world"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000925,"raw_usage":{"total_tokens":3970,"prompt_tokens":954,"completion_tokens":3016,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2937}},"tokens_in":570,"tokens_out":3016,"duration_ms":24879,"temperature":1.0,"reasoning_tokens":2937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:54:29.004561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute each geography's five-dimensional vector after dividing every entry by the country's total number of occupations (or number of human entities), then re-run the spectral clustering. If the clean Global North/Global South split does not survive this normalization, the reported partition is an artifact of data volume and curation effort rather than of qualitative bias differences.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DistMult semantic-matching embedding model used as a second, structurally different embedding family in the audit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GeKC, the generative-circuits knowledge-graph embedding model, representing the newest family of embedding learners tested."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work by the same group showing that diversity of measurements affects bias estimates in Wikidata, motivating the geography-disaggregated audit design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the American cultural distance measure used as a country-level attribute to validate that the detected clusters align with cultural and economic divisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Global North/Global South socio-economic partition that the paper uses as the reference classification for interpreting the spectral clusters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides labor-force participation evidence used to support the paper's interpretation of opposite gender biases across the two blocs."}],"review_version":1}