REVIEW 4 major objections 5 minor 43 references
Quantifying Misattribution Unfairness in Authorship Attribution
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Authorship attribution models are unfair, and the risk falls most on authors closest to the center of the embedding space.
desk verdict The paper asks a genuinely important fairness question for authorship attribution, but the metric as written likely counts correct attributions as unfairness and, even on its own terms, lacks a random baseline, so the central 'high unfairness' claim is not supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs through two instruments. The first is the Misattribution Unfairness Index (MAUI$_k$): for each author $a_j$ in a haystack of $N_h$ authors and $N_q$ queries, count $c_j^k$, the number of times that author appears in the top $k$ ranking; subtract the expected count under a random permutation, $E_k=\lceil k N_q / N_h \rceil$; sum the positive excesses; and divide by the maximum possible excess, $k(N_q - E_k)$. This produces a score between 0 and 1, where 0 means top-$k$ appearances match random expectation. The second instrument is geometric: average all haystack author embeddings to obtain a centroid, measure each author's distance from it as $1-\cos(\cdot)$, and correlate that distance with the author's mean rank over all queries. The two instruments carry the argument together: the index shows that the unfairness exists, and the centroid analysis shows which authors are most exposed.
What would settle it
Recompute MAUI$_k$ after removing the true author's embedding from every query's ranked list before counting top-$k$ appearances, then rerun the centroid correlation on the corrected counts. If the excess over the random baseline and the near-centroid pattern largely disappear, the reported unfairness is an artifact of counting correct hits as misattribution; if they persist, the central claim stands.
Extended reading notes
Core claim
The central discovery is that misattribution risk in embed-and-rank authorship attribution is unequal and spatially predictable. Using MAUI$_k$, the paper reports that all five tested models exceed the random baseline for many authors, with the most exposed author ranked in the top 10 up to about 40 times as often as chance. It then shows that an author's mean rank across randomly chosen queries is strongly correlated with the author's distance to the centroid of the haystack's author embeddings: the closer to the center, the higher the average rank, across models and datasets. The paper interprets this as evidence that embedding geometry drives the unfairness, and that accuracy and fairness are separate properties, since the model with the highest ranking performance on one dataset is also among its most unfair.
Load-bearing premise
The load-bearing assumption is that the count of top-$k$ appearances captures only texts the author did not write; the paper never states that the true author is removed from the ranked list before counting, so correct identifications may be counted as unfairness.
Editorial extensions
If this is right
- MAUI$_k$ should be reported alongside recall and MRR, because a model can be highly accurate and still place many innocent authors at far above random risk.
- In forensic or investigative deployments, a top-$k$ hit must be treated as a suspicion-generating event, and its concentration means some authors are repeatedly exposed to scrutiny.
- For a fixed model and dataset, an author's distance to the centroid of the author embedding space is a usable predictor of expected misattribution risk.
- High accuracy does not excuse unfair rankings; the paper's results show the most accurate model on one dataset is also among its most unfair.
- Authors near the centroid can be both more likely to be falsely suspected and not more likely to be correctly found, so the same authors can carry a double burden.
Reading between the lines
- Editorial extension: if the centroid-risk relationship holds outside these datasets, MAUI$_k$ could become a no-annotation model-selection diagnostic: compute author embeddings, locate the centroid, and estimate which authors are most exposed before deployment.
- Editorial extension: the uniform-random baseline is a conservative choice; a more realistic forensic baseline would condition on stylistic or demographic similarity between query and candidate authors, which would shift the expected counts and the MAUI values, as the paper's own limitation section acknowledges.
- Editorial extension: a direct test of the geometric mechanism would be to flatten the embedding distribution (for example by whitening or re-centering author embeddings) and check whether MAUI$_k$ drops toward the random baseline; the paper does not run this experiment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MAUI_k, a normalized measure of how much authors' top-k counts exceed a per-author expected count under a random-permutation baseline, and evaluates it for five embedding-based authorship attribution models on Reddit, Blogs, and Fanfiction. It reports that all models exhibit high misattribution unfairness and that authors closer to the centroid of the embedding space are at higher risk. A second analysis relates true-author MRR to centroid distance using Mann-Whitney tests.
Significance. The paper addresses a genuine gap: standard effectiveness metrics such as R@k and MRR do not capture systematic over-ranking of non-author candidates, and forensic settings make this a plausible harm. If the measurement is corrected, the proposal of a simple top-k unfairness index and the empirical survey of five embedding models on three datasets would be a useful contribution. The centroid-risk relationship, if validated against a proper null model, is a falsifiable and practically relevant prediction. The main weakness is that the numerical claims currently rest on an ambiguous count definition and an absent random baseline.
major comments (4)
- [§2, Eq. (1) and abstract] The definition of c_j^k as "the number of times author a_j is ranked in the top k" does not exclude the true author of the query, even though the abstract defines MAUI over "texts they did not write." This ambiguity is load-bearing: under the literal reading of Eq. (1), self-hits are counted as misattributions, so a high-recall model such as LUAR on Blogs (R@8=0.97) can receive large MAUI values from correct attributions rather than from unfairness. The implementation must be stated explicitly, and all reported MAUI values, including Tables 2, 3, 4, and 6, must be recomputed with self-hits excluded.
- [§2 and §3.2] The metric's zero point is exact count equality, not the random-permutation baseline introduced in Section 2. Under random rankings, each author's count follows Binomial(Nq, k/Nh), so the expected value of MAUI_k is strictly positive for finite Nq. For the paper's configurations, this random-baseline value is approximately 0.13 on Reddit and 0.19 on Blogs at k=10, which is at or above several Table 2 entries (LUAR on Reddit 0.12; Wegmann on Blogs 0.14). Without a null distribution or confidence intervals, the abstract's claim that "all models exhibit high levels of unfairness" is unsupported, and the metric conflates finite-sample noise with systematic unfairness.
- [§3.3] The expected horizontal line for average rank is not a valid null model. Query authors are drawn at random from the same haystack, so query embeddings have the same distribution as author embeddings; an author's expected similarity to a random query therefore depends on the author's position relative to the centroid of that distribution. Central authors are expected to receive higher average ranks even under a system with no systematic misattribution bias. The reported centroid correlation should be compared against a null model that randomizes rankings while preserving the query-author embedding distribution; otherwise the second contribution is, to a significant degree, a consequence of the query distribution rather than a distinct unfairness phenomenon.
- [Limitations section] The Limitations paragraph acknowledges that the query distribution affects the "most fair" baseline and therefore the unfairness measurements, but it does not acknowledge the more basic finite-sample issue that even a random ranking produces a positive expected MAUI. This omission is not merely a caveat; it directly determines whether the headline values in Table 2 can be interpreted as evidence of unfairness.
minor comments (5)
- [Abstract] The abstract says "five models on two different datasets," but Section 3 and Table 2 report three datasets (Reddit, Blogs, Fanfiction).
- [Section 1] The sentence "or Recall at various ranks (R@k), or Recall at various ranks (R@k)" contains a duplicated phrase.
- [Figure 4] Panel (e) of Figure 4 is labeled "MPNetAR" although Figure 4 should include StyleDist.; the label appears to be a copy-paste error.
- [Section 2] The phrase "probability of any specific author being ranked higher than k" should read "ranked in the top k," to match the definition used in Eq. (1).
- [Table 7] The table reports many Mann-Whitney tests without any multiple-comparison correction; a note explaining the number of tests or an adjustment would strengthen the claims.
Circularity Check
No significant circularity: MAUI is defined independently, the centroid analysis is a post-hoc empirical correlation, and the only self-citation is not load-bearing.
full rationale
The paper defines MAUI explicitly in Eq. (1) as a normalized positive deviation of author top-k counts from a random-permutation expectation, and then applies this definition to fixed model outputs; no parameter is fitted to the target conclusion, so the reported unfairness scores are not forced by the metric's construction. The centroid-distance analysis is a separate empirical correlational claim: centroid distances are computed from author embeddings independently of the ranking counts, and the observed rank-to-centroid relationship is not an algebraic identity with the MAUI definition. The sole self-citation, Patel et al. (2024), is used only to identify the StyleDist evaluation model, and no unverified 'uniqueness theorem' or ansatz is imported. Concerns raised by the paper's own Limitations paragraph about random-baseline realism and query-selection dependence, and the possible inclusion of true-author hits in c_k_j, are threats to construct validity and statistical calibration rather than circular reductions.
Assumptions & free parameters
assumptions (3)
- domain assumption The fairness baseline is a random permutation of all haystack authors, with each author's inclusion probability k/Nh.
- domain assumption Query embeddings are drawn from a distribution symmetric enough that an author's expected rank should be independent of their distance to the centroid.
- ad hoc to paper The top-k counts c_k_j include only misattributions, not cases where a_j is the true author.
invented entities (1)
-
MAUI_k (Misattribution Unfairness Index)
Cite this review
Pith. "Pith review of Quantifying Misattribution Unfairness in Authorship Attribution." pith.science (2026). https://pith.science/paper/M42HKBVK
@misc{pith2026250602321,
author = {Pith},
title = {Pith review of: Quantifying Misattribution Unfairness in Authorship Attribution},
year = {2026},
howpublished = {\url{https://pith.science/paper/M42HKBVK}},
note = {Machine review of arXiv:2506.02321}
}
read the original abstract
Authorship misattribution can have profound consequences in real life. In forensic settings simply being considered as one of the potential authors of an evidential piece of text or communication can result in undesirable scrutiny. This raises a fairness question: Is every author in the candidate pool at equal risk of misattribution? Standard evaluation measures for authorship attribution systems do not explicitly account for this notion of fairness. We introduce a simple measure, Misattribution Unfairness Index (MAUIk), which is based on how often authors are ranked in the top k for texts they did not write. Using this measure we quantify the unfairness of five models on two different datasets. All models exhibit high levels of unfairness with increased risks for some authors. Furthermore, we find that this unfairness relates to how the models embed the authors as vectors in the latent search space. In particular, we observe that the risk of misattribution is higher for authors closer to the centroid (or center) of the embedded authors in the haystack. These results indicate the potential for harm and the need for communicating with and calibrating end users on misattribution risk when building and providing such models for downstream use.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Nicholas Andrews and Marcus Bishop. 2019. https://arxiv.org/abs/1910.04979 Learning invariant representations of social media users . Preprint, arXiv:1910.04979
work page Pith review arXiv 2019
-
[4]
Harald Baayen, Hans Van Halteren, Anneke Neijt, and Fiona Tweedie. 2002. An experiment in authorship attribution. In 6th JADT, volume 1, pages 69--75. Citeseer
work page 2002
-
[5]
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, and Sebastian Kohlmeier. 2020. https://arxiv.org/abs/2003.11650 Overview of the trec 2019 fair ranking track . Preprint, arXiv:2003.11650
work page Pith review arXiv 2020
-
[6]
Asia J. Biega, Krishna P. Gummadi, and Gerhard Weikum. 2018. https://api.semanticscholar.org/CorpusID:13697725 Equity of attention: Amortizing individual fairness in rankings . The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
work page 2018
-
[7]
Uitdenbogerd, and Andrew Turpin
Steven Burrows, Alexandra L. Uitdenbogerd, and Andrew Turpin. 2009. Application of information retrieval techniques for source code authorship attribution. In Database Systems for Advanced Applications, pages 699--713, Berlin, Heidelberg. Springer Berlin Heidelberg
work page 2009
-
[8]
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, J. Dhamala, and A. G. Galstyan. 2022. https://api.semanticscholar.org/CorpusID:247762845 On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations . ArXiv, abs/2203.13928
work page Pith review arXiv 2022
Show all 43 references
-
[9]
Carole E Chaski. 2001. Empirical evaluations of language-based author identification techniques. Forensic linguistics, 8:1--65
2001
-
[10]
Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. 2021. https://doi.org/10.1162/tacl_a_00425 Quantifying social biases in NLP : A generalization and empirical comparison of extrinsic fairness metrics . Transactions of the Association for Computational Linguistics, 9:1249--1267
2021 doi
-
[11]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library
2024 arXiv
-
[12]
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daum \'e , Alexandra Olteanu, Emily Sheng, Dan Vann, and Hanna M. Wallach. 2023. https://api.semanticscholar.org/CorpusID:259092939 Fairprism: Evaluating fairness-related harms in text generation . In Annual Meetin...
2023
-
[13]
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097--1179
2024
-
[14]
Ruoyuan Gao and Chirag Shah. 2019. https://api.semanticscholar.org/CorpusID:203592216 How fair can we go: Detecting the boundaries of fairness optimization in information retrieval . Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval
2019
-
[15]
David Garc \'i a-Soriano and Francesco Bonchi. 2021. https://api.semanticscholar.org/CorpusID:235446658 Maxmin-fair ranking: Individual fairness under group-fairness constraints . Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
2021
-
[16]
Baixiang Huang, Canyu Chen, and Kai Shu. 2025. https://arxiv.org/abs/2408.08946 Authorship attribution in the era of llms: Problems, methodologies, and challenges . Preprint, arXiv:2408.08946
2025 arXiv
-
[17]
Mike Kestemont, Enrique Manjavacas, Ilia Markov, Janek Bevendorff, Matti Wiegmann, Efstathios Stamatatos, Martin Potthast, and Benno Stein. 2020. https://api.semanticscholar.org/CorpusID:268091356 Overview of the cross-domain authorship verification task at pan 2020 . In Confe...
2020
-
[18]
Mike Kestemont, Efstathios Stamatatos, Enrique Manjavacas, Walter Daelemans, Martin Potthast, and Benno Stein. 2019. https://api.semanticscholar.org/CorpusID:198489009 Overview of the cross-domain authorship attribution task at pan 2019 . In Conference and Labs of the Evaluation Forum
2019
-
[19]
Aleem Khan, Elizabeth Fleming, Noah Schofield, Marcus Bishop, and Nicholas Andrews. 2021. https://doi.org/10.18653/v1/2021.naacl-main.415 A deep metric learning approach to account linking . In Proceedings of the 2021 Conference of the North American Chapter of the Association...
2021 doi
-
[20]
Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2019 a . https://api.semanticscholar.org/CorpusID:204838020 Does gender matter? towards fairness in dialogue systems . ArXiv, abs/1910.10486
2019 arXiv
-
[21]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692
2019 arXiv
-
[22]
Coulthard M., Johnson A., and Wright D. 2016. https://doi.org/10.4324/9781315630311 An Introduction to Forensic Linguistics: Language in Evidence (2nd ed.) . Routledge
2016 doi
-
[23]
Mann and Douglas R
Henry B. Mann and Douglas R. Whitney. 1947. https://api.semanticscholar.org/CorpusID:14328772 On a test of whether one of two random variables is stochastically larger than the other . Annals of Mathematical Statistics, 18:50--60
1947
-
[24]
Chuizheng Meng, Loc Trinh, Nan Xu, James Enouen, and Yan Liu. 2022. https://api.semanticscholar.org/CorpusID:260505304 Interpretability and fairness evaluation of deep learning models on mimic-iv dataset . Scientific Reports, 12
2022
-
[25]
Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2024. https://arxiv.org/abs/2410.12757 Styledistance: Stronger content-independent style embeddings with synthetic parallel examples . Preprint, arXiv:2410.12757
2024 arXiv
-
[26]
Dhamala, Rahul Gupta, and Kai Wei Chang
Yada Pruksachatkun, Satyapriya Krishna, J. Dhamala, Rahul Gupta, and Kai Wei Chang. 2021. https://api.semanticscholar.org/CorpusID:235489989 Does robustness improve fairness? approaching fairness with word substitution robustness methods for text classification . ArXiv, abs/2106.10826
2021 arXiv
-
[27]
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. 2022. https://api.semanticscholar.org/CorpusID:249062690 Perturbation augmentation for fairer nlp . In Conference on Empirical Methods in Natural Language Processing
2022
-
[28]
Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084
2019 arXiv
-
[29]
Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y
Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.70 Learning universal authorship representations . In Proceedings of the 2021 Conference on Empiric...
2021 doi
-
[30]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108
2019 arXiv
-
[31]
Yunita Sari, Mark Stevenson, and Andreas Vlachos. 2018. https://api.semanticscholar.org/CorpusID:52011663 Topic or style? exploring the most useful features for authorship attribution . In International Conference on Computational Linguistics
2018
-
[32]
Pennebaker
Jonathan Schler, Moshe Koppel, Shlomo Engelson Argamon, and James W. Pennebaker. 2006. https://api.semanticscholar.org/CorpusID:2075411 Effects of age and gender on blogging . In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs
2006
-
[33]
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297
2020 arXiv
-
[34]
Efstathios Stamatatos. 2013. https://api.semanticscholar.org/CorpusID:15288034 On the robustness of authorship attribution based on character n -gram features . Journal of law and policy, 21:7
2013
-
[35]
Urszula Sta \'n czyk and Krzysztof A Cyran. 2007. Machine learning approach to authorship attribution of literary texts. International journal of applied mathematics and informatics, 1(4):151--158
2007
-
[36]
Michael Tschuggnall, Benjamin Murauer, and G \"u nther Specht. 2019. https://doi.org/10.18653/v1/K19-1089 Reduce & attribute: Two-step authorship attribution for large-scale problems . In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pa...
2019 doi
-
[37]
Jacob Tyo, Bhuwan Dhingra, and Zachary C. Lipton. 2022. https://arxiv.org/abs/2209.06869 On the state of the art in authorship attribution and authorship verification . Preprint, arXiv:2209.06869
2022 arXiv
-
[38]
Rivera Soto, Marcus Bishop, and Nicholas Andrews
Andrew Wang, Cristina Aggazzotti, Rebecca Kotula, Rafael A. Rivera Soto, Marcus Bishop, and Nicholas Andrews. 2023. https://api.semanticscholar.org/CorpusID:261065015 Can authorship representation learning capture stylistic features? Transactions of the Association for Computa...
2023
-
[39]
Yuan Wang, Xuyang Wu, Hsin-Tai Wu, Zhiqiang Tao, and Yi Fang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.319 Do large language models rank fairly? an empirical study on the fairness of LLM s as rankers . In Proceedings of the 2024 Conference of the North American Chapte...
2024 doi
-
[40]
Anna Wegmann, Marijn Schraagen, and Dong Nguyen. 2022. https://doi.org/10.18653/v1/2022.repl4nlp-1.26 Same author or just same topic? towards content-independent style representations . In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 249--268, Dubl...
2022 doi
-
[41]
Zichen Wen, Dadi Guo, and Huishuai Zhang. 2024. https://api.semanticscholar.org/CorpusID:274149963 Aidbench: A benchmark for evaluating the authorship identification capability of large language models . ArXiv, abs/2411.13226
2024 arXiv
-
[42]
Ke Yang and Julia Stoyanovich. 2016. https://arxiv.org/abs/1610.08559 Measuring fairness in ranked outputs . Preprint, arXiv:1610.08559
2016 arXiv
-
[43]
Ying Zhao and Justin Zobel. 2007. Searching with style: Authorship attribution in classic literature. In ACM international conference proceeding series, volume 244, pages 59--68. Citeseer
2007
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.