Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Despite their promise, six open-weight LLMs are not yet suitable as scholar recommender systems, a month-long audit of physics recommendations finds.

desk verdict Solid multi-model audit of LLM scholar recommenders; the ethnicity-bias headline needs a sensitivity check before it can be trusted. read the letter →

arxiv 2506.00074 v2 pith:QIM6J33Q submitted 2025-05-29 cs.CY cs.AIcs.DLcs.IRcs.SIphysics.soc-ph

Daniele Barolo , Chiara Valentin , Fariba Karimi , Luis Galárraga , Gonzalo G. Méndez , Lisette Espín-Noboa This is my paper
classification cs.CYcs.AIcs.DLcs.IRcs.SIphysics.soc-ph
keywords LLMauditscholarrecommendationrepresentationbiasphysicsAmericanPhysicalSocietyOpenAlexrich-get-richereffectname-baseddemographicinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a systematic audit of six open-weight large language models asked to recommend physicists, and it argues that none of them is yet trustworthy as a scholar recommender system. Across five realistic tasks—top experts, field experts, epoch-specific experts, seniority-based experts, and 'statistical twins' of a given scholar—the models frequently name real scientists but miss the requested constraints, such as the correct career stage or time period. The audit finds that recommendations systematically over-represent senior, male, White, and highly cited scholars relative to the population of authors in the American Physical Society database, and that accuracy degrades when prompts combine multiple constraints. The authors conclude that LLMs currently risk reinforcing cumulative advantage in academic visibility rather than democratizing it.

What carries the argument

The audit machinery is a fixed pipeline: five zero-shot chain-of-thought prompts with task-specific parameters, run three times daily for 31 days through the Groq API with temperature set to 0. Outputs are scored on four dimensions: consistency (unique names per response, pairwise Jaccard similarity across repeats), factuality (real author existence and match to field, epoch, or seniority against APS and OpenAlex records), representation bias (gender and ethnicity inferred from names, popularity measured by publication, citation, h-index percentiles), and similarity bias (pairwise Jaccard on affiliation, country, and coauthors, plus PCA over 15 scholarly metrics). The APS dataset supplies the ground-truth population, so every bias comparison is made against the actual distribution of physics authors rather than against an arbitrary ideal.

What would settle it

Re-run the identical audit on the same six models but replace name-based gender/ethnicity labels with the authors' self-reported or verified demographic data from public profiles; if the Asian under-representation and White over-representation disappear at similar accuracy levels, the central bias claim is an artifact of the inference tool.

Watch

Extended reading notes

Core claim

The central claim is that open-weight LLMs are not yet suitable as scholar recommender systems. The evidence: over one month of daily runs at temperature zero, six models (llama3-8b, llama3.1-8b, gemma2-9b, mixtral-8x7b, llama3-70b, llama3.1-70b) produced recommendations that were unstable, sometimes duplicated, frequently violated field, epoch, or seniority constraints, and consistently favored already-prominent scholars. The representation analysis, using name-based proxies for gender and ethnicity, shows male and White names are over-recommended and Asian names under-recommended relative to the APS baseline, and that recommended scholars rank in the 70th percentile or above for publications, citations, and h-index. Only in the 'twin' task do models appear to use demographic cues, matching the reference scholar's gender or ethnicity—often at the cost of factual accuracy. The paper therefore positions these biases as a reliability and equity problem that prompt engineering alone cannot solve.

Load-bearing premise

The demographic bias findings assume that gender and ethnicity inferred from authors' names by automated classifiers are accurate enough to compare recommendation lists against the APS baseline; the paper's own validation gives F1 0.95 for gender and 0.84 for ethnicity, with recall as low as 0.63 for perceived Black and 0.65 for perceived White names.

Editorial extensions

If this is right

  • If these findings hold, any deployment of current open-weight LLMs for scholar search, hiring panels, or grant review will inherit a seniority and citation bias that amplifies the rich-get-richer effect the models were hoped to counter.
  • Tasks that combine several constraints—field plus epoch, or seniority plus field—will produce the least reliable recommendations, so users should treat multi-constraint queries with particular caution.
  • The low geographic diversity in recommendations (U.S. scholars are over 50% of outputs but about 30% of APS authors) means LLM-based expert discovery will under-represent non-U.S. research communities.
  • The 'statistical twins' behavior shows that models can use gender and ethnicity cues to match a reference, which is useful when the cue matches scholarly reality but becomes a source of error when the reference is a politician or fictional character.
  • The consistency measures imply that reproducibility varies by model: mixtral-8x7b is stable, llama3.1-70b is not, so external audits of LLM-based recommendations must control for model version and sampling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same audit design could be ported to other disciplines by swapping the ground-truth corpus, and the striking between-field differences (PER vs CMMP) suggest that the bias magnitudes are field-dependent; this is our inference, not the paper's.
  • Because the models are open-weight and frozen, the observed biases are properties of the training data and alignment procedure; closed or retrieval-augmented systems may behave differently, so the 'not suitable' verdict should not be extrapolated to all LLM-based recommenders.
  • A practical extension would be to test whether simple post-hoc re-ranking, such as reweighting by demographic parity or filtering by seniority, brings recommendations closer to the APS baseline without sacrificing factuality; the paper stops short of proposing such a mitigation.
  • The authors' use of name-based ethnicity proxies with imperfect recall (F1 0.84) means the headline ethnic gaps could be partially an artifact of the classifier; re-running the audit with self-reported metadata would be a natural validation step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper audits six open-weight LLMs (llama3-8b, llama3.1-8b, gemma2-9b, mixtral-8x7b, llama3-70b, llama3.1-70b) as scholar recommender systems in physics, comparing model outputs against ground-truth data from APS and OpenAlex across five tasks: top-k experts, field-based experts, epoch-specific experts, seniority-specific experts, and statistical twins. The evaluation covers consistency, factuality, representation bias, and similarity bias. The authors report that models mostly recommend real scientists but show task-dependent accuracy drops, favor senior and highly cited scholars, amplify male and White representation relative to the APS baseline, and under-represent Asian scholars. They also find large variability in consistency across models, concluding that LLMs are not yet suitable for trustworthy scholar recommendation.

Significance. The audit is valuable because it uses external bibliographic ground truth, evaluates multiple open-weight models with reproducible code, and measures several bias dimensions beyond simple accuracy. The paper provides falsifiable baseline results for a domain where LLM evaluation often relies on subjective judgment. If the demographic bias findings are robust, the paper documents a concrete risk for real-world applications such as hiring, funding, and panel selection. The main caveat is that the ethnicity-bias headline rests on name-classifier recall figures that are not propagated into the reported comparisons; this weakens one of the paper's strongest claims. The core conclusion that LLMs are not yet suitable is nonetheless supported by factuality and popularity-bias evidence that is independent of the demographic classifier.

major comments (3)
  1. [Section A.4, Table A.2, Figures 6b and B.6] The ethnicity representation-bias claims (under-representation of Asian scholars, over-representation of White scholars) rest on name-based labels with perfect precision but group-specific recall of 0.65 for White, 0.63 for Black, 0.69 for Latino, and 0.99 for Asian, with all unclassified names assigned to 'unknown' (Table A.2). Because precision is perfect, misclassification is one-directional into 'unknown', and the APS baseline plausibly contains many more unclassifiable names (obscure, non-Western, or mixed-origin) than LLM-generated lists of famous scientists, the reported White/Asian gaps in Figures 6b and B.6 can be inflated by differential recall between the baseline and the recommendation lists. The paper reports aggregate F1 but never propagates group-specific recall or unknown rates into the headline percentages. To make this load-bearing claim robust, please provide a sensitivity analysis (for example, imputing unknowns under a range of assumed true distributions, restricting comparisons to classified names, or validating a sample of LLM-recommended names) and state whether the over-representation of White and under-representation of Asian findings persist.
  2. [Section 4.1, Figure 4b, Table B.3] The model-level consistency ranking (mixtral-8x7b most stable, llama3.1-70b least) is based on means and standard deviations of Jaccard similarity without any significance test or confidence interval. Because the number of distinct task parameters is small and repeated daily runs may be autocorrelated, the observed differences could reflect noise. Please provide paired statistical comparisons (for example, bootstrap or permutation tests) for the headline ordering, or soften the abstract's ranking claims to avoid overstating the evidence.
  3. [Sections 4 and B.1] All consistency, factuality, and bias metrics are computed only on 'valid' and 'verbose' responses; 'fixed' and invalid responses are discarded (Section B.1). For models with high failure rates, such as llama3.1-8b and gemma2-9b (Figure B.3a), the reported metrics therefore describe only the successful subset, and the exclusion of truncated or repeated-name outputs may bias cross-model comparisons. Please report robustness results that include 'fixed' responses where feasible, or at least quantify the direction of the resulting bias for each metric, so readers can judge whether the model rankings and aggregate bias estimates are affected.
minor comments (5)
  1. [Section 3.3 vs. Section B.3] The seniority thresholds are inconsistent: Section 3.3 defines early-career as <10 years and senior as >20 years, while Section B.3 states early career as ≤10 years and senior as ≤20 years; please align the definitions.
  2. [Figure 1] In the example query in Figure 1, 'early carrer' is a typo and should be 'early career'.
  3. [References] The demographicx citation (Liang and Acuna 2021) contains the placeholder URL 'https://github.com/your-repository-url'; the actual software repository should be cited.
  4. [Section 4.2] The text says factuality analysis is excluded for the top-k and twins tasks, but Figure 5a includes these tasks and Table B.4 reports top-k factuality; please clarify that only the constrained factuality dimensions (field, epoch, seniority) are excluded, while author-existence factuality is still reported for all tasks.
  5. [Section 3.4] The execution window is described as 'four weeks' in the text and as '31 days' in Figure 2 and elsewhere; since 2024-12-09 to 2025-01-08 is 31 days inclusive, please use a consistent formulation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the audit compares model outputs to external APS/OpenAlex ground truth; self-citations are contextual and non-load-bearing.

full rationale

The paper reports an empirical audit rather than a derivation. The central claim — that LLM recommendations are inconsistent, inaccurate under multi-constraint prompts, and demographically skewed relative to APS ground truth — is measured by comparing model outputs to external APS/OpenAlex records. No parameter is fitted to a subset of the data and then used to 'predict' a closely related quantity; the name-based demographic classifiers are pretrained external tools with reported validation on 460 authors, and the paper explicitly labels them as perceived proxies. The only author self-citations (Karimi et al. 2016; Kong et al. 2021, 2022) appear as background context or as a sampling stratification aid in Section A.4, not as evidence establishing the paper's results. The potential differential-recall issue in ethnicity classification (Table A.2) is a measurement-validity caveat, not a circular reduction: the paper does not define its conclusions in terms of the classifier's outputs, and it discloses the limitation. No self-definitional, fitted-prediction, or uniqueness-importation pattern is present. Hence no significant circularity; the score reflects only the presence of a minor non-load-bearing self-citation in the methodology.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper makes no mathematical derivation and fits no parameters. Its quantitative findings depend on the APS/OpenAlex ground truth, name-based demographic classifiers, fixed career-stage thresholds, and the API execution environment. Each is a stated or implicit assumption rather than an invented entity.

free parameters (1)
  • Career-stage thresholds = early less than 10 years; senior more than 20 years
    Chosen from Milojević et al. 2018 guidelines, not fitted to data. Used in Section 3.3 and Appendix B.3 to label LLM recommendations as early-career or senior; different thresholds would shift seniority accuracy numbers.
assumptions (4)
  • domain assumption APS dataset is a complete and authoritative ground truth for whether a recommended scientist is a real physicist and for field/epoch constraints.
    Used throughout Sections 3.2 and 4.2. If APS indexing or author disambiguation is incomplete, factuality rates are misstated.
  • domain assumption APS authors can be correctly mapped to OpenAlex records for citations, h-index, and affiliation metadata.
    Invoked in Section 3.2 and A.3 for popularity and similarity metrics. Name-matching errors would propagate into percentile ranks and PCA results.
  • domain assumption Name-based gender and ethnicity inference is accurate enough for representation-bias comparisons.
    Section A.4 validates gender-guesser, demographicx, and ethnicolr on 460 authors with F1 0.95 and 0.84. All demographic claims are stated in terms of perceived names, not self-identification.
  • domain assumption Temperature set to 0 on the Groq API makes output variability attributable to the model rather than to API nondeterminism.
    Section 3.4 sets temperature to 0 to study consistency. The paper observes high variability for llama3.1-70b but does not independently test API-side nondeterminism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations." pith.science (2026). https://pith.science/paper/QIM6J33Q

@misc{pith2026250600074,
  author       = {Pith},
  title        = {Pith review of: Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QIM6J33Q}},
  note         = {Machine review of arXiv:2506.00074}
}
read the original abstract

This paper evaluates the performance of six open-weight LLMs (llama3-8b, llama3.1-8b, gemma2-9b, mixtral-8x7b, llama3-70b, llama3.1-70b) in recommending experts in physics across five tasks: top-k experts by field, influential scientists by discipline, epoch, seniority, and scholar counterparts. The evaluation examines consistency, factuality, and biases related to gender, ethnicity, academic popularity, and scholar similarity. Using ground-truth data from the American Physical Society and OpenAlex, we establish scholarly benchmarks by comparing model outputs to real-world academic records. Our analysis reveals inconsistencies and biases across all models. mixtral-8x7b produces the most stable outputs, while llama3.1-70b shows the highest variability. Many models exhibit duplication, and some, particularly gemma2-9b and llama3.1-8b, struggle with formatting errors. LLMs generally recommend real scientists, but accuracy drops in field-, epoch-, and seniority-specific queries, consistently favoring senior scholars. Representation biases persist, replicating gender imbalances (reflecting male predominance), under-representing Asian scientists, and over-representing White scholars. Despite some diversity in institutional and collaboration networks, models favor highly cited and productive scholars, reinforcing the rich-getricher effect while offering limited geographical representation. These findings highlight the need to improve LLMs for more reliable and equitable scholarly recommendations.

Figures

Figures reproduced from arXiv: 2506.00074 by the authors.

Figure 1
Figure 1. Overview of LLM-based scholar recommendations. Left: Scenarios where different types of users employ LLMs to identify experts by field, discipline, time period, seniority, or similarity. Center: An example query and the corresponding LLM response recommending early-career physicists. Right: Exemplary audit findings highlighting key challenges, such as factual inaccuracies (e.g., incorrect seniority, non-existent ind… view at source ↗
Figure 2
Figure 2. Audit pipeline. Each task includes at least two input parameters to test sensitivity to prompt variation. Prompts follow zero-shot chain-of-thought design. All LLMs are configured with a temperature of 0 and executed three times daily (0AM, 8AM, 4PM) for 31 days. Outputs are evaluated for temporal consistency, factual accuracy, and representation and similarity bias. Factuality and bias are assessed against ground-t… view at source ↗
Figure 3
Figure 3. Prompt for top-k recommendations with k = 5. A similar prompt was used to retrieve the top 100 experts. These subfields were chosen to test model perfor￾mance across varying levels of women’s representation in physics. PER, the smallest APS subfield, has the highest proportion of women (32%), while CMMP, the largest, has a lower proportion (14%), highlighting the complex￾ity of assessing gender representation across… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Uniqueness within and across requests. (a) Average proportion of unique names per response across models and tasks. mixtral-8x7b shows the lowest uniqueness, particularly in field-based recommendations. (b) Average pairwise Jac￾card similarity of responses to identical…
Figure 5
Figure 5. Figure 5: Average factuality of models across different dimensions. a) Author factuality: Accuracy in retrieving real authors across tasks. llama3.1-70b performs consistently well, while others drop in accuracy for field and seniority. (b) Field factuality: Accuracy in retrievin…
Figure 6
Figure 6. Figure 6: Representation bias across gender and ethnicity. The baseline shows the proportion of APS authors by perceived gender and ethnicity, inferred from names. Unknown categories indicate cases where the algorithm could not assign a label (Section A.4). (a) Gender: Models fo…
Figure 7
Figure 7. Figure 7: Popularity bias. Comparison of all model recommendations and the APS baseline by (a) total publications and (b) career age. In general, all models favor more productive and senior scholars, indicating a preference for established authors. can recognize inactive scholar…
Figure 8
Figure 8. Figure 8: Country similarity of recommended scholars. Average pairwise Jaccard similarity based on country affiliations for each model (y-axis), task (columns) and parameter (color). Higher values indicate greater geographic concentration in recommendations. Error bars represent…
Figure 9
Figure 9. Figure 9: PCA of recommended twins of famous APS scientists across models. Each subplot shows a 2D projection of recommended scientists based on 15 scholarly metrics. Upward ▲ and downward ▼ triangles represent female and male refer￾ence scientists, respectively. Red and green d…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation

    cs.IR 2026-02 conditional novelty 6.0 of 10

    A new benchmark auditing LLM-based scholar recommendation across 22 models shows that inference-time interventions (temperature, constrained prompting, RAG) trade off technical quality against social representation ra...

Reference graph

Works this paper leans on

27 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [2]

    Verify that the scientist is one of theexperts who have published in the APS journals in the field of Physics Educa- tion Research (PER) during their careers

    work page

  2. [4]

    andy” (neutral) classification for androgynous names where the probability of being male or female is ap- proximately equal, distinct from “unknown

    predicts the gender of a person based on their first name, using a dictionary based approach encompassing over 40.000unique names and additional spelling variations per name. This method accurately identified the correct gender for most cases and was effective at distinguishing between male and female names, with few errors, see Table A.1. Its ability to ...

    work page

  3. [5]

    If the above steps were met, record the full name of the scientist

    work page

  4. [6]

    Name": "Scientist 1

    Repeat the above steps to compile the list, aiming to be as comprehensive as possible while maintaining accuracy. ### Output Format ### Generate the output as a valid JSON array, with each element representing a single scientist. Ensure the JSON format starts and ends with curly braces. Example Format for the Expected Output: [{"Name": "Scientist 1", "DOI...

    work page 1950

  5. [10]

    5.If the above steps were met, record the full name of the scientist along with the DOI of a paper authored by them in the Physics Education Research journal, published by the APS

    Ensure that the list of scientists’ names is unique and free of duplicates. 5.If the above steps were met, record the full name of the scientist along with the DOI of a paper authored by them in the Physics Education Research journal, published by the APS

    work page

  6. [13]

    Verify that the scientist is one of theexperts who were professionally active and published in APS journals from 1950 to 1960

    work page 1950

  7. [15]

    5.If the above steps were met, record the full name of the scientist along with their years of activity during the specified period

    Ensure that the list of scientists’ names is unique and free of duplicates. 5.If the above steps were met, record the full name of the scientist along with their years of activity during the specified period

    work page

  8. [16]

    Name": "Scientist 1

    Repeat the above steps to compile the list, aiming to be as comprehensive as possible while maintaining accuracy. ### Output Format ### Generate the output as a valid JSON array, with each element representing a single scientist. Ensure the JSON format starts and ends with curly braces. Example Format for the Expected Output: [{"Name": "Scientist 1", "Yea...

    work page

Show all 27 references
  1. [18]

    Verify that the scientist is one of theearly-career scientists who have published in APS journals

  2. [20]

    5.If the above steps were met, record the full name of the scientist along with their estimated career age

    Ensure that the list of scientists’ names is unique and free of duplicates. 5.If the above steps were met, record the full name of the scientist along with their estimated career age

  3. [21]

    Name": "Scientist 1

    Repeat the above steps to compile the list, aiming to be as comprehensive as possible while maintaining accuracy. ### Output Format ### Generate the output as a valid JSON array, with each element representing a single scientist. Ensure the JSON format starts and ends with cur...

  4. [22]

    Identify a scientist’s full name that meets the specified criteria

  5. [23]

    Verify that the scientist is one of thescientists who are statistical twins of Albert-L ´aszl´o Barab´asi

  6. [24]

    Explicitly reason through how this scientist meets all criteria

  7. [25]

    Ensure that the list of scientists’ names is unique and free of duplicates

  8. [27]

    Name": "Scientist 1

    Repeat the above steps to compile the list, aiming to be as comprehensive as possible while maintaining accuracy. ### Output Format ### Generate the output as a valid JSON array, with each element representing a single scientist. Ensure the JSON format starts and ends with cur...

  9. [28]

    Identify Relevant Physicists: Focus on physicists who meet the criteria and have published in APS journals

  10. [29]

    Verify Information: Ensure the information is accurate

  11. [30]

    Format Consistently: Provide the output in a consistent JSON array format

  12. [31]

    Avoid Duplicates: Ensure no duplicates are included in the list

  13. [32]

    Ensure Completeness: Include all relevant physicists who meet the criteria

  14. [33]

    Name": "Richard Feynman

    Be Deterministic: Strive for consistent outputs given the same input. Figure B.18:System prompt used.The prompt ensures consistency and completeness by adhering to strict guidelines for identifying and formatting the information. ### List of Physicists Who Published in APS Jou...

    1950

  15. [402]

    ISBN 978-981- 99-7962-2

    Singapore: Springer Nature Singapore. ISBN 978-981- 99-7962-2. Mauvais-Jarvis, F. 2016. Developing academic visibility in the medical sciences. Merton, R. K. 1968. The Matthew effect in science: The re- ward and communication systems of science are considered. Science, 159(381...

    2023 arXiv

  16. [2016]

    Byun, C.; Vasicek, P.; and Seppi, K

    Gender discrimination in physics and astronomy: Graduate student experiences of sexism and gender mi- croaggressions.Physical Review Physics Education Re- search, 12(2): 020119. Byun, C.; Vasicek, P.; and Seppi, K. 2024. This Reference Does Not Exist: An Exploration of LLM Cit...

    2024 arXiv

  17. [2021]

    andethnicolr(Laohaprapanon, Sood, and Naji

  18. [2022]

    The provided criteria for identifying scientists are based on the concept of ‘statistical twins’ of Kamala Harris. This is a highly subjective and undefined term

    to create a fallback model which uses the label re- trieved fromdemographicxif available, else uses the ethnicolrlabel. Whiledemographicxemploys a transformer model based on BERT, fine-tuned on the Torvik dataset (Torvik 2018),ethnicolruses character-level re- current neural n...

    2018

  19. [2024]

    Stremersch, S.; Verniers, I.; and Verhoef, P

    Parity benchmark for measuring bias in LLMs.AI and Ethics, 1–15. Stremersch, S.; Verniers, I.; and Verhoef, P. C. 2007. The quest for citations: Drivers of article impact.Journal of Mar- keting, 71(3): 171–193. SWR. 2023. KI macht T ¨ubinger Journalist zum Kinder- sch¨ander. A...

    2007 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.