REVIEW 4 major objections 5 minor 38 references
The paper demonstrates that explanation quality in expert-facing AI is perspective-dependent, and that this variation can be captured by LLM-based personas used as reward models, enabling adaptive explanations that experts consistently pref
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Expert explanation preferences can be captured by two LLM-generated personas and used as reinforcement-learning rewards to produce adaptive knowledge-graph explanations that experts prefer over generic ones.
T0 review reviewed 2026-08-03 challenge →
load-bearing objection A real shot at persona-conditioned KG explanations with a meaningful preference result, but the persona validation is partly circular and the paper isn't reproducible as written. the 4 major comments →
Shaping Scientific Explanations to Expert Perspectives with Persona-Conditioned Reinforcement Learning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central discovery is that expert preferences for explanations cluster into coherent, reproducible epistemic stances, and that these stances can be represented as agentic personas—natural-language profiles synthesized by an LLM from clustered expert feedback. When these personas are used as reward functions in reinforcement learning, the resulting explanations align with the corresponding experts' ratings (correlations 0.56–0.91) and are significantly preferred over non-adaptive baselines. The paper's key claim is that this alignment is achievable with only a handful of expert responses, reducing feedback time by two orders of magnitude, without sacrificing predictive accuracy.
What carries the argument
The central object is the agentic persona: a structured, LLM-synthesized narrative describing how an expert evaluates explanations, distilled from clustered, embedded free-text feedback (e.g., Elena favors mechanistic rigor, Leo favors concise clarity). The persona is operationalized as a reward function R_persona = w_v·v + w_c·c + w_r·r over validity, completeness, and relevance, scored by GPT-4o-mini at each RL rollout, with a self-paced curriculum threshold that only invokes persona scoring for high-relevance paths. This turns an epistemic stance into a trainable objective for selecting knowledge-graph reasoning paths.
Load-bearing premise
The load-bearing premise is that clustering 15 free-text responses into two clusters captures stable, generalizable epistemic stances—especially the Leo persona, which was derived from only two responses and then used as the reward model for twelve new participants; if that cluster is noise, the reported preference results reflect the questionnaire rather than genuine epistemic alignment.
What would settle it
Run the same persona-construction pipeline on a new cohort of, say, 50 experts and check whether two clusters with the Elena/Leo characteristics reappear; if the two-response cluster does not reproduce, or if participants assigned to Leo by the three-question instrument show no preference for Leo-conditioned explanations over a randomly named persona in a blind test, the central claim would fail.
If this is right
- Explanation quality is partly perspective-dependent; no single explanation fits all experts.
- Persona-conditioned rewards can train explanation policies with minimal expert feedback (about 1.34 hours vs 250 hours estimated).
- The method matches or exceeds state-of-the-art link prediction (Hits@1, Hits@3, MRR) while adapting explanations.
- The framework extends beyond drug discovery to other expert domains where interpretive diversity matters (legal, educational, climate).
- The three-question profiling instrument can assign new experts to personas without retraining.
Where Pith is reading between the lines
- The three-question assignment instrument is a potential bottleneck: its validity rests on the stability of the k=2 clustering, and a reader might test whether a different set of discrimination questions changes which persona a participant is assigned to.
- The persona approach could be extended to dynamic personas that update as an expert gives more feedback, rather than a fixed assignment.
- If personas are genuinely epistemic stances, then the same method might predict which explanation style improves actual decision-making accuracy, not just preference—a testable extension.
- The large increase in preference for Elena (76%) vs Leo (63.3%) and poor inter-annotator agreement for Leo suggest the Leo persona may be less coherent; a replication with more experts would clarify whether the two-persona taxonomy is stable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes 'agentic personas' as a way to personalize knowledge-graph-based explanations to expert epistemic stances. In Phase I, the authors cluster free-text feedback from 11 experts (15 responses) into two clusters, Elena (n=13) and Leo (n=2), and use an LLM to synthesize persona narratives. In Phase II, they use GPT-4o-mini prompted with these personas as reward models in a reinforcement-learning framework to generate adaptive explanations for drug-repurposing and drug-target hypotheses. They report: (i) persona ratings correlate with expert ratings on the same 40 explanations (r=0.56–0.91, all p<.001); (ii) in a new study (n=22), participants preferred adaptive explanations over the non-adaptive REx baseline 63.3% (p=.005) and 76.0% (p<.001) of the time; (iii) adaptive explanations achieve slightly better Hits@k/MRR than existing methods; and (iv) persona-based feedback is estimated to be ~187x faster than full expert feedback. The paper concludes that explanation quality is perspective-dependent and that personas enable scalable adaptive explainability.
Significance. If the findings hold, the paper makes a useful contribution: it provides a concrete method for turning small amounts of expert feedback into LLM-instantiated reward models and shows a preference effect in a realistic drug-discovery setting. The 22-participant study with significant binomial tests and the explicit comparison against a non-adaptive baseline are strengths, as is the transparent reporting of the Leo sample-size limitation. However, the current evidence is not yet sufficient to support the central claim that persona preferences 'closely align' with expert preferences, because the main alignment result is validated on the same feedback used to construct the personas.
major comments (4)
- [§6.1, Fig. 6] The persona credibility analysis is circular. Section 6.1 compares persona ratings with 'the original assessment by human experts' on the same 40 explanations per task used to construct the personas in Phase I (Section 4.1). Since the expert ratings and comments were used to build the persona narratives and trait frequency bands, the reported correlations (r=0.56–0.91) are in-sample fits and cannot establish that personas generalize to corresponding experts. A held-out validation, such as a second expert cohort or leave-one-expert-out analysis, is needed. The posterior Leo analysis in Section 6.2 is only partially independent: it covers one persona, and participants were assigned using the three-question instrument derived from Leo's own two-person feedback.
- [§5.2, §6.1, Table 2] The Leo persona is derived from only two expert responses out of 15, yet 12 of 22 participants are assigned to it. The paper reports poor inter-annotator agreement for Leo (ICC values negative and 0.03 for the adaptive condition), which is consistent with the construct being unstable. The posterior correlations with the 12 assigned participants (r=0.47–0.56, p<0.05) do not resolve this concern because the assignment instrument was derived from Leo's own feedback, making the match partly by construction. Please report cluster stability metrics (e.g., bootstrap or silhouette values for k=2 vs. alternatives) and validate the assignment instrument independently.
- [Table 3] The abstract and conclusions state that adaptive explanations 'match or exceed state-of-the-art predictive performance,' but Table 3 reports only point estimates for Hits@1, Hits@3, and MRR, with no confidence intervals or significance tests against REx, MINERVA, or PoLo. The improvements (e.g., Hits@1 0.358 vs. REx 0.338) may be within noise. Add measures of uncertainty and, if possible, statistical tests to support the performance claim.
- [§4.1, §6.1] The claim that 'the cohort exhibited optimal clustering at k=2 with remarkable stability across all three algorithms' is not supported by the quantitative clustering metrics described in Section 4.1 (Silhouette Score, Davies–Bouldin Index, Calinski–Harabasz Index, Inertia). None of these values are reported in the main text or supplement summary, despite the very small n=15 and the two-member Leo cluster. Please provide the metric values, especially for k=2 vs. k=3, to justify the clustering decision.
minor comments (5)
- [Declaration of use of Generative AI] The declaration states that the authors 'have not employed any Generative AI tools,' but Section 4.1 explicitly uses OpenAI o3-pro and GPT-4o-mini for persona generation and evaluation. Please correct this inconsistency.
- [§7] The concluding section contains a duplicated paragraph: 'Moreover, adaptive explanations were strongly preferred by experts (63.3–76.0%). Importantly, agentic personas not only mirror expert feedback faithfully...' appears twice verbatim.
- [Fig. 6] The statement 'all p < .001' is too coarse for correlations with n=40 explanations per task; please report the exact correlation coefficients with confidence intervals or at least the degrees of freedom.
- [Table 2] Negative ICC values for the Leo condition (e.g., -1.40) are unusual; please explain how ICC(3,k) can be negative and how readers should interpret such values.
- [§6.2] Please report the exact binomial denominators for the preference percentages (63.3% and 76.0%) so readers can verify the tests.
Circularity Check
Persona credibility is validated on the same expert feedback used to construct the personas; the reported alignment correlations are in-sample fits rather than evidence of generalization.
specific steps
-
fitted input called prediction
[Section 5.1 (Agentic Personas), Section 6.1 (Persona Credibility, Figure 6), Section 7]
"we compared personas’ ratings for the same 40 explanations per task (DR and DTI) with the original assessment by human experts... Persona ratings were compared against aggregated expert scores. As shown in Figure 6, persona ratings correlate significantly with expert ratings (all p < .001, n = 40 explanations per task)."
The personas were constructed from the same expert evaluation study whose ratings are used as ground truth: Phase I used 'curated preference statements, background metadata' from the 11 experts (15 responses, 125 statements) to generate Elena and Leo, and Section 5.1 then validates the personas against 'the original assessment by human experts' on the same 40 explanations. Because the persona narratives are an LLM summary of the very feedback that co-occurred with those ratings, the high correlations (r=0.56–0.91) are an in-sample fit, not evidence that personas generalize to corresponding experts. Reporting this as persona credibility and as evidence that 'LLM-synthesized profiles can capture genuine interpretive patterns' makes the alignment claim circular.
full rationale
The central preference result (n=22 participants preferring adaptive over non-adaptive explanations) and the predictive performance comparison (Table 3 vs MINERVA, PoLo, REx) are independent of persona construction and give the paper substantial non-circular content. However, the abstract's claim that 'persona preferences closely align with those of their corresponding experts' is supported primarily by the Section 6.1 correlation between persona ratings and the aggregated ratings of the very experts used to build the personas. That validation reduces by construction: the persona is a synthesis of the training cohort's feedback, so high correlation with that cohort's aggregate scores is expected. The posterior Leo analysis with 12 new participants is not a clean held-out test either, because those participants were assigned via questions derived from Leo's own feedback. No significant load-bearing self-citation was found; the REx baseline is prior work but is used as a comparator, not as justification for the persona-alignment claim. Score 6 reflects one central validation that is circular while other contributions remain independent.
Axiom & Free-Parameter Ledger
free parameters (4)
- Relevance threshold schedule τ_relevance(t) =
increasing schedule; final value chosen from best validation model
- Low/medium relevance reward constants =
0.25 and 0.10
- Cluster count k =
2
- Persona trait frequency bands =
core >40%, secondary 25–40%, weak <25%
axioms (5)
- domain assumption Expert Likert ratings on relevance, completeness, and validity constitute valid ground truth for explanation quality.
- domain assumption Hetionet adequately represents biomedical knowledge for drug repurposing and drug-target interaction explanations.
- domain assumption GPT-4o-mini scoring with a persona prompt faithfully implements that persona's epistemic preferences.
- ad hoc to paper The k=2 cluster structure of 15 responses reflects stable epistemic stances rather than sampling noise.
- domain assumption Three single-choice self-report questions validly assign new experts to the correct epistemic stance.
invented entities (1)
-
Agentic personas (Elena and Leo)
no independent evidence
Cite this review
Pith. "Pith review of Shaping Scientific Explanations to Expert Perspectives with Persona-Conditioned Reinforcement Learning." pith.science (2026). https://pith.science/paper/ALRP6V6X
@misc{pith2026260321846,
author = {Pith},
title = {Pith review of: Shaping Scientific Explanations to Expert Perspectives with Persona-Conditioned Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ALRP6V6X}},
note = {Machine review of arXiv:2603.21846}
}
read the original abstract
Explainable AI is increasingly important to scientific discovery. However, existing methods largely ignore that explanation quality is not universal: experts differ in how they assess evidence, prioritize mechanisms, and construct explanatory narratives. We introduce perspective-conditioned explanations, a framework for adapting explanation generation to epistemic variation in expert judgment. Using knowledge graph reasoning paths in drug discovery, we show that preferences organize into coherent epistemic perspectives that can be captured by agentic personas, representations of how experts evaluate explanations. Persona-aligned rewards then guide reinforcement learning-based explanation generation without large-scale expert supervision. Expert user studies show that perspective-conditioned explanations are preferred over general-purpose explanations and improve perceived relevance and validity. Moreover, they match or exceed state-of-the-art predictive performance and reduce expert feedback time by two orders of magnitude. Together, these findings demonstrate that explanation quality is perspective-dependent and that modeling this variation enables scalable and human-aligned explanation generation for scientific discovery.
Figures
Reference graph
Works this paper leans on
-
[1]
Artificial Intelligence Review57(9), 244 (2024)
Akujuobi, U., Kumari, P., Choi, J., Badreddine, S., Maruyama, K., Palaniappan, S.K., Besold, T.R.: Link prediction for hypothesis generation: an active curriculum learning infused tem- poral graph-based approach. Artificial Intelligence Review57(9), 244 (2024)
2024
-
[2]
the other
Cabrero, D.G., Winschiers-Theophilus, H., Abdelnour-Nocera, J.: A critique of personas as representations of" the other" in cross-cultural technology design. In: Proceedings of the First African Conference on Human Computer Interaction. pp. 149–154 (2016)
2016
-
[3]
Communications in Statistics-theory and Methods3(1), 1–27 (1974)
Cali ´nski, T., Harabasz, J.: A dendrite method for cluster analysis. Communications in Statistics-theory and Methods3(1), 1–27 (1974)
1974
-
[4]
Advances in neural information processing systems30 (2017)
Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in neural information processing systems30 (2017)
2017
-
[5]
Artificial intelligence298, 103503 (2021)
Conati, C., Barral, O., Putnam, V ., Rieger, L.: Toward personalized xai: A case study in intelligent tutoring systems. Artificial intelligence298, 103503 (2021)
2021
-
[6]
arXiv preprint arXiv:1711.05851 (2017)
Das, R., Dhuliawala, S., Zaheer, M., Vilnis, L., Durugkar, I., Krishnamurthy, A., Smola, A., McCallum, A.: Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. arXiv preprint arXiv:1711.05851 (2017)
Pith/arXiv arXiv 2017
-
[7]
IEEE transactions on pattern analysis and machine intelligence (2), 224–227 (2009) 16 Susana Nunes, Tiago Guerreiro, and Catia Pesquita
Davies, D.L., Bouldin, D.W.: A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence (2), 224–227 (2009) 16 Susana Nunes, Tiago Guerreiro, and Catia Pesquita
2009
-
[8]
In: HCXAI 2023-Workshop on Human-Centered Explainable AI
Delaunay, J., Largouët, C., Galárraga, L., Van Berkel, N.: Adaptation of ai explanations to users’ roles. In: HCXAI 2023-Workshop on Human-Centered Explainable AI. pp. 1–7 (2023)
2023
-
[9]
arXiv preprint arXiv:1702.08608 (2017)
Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017)
Pith/arXiv arXiv 2017
-
[10]
Artificial Intelligence297, 103498 (2021)
Durán, J.M.: Dissecting scientific explanation in ai (sxai): A case for medicine and health- care. Artificial Intelligence297, 103498 (2021)
2021
-
[11]
NPJ digital medicine4(1), 31 (2021)
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S.J., Lermer, E., Coughlin, J.F., Guttag, J.V ., Colak, E., Ghassemi, M.: Do as ai say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine4(1), 31 (2021)
2021
-
[12]
Technological Forecasting and Social Change186, 122120 (2023)
Haque, A.B., Islam, A.N., Mikalef, P.: Explainable artificial intelligence (xai) from a user perspective: A synthesis of prior literature and problematizing avenues for future research. Technological Forecasting and Social Change186, 122120 (2023)
2023
-
[13]
Elife6, e26726 (2017)
Himmelstein, D.S., Lizee, A., Hessler, C., Brueggeman, L., Chen, S.L., Hadley, D., Green, A., Khankhanian, P., Baranzini, S.E.: Systematic integration of biomedical knowledge prior- itizes drugs for repurposing. Elife6, e26726 (2017)
2017
-
[14]
Nature Medicine30(12), 3601–3613 (2024)
Huang, K., Chandak, P., Wang, Q., Havaldar, S., Vaid, A., Leskovec, J., Nadkarni, G.N., Glicksberg, B.S., Gehlenborg, N., Zitnik, M.: A foundation model for clinician-centered drug repurposing. Nature Medicine30(12), 3601–3613 (2024)
2024
-
[15]
Pattern recognition letters31(8), 651– 666 (2010)
Jain, A.K.: Data clustering: 50 years beyond k-means. Pattern recognition letters31(8), 651– 666 (2010)
2010
-
[16]
Nature Machine Intelligence2(10), 573–584 (2020)
Jiménez-Luna, J., Grisoni, F., Schneider, G.: Drug discovery with explainable artificial intel- ligence. Nature Machine Intelligence2(10), 573–584 (2020)
2020
-
[17]
Synthese195(6), 2761–2793 (2018)
Keas, M.N.: Systematizing the theoretical virtues. Synthese195(6), 2761–2793 (2018)
2018
-
[18]
Journal of chiropractic medicine15(2), 155–163 (2016)
Koo, T.K., Li, M.Y .: A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine15(2), 155–163 (2016)
2016
-
[19]
Lazik, C.K., Katins, C., Kauter, C., Jakob, J., Jay, C., Grunske, L., Kosch, T.: The impos- tor is among us: Can large language models capture the complexity of human personas? In: Proceedings of the Mensch Und Computer 2025. p. 434–451. MuC ’25, Association for Computing Machinery, New York, NY , USA (2025).https://doi.org/10.1145/ 3743049.3743057,https:...
arXiv 2025
-
[20]
Semantic Web11(1), 41–51 (2020)
Lecue, F.: On the role of knowledge graphs in explainable ai. Semantic Web11(1), 41–51 (2020)
2020
-
[21]
In: European Semantic Web Conference
Liu, Y ., Hildebrandt, M., Joblin, M., Ringsquandl, M., Raissouni, R., Tresp, V .: Neural multi- hop reasoning with logical rules on biomedical knowledge graphs. In: European Semantic Web Conference. pp. 375–391. Springer (2021)
2021
-
[22]
Artificial intelligence267, 1–38 (2019)
Miller, T.: Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence267, 1–38 (2019)
2019
-
[23]
Nimmo, R., Constantinides, M., Zhou, K., Quercia, D., Stumpf, S.: User characteristics in explainable ai: The rabbit hole of personalization? In: Proceedings of the 2024 CHI Confer- ence on Human Factors in Computing Systems. pp. 1–13 (2024)
2024
-
[24]
Nunes, S., Badreddine, S., Pesquita, C.: Rewarding explainability in drug repurposing with knowledge graphs. In: Kwok, J. (ed.) Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. pp. 4624–4632. International Joint Conferences on Artificial Intelligence Organization (8 2025).https://doi.org/10. 24963/ijcai...
-
[25]
Wiley Interdisciplinary Reviews: Computational Molecular Science13(6), e1681 (2023)
Ponzoni, I., Páez Prosper, J.A., Campillo, N.E.: Explainable artificial intelligence: A taxon- omy and guidelines for its application to drug discovery. Wiley Interdisciplinary Reviews: Computational Molecular Science13(6), e1681 (2023)
2023
-
[26]
Jour- nal of information science50(4), 1019–1029 (2024) Agentic Personas for Adaptive Scientific Explanations with Knowledge Graphs 17
Rajabi, E., Etminani, K.: Knowledge-graph-based explainable ai: A systematic review. Jour- nal of information science50(4), 1019–1029 (2024) Agentic Personas for Adaptive Scientific Explanations with Knowledge Graphs 17
2024
-
[27]
In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019),https:// arxiv.org/abs/1908.10084
Pith/arXiv arXiv 2019
-
[28]
Journal of computational and applied mathematics20, 53–65 (1987)
Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics20, 53–65 (1987)
1987
-
[29]
In: Proceedings of the 24th international conference on intelligent user interfaces
Schaffer, J., O’Donovan, J., Michaelis, J., Raglin, A., Höllerer, T.: I can do better than your ai: expertise and explanations. In: Proceedings of the 24th international conference on intelligent user interfaces. pp. 240–251 (2019)
2019
-
[30]
International Journal of Human-Computer Studies154, 102684 (06 2021).https://doi.org/10
Schoonderwoerd, T., Jorritsma, W., Neerincx, M., Bosch, K.: Human-centered xai: Devel- oping design patterns for explanations of clinical decision support systems. International Journal of Human-Computer Studies154, 102684 (06 2021).https://doi.org/10. 1016/j.ijhcs.2021.102684
arXiv 2021
-
[31]
In: Proceedings of the 2024 ACM Designing Interactive Systems Conference
Shin, J., Hedderich, M.A., Rey, B.J., Lucero, A., Oulasvirta, A.: Understanding human-ai workflows for generating personas. In: Proceedings of the 2024 ACM Designing Interactive Systems Conference. pp. 757–781 (2024)
2024
-
[32]
Psycho- logical bulletin86(2), 420 (1979)
Shrout, P.E., Fleiss, J.L.: Intraclass correlations: uses in assessing rater reliability. Psycho- logical bulletin86(2), 420 (1979)
1979
-
[33]
Advances in Neural Information Processing Systems33, 16857– 16867 (2020)
Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y .: Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems33, 16857– 16867 (2020)
2020
-
[34]
In: Proceedings of the 2021 CHI conference on human factors in computing systems
Suresh, H., Gomez, S.R., Nam, K.K., Satyanarayan, A.: Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In: Proceedings of the 2021 CHI conference on human factors in computing systems. pp. 1– 16 (2021)
2021
-
[35]
Artificial Intelligence302, 103627 (2022)
Tiddi, I., Schlobach, S.: Knowledge graphs as tools for explainable machine learning: A survey. Artificial Intelligence302, 103627 (2022)
2022
-
[36]
Journal of Chemical Information and Modeling63(24), 7617–7627 (2023)
Wu, Z., Chen, J., Li, Y ., Deng, Y ., Zhao, H., Hsieh, C.Y ., Hou, T.: From black boxes to ac- tionable insights: a perspective on explainable artificial intelligence for scientific discovery. Journal of Chemical Information and Modeling63(24), 7617–7627 (2023)
2023
-
[37]
arXiv preprint arXiv:1707.06690 (2017)
Xiong, W., Hoang, T., Wang, W.Y .: Deeppath: A reinforcement learning method for knowl- edge graph reasoning. arXiv preprint arXiv:1707.06690 (2017)
Pith/arXiv arXiv 2017
-
[38]
Nature Medicine30(3), 837–849 (2024)
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., Rajpurkar, P.: Heterogeneity and predictors of the effects of ai assistance on radiologists. Nature Medicine30(3), 837–849 (2024)
2024
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.