REVIEW 4 major objections 38 references
Routing new papers into a personal library works best when an agent reads what each folder already holds, not just its name.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
2026-07-14 04:43 UTC pith:ET7FRCIC
load-bearing objection Clean formalization of a real personal-library routing problem plus a training-free agent that actually reads folder members; the user-study gains are real but partly softened by post-hoc gold labeling. the 4 major comments →
PaperRouter-Agent: A Content-Grounded LLM Agent for Personalized Hierarchical Paper Routing
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When folder meaning is defined by its members rather than its label, a training-free LLM agent that inspects those members (and routes metadata folders by metadata) substantially outperforms name-only single-shot classification on real personal libraries and on a personalized tagging proxy, recovering the organizational folders that name matching collapses on.
What carries the argument
PaperRouter-Agent: a four-stage, training-free pipeline (Planner for top-down candidate narrowing, Retriever for type-aware evidence selection, Inspector for content-grounded Fit/No-Fit/Abstain verification plus cross-comparison, Reflector for similarity-gated reuse of past user rejections).
Load-bearing premise
That results from a two-round study with only five research students and twenty papers each, with gold labels collected after the system has already proposed folders, generalize to the wider range of personal libraries and folder styles.
What would settle it
A larger multi-user study in which content-grounded inspection fails to improve Recall@1 over a strong name-based baseline on venue/year or shorthand folders, or where steady-state latency and call counts exceed the single-shot baseline without recovering those folder types.
If this is right
- Reference managers can automate folder placement by reading existing members instead of relying on folder names.
- Organizational folders defined by venue, year, or status become recoverable once the agent routes them by metadata rather than content alone.
- Online accept/reject memory can refine later decisions without any per-user fine-tuning.
- The same content-grounded inspection transfers to other personalized tagging settings that lack a shared taxonomy.
Where Pith is reading between the lines
- Any personal-organization task whose labels are folksonomic (bookmarks, notes, media libraries) is likely to benefit from the same member-first inspection pattern.
- Process/status folders remain hard because membership encodes user intent rather than content or metadata; solving them may require an explicit intent model.
- Building an open multi-user PHPR benchmark would let long-term Reflector gains be measured properly, which the single-session study cannot yet do.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes personalized hierarchical paper routing (PHPR) as the task of assigning an incoming paper to folders in a user-specific, evolving folksonomy (rather than a fixed shared taxonomy), without per-user training. It proposes PaperRouter-Agent, a training-free four-stage LLM agent (Planner, Retriever, Inspector, Reflector) that narrows candidates top-down, types folders for type-aware evidence, verifies fit by inspecting sampled member papers (with abstention), and injects similarity-gated past rejections. On a two-round within-subjects study of five researchers’ real Zotero libraries (100 papers total), it reports Recall@1 rising from 0.39 to 0.61 and Recall@3 from 0.57 to 0.83 versus a single-shot name-based baseline, with largest gains on venue/year folders (0.09→0.50). On LaMP-2 it lifts accuracy from 44.5% to 51.5% (+9.0 macro-F1) by content-grounded inspection alone, with a cost/latency analysis showing practical steady-state overhead.
Significance. If the gains hold, the work usefully carves out PHPR as distinct from classical hierarchical text classification and from flat personalized tagging, and shows that content-grounded inspection of folder members (plus metadata routing for organizational folders) recovers failures that pure name matching cannot. Strengths include a clear problem formalization (Table 1), a deployed Zotero integration used in the study, explicit per-type breakdowns, cost/latency measurements against single-shot and inspect-all, and a public-benchmark result that matches a retrieval-augmented LLM without a separate retriever or fine-tuning. The training-free, online-feedback design is practically relevant for personal knowledge management tools.
major comments (4)
- Section 5.2 and Appendix B (Table 5): the gold set R⋆ is constructed after participants see the system’s ranked candidates (accept correct returned folders and name omitted correct ones). Because the same papers are run first under the single-shot baseline then under the agent, and because the agent surfaces metadata folders the baseline rarely proposes, ratification bias and order/learning effects can inflate the reported Recall@1/3 lifts (overall 0.39→0.61, venue/year 0.09→0.50). An independent pre-system gold, counterbalanced condition order, or a blinded second labeling pass is needed to secure the magnitude of the central claim.
- Section 5.2: n=5 participants × 20 papers is too small and homogeneous (research students) to underwrite generalization across library sizes, depths, and folder-type mixtures in the broader population. Per-type cells are especially thin (process/status n=12, venue/year n=18). Either enlarge the study or substantially strengthen the limitations discussion and confidence intervals; the current formative framing understates how load-bearing these numbers are for the abstract and introduction claims.
- Section 5.3 and Discussion: LaMP-2 evaluates only the Inspector on a flat, fixed-tag proxy and does not exercise Planner, Retriever, or Reflector, nor hierarchical multi-type routing. The Reflector’s online feedback mechanism is described but never longitudinally measured (explicitly left to future work). The paper’s strongest claim is hierarchical multi-type gains on real libraries; the public result cannot substitute for that evaluation. Either add a longitudinal or leave-one-out feedback experiment, or clearly demote the Reflector from a core contribution to a designed-but-untested component.
- Section 4.3–4.4 and Table 3: process/status folders remain a residual hard case (Recall@1 0.12→0.44) and the Retriever’s binary topical-vs-organizational typing leaves them without a principled evidence route. Given that the paper advertises four folder types, the agent’s handling of process/status needs either a concrete mechanism (e.g., user-intent cues or explicit abstention policy) or a clearer statement that these folders are out of scope for content grounding.
Circularity Check
Empirical systems paper with no derivation chain that reduces to its inputs; claims are measured against user gold and a public benchmark.
full rationale
PaperRouter-Agent is a training-free multi-stage LLM agent evaluated with standard Recall@K on five users' Zotero libraries and accuracy/macro-F1 on the public LaMP-2 benchmark. There is no mathematical derivation, fitted parameter renamed as a prediction, uniqueness theorem imported from the authors, or ansatz smuggled via self-citation. The agent does not train on the evaluation libraries; folder typing and member sampling are operational design choices, not circular definitions of the reported metrics. Self-citations in Related Work are ordinary positioning, not load-bearing premises that force the empirical gains. Methodological concerns about post-hoc gold labeling (participants naming correct folders after seeing candidates) affect evaluation validity, not circularity of a derivation. Score 0 is therefore appropriate: the paper is self-contained against external labels and contains no circular reduction.
Axiom & Free-Parameter Ledger
free parameters (2)
- members sampled per folder (N) =
5
- Planner top-k branches kept =
5
axioms (3)
- domain assumption A modern LLM (gpt-4o-mini) can judge topical fit of a new paper against a small sample of member abstracts with usable accuracy.
- domain assumption Folder type (topical vs organizational) can be reliably inferred from name plus a quick look at members.
- domain assumption User accept/reject feedback is a sufficient online signal for the Reflector without offline training.
invented entities (2)
-
Personalized Hierarchical Paper Routing (PHPR)
no independent evidence
-
PaperRouter-Agent four-stage pipeline (Planner/Retriever/Inspector/Reflector)
no independent evidence
Cite this review
Pith. "Pith review of PaperRouter-Agent: A Content-Grounded LLM Agent for Personalized Hierarchical Paper Routing." pith.science (2026). https://pith.science/paper/ET7FRCIC
@misc{pith2026260711564,
author = {Pith},
title = {Pith review of: PaperRouter-Agent: A Content-Grounded LLM Agent for Personalized Hierarchical Paper Routing},
year = {2026},
howpublished = {\url{https://pith.science/paper/ET7FRCIC}},
note = {Machine review of arXiv:2607.11564}
}
read the original abstract
Researchers organize the papers they collect into personal folder hierarchies in reference managers, and route each new paper into the folder where it belongs. This task differs from standard hierarchical text classification. A user's folder hierarchy is not a fixed, shared taxonomy but a private and evolving folksonomy whose folder meanings may be topical, shorthand, venue-based, or process-oriented, and are often defined by the papers already stored inside them. We formalize this setting as personalized hierarchical paper routing (PHPR): assigning an incoming paper to folders in a user-specific hierarchy without per-user training. We propose PaperRouter-Agent, a training-free LLM agent that grounds routing decisions in folder members rather than folder names alone. The agent first narrows the candidate hierarchy, retrieves folder-specific evidence, verifies fit by inspecting member papers, and incorporates similarity-gated feedback from past user rejections. A formative study on real personal libraries shows that PaperRouter-Agent raises overall Recall@1 from 0.39 to 0.61 and Recall@3 from 0.57 to 0.83, with the largest gains on organizational folders defined by metadata such as venue or year, where single-shot methods collapses (Recall@1 0.09 to 0.50). On the public LaMP-2 benchmark, the same approach improves accuracy from 44.5% to 51.5% (+9.0 macro-F1) over a single-shot baseline, while remaining low-cost for practical use.
Figures
Reference graph
Works this paper leans on
-
[1]
Cornell University Library (2026), https://arxiv.org/stats/monthly_submissions, accessed 2026-05-19; monthly submissions 24,226 (Oct 2024), 27,692 (Oct 2025), 30,045 (Mar 2026)
arXiv: arXiv Monthly Submission Statistics. Cornell University Library (2026), https://arxiv.org/stats/monthly_submissions, accessed 2026-05-19; monthly submissions 24,226 (Oct 2024), 27,692 (Oct 2025), 30,045 (Mar 2026)
2026
-
[2]
In: Proceedings of the International Conference on Learning Representations (ICLR) (2024)
Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H.: Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In: Proceedings of the International Conference on Learning Representations (ICLR) (2024)
2024
-
[3]
BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions
Bohra, A., Verkes, G., Harutyunyan, A., Weinberger, P., Campagna, G.: BYOC: Personalized few-shot classification with co-authored class descriptions. In: Find- ings of the Association for Computational Linguistics: EMNLP (2023).https: //doi.org/10.48550/arXiv.2310.06111
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.06111 2023
-
[4]
Chang, J.C., Zhang, A.X., Bragg, J., Head, A., Lo, K., Downey, D., Weld, D.S.: CiteSee: Augmenting citations in scientific papers with persistent and personalized historical context. In: Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (2023).https://doi.org/10.1145/3544548.3580847
-
[5]
Journal of Machine Learning Research (JMLR) (2024)
Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S.S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E.H., Dean, J., Devlin, J., Roberts, A....
2024
-
[6]
Ding, H., Yang, J., Deng, Y., Zhang, H., Roth, D.: Towards open-domain topic classification. In: Proceedings of NAACL: System Demonstrations (2022).https: //doi.org/10.18653/v1/2022.naacl-demo.10
-
[7]
In: Proceedings of the Conference of the European Chap- ter of the Association for Computational Linguistics (EACL) (2026)
Golde, J., Jedema, N.P., Krishnan, R., Le, P.: Hierarchical text classification with LLM-refined taxonomies. In: Proceedings of the Conference of the European Chap- ter of the Association for Computational Linguistics (EACL) (2026)
2026
-
[8]
Journal of Information Science32(2), 198–208 (2006)
Golder, S.A., Huberman, B.A.: Usage patterns of collaborative tagging systems. Journal of Information Science32(2), 198–208 (2006)
2006
-
[9]
arXiv preprint (2025).https://doi.org/10.48550/arXiv.2510.19897
Hassell, J., Zhang, D., Kim, H., Mitchell, T., Hruschka, E.: Learning from supervi- sionwithsemanticandepisodicmemory:Areflectiveapproachtoagentadaptation. arXiv preprint (2025).https://doi.org/10.48550/arXiv.2510.19897
-
[10]
Transac- tions on Machine Learning Research (TMLR) (2022)
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning. Transac- tions on Machine Learning Research (TMLR) (2022)
2022
-
[11]
Kargupta, P., Zhang, N., Zhang, Y., Zhang, R., Mitra, P., Han, J.: TaxoAdapt: Aligning LLM-based multidimensional taxonomy construction to evolving research corpora. In: Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics (ACL) (2025).https://doi.org/10.18653/v1/2025.acl-long. 1442
-
[12]
In: Pro- ceedings of the 16th IEEE International Conference on Machine Learning and Applications (ICMLA)
Kowsari, K., Brown, D.E., Heidarysafa, M., Jafari Meimandi, K., Gerber, M.S., Barnes, L.E.: HDLTex: Hierarchical deep learning for text classification. In: Pro- ceedings of the 16th IEEE International Conference on Machine Learning and Applications (ICMLA). pp. 364–371 (2017)
2017
-
[13]
Lee, Y., Kang, H.B., Latzke, M., Kim, J., Bragg, J., Chang, J.C., Siangliulue, P.: PaperWeaver: Enriching topical paper alerts by contextualizing recommended papers with user-collected papers. In: Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (2024).https://doi.org/10.1145/ 3613904.3642196 14 K. Zhou et al
arXiv 2024
-
[14]
Semantic Web6(2), 167–195 (2015)
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., Bizer, C.: DBpedia – a large- scale, multilingual knowledge base extracted from Wikipedia. Semantic Web6(2), 167–195 (2015)
2015
-
[15]
arXiv preprint (2026).https://doi.org/10.48550/arXiv.2603.19711
Li, Y., Ma, T., Ye, Y.: EvoTaxo: Building and evolving taxonomy from social media streams. arXiv preprint (2026).https://doi.org/10.48550/arXiv.2603.19711
-
[16]
arXiv preprint (2024).https: //doi.org/10.48550/arXiv.2401.05459
Li, Y., Wen, H., Wang, W., Li, X., Yuan, Y.: Personal LLM agents: Insights and survey about the capability, efficiency and security. arXiv preprint (2024).https: //doi.org/10.48550/arXiv.2401.05459
-
[17]
Bulletin of the Medical Library Association88(3), 265–266 (2000)
Lipscomb, C.E.: Medical subject headings (MeSH). Bulletin of the Medical Library Association88(3), 265–266 (2000)
2000
-
[18]
GitHub repository (2026),https://github.com/ lostwildland/zotero-llm-tagger, accessed 2026-05-19
lostwildland: AI Tagger for Zotero (zotero-llm-tagger): LLM-based automatic tag suggestion for zotero 7. GitHub repository (2026),https://github.com/ lostwildland/zotero-llm-tagger, accessed 2026-05-19
2026
-
[19]
arXiv preprint (2025).https://doi.org/10.48550/arXiv.2510.01609
Ma, B., Li, H., Hu, Z., Gui, X., Liu, L.: AgentRec: Next-generation LLM- powered multi-agent collaborative recommendation with adaptive intelligence. arXiv preprint (2025).https://doi.org/10.48550/arXiv.2510.01609
-
[20]
In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Train- ing language models to follow instructions with human feedback. In: Advances in Neural Information Process...
2022
-
[21]
GitHub repository (2024),https://github.com/papersgpt/ papersgpt-for-zotero, accessed 2026-05-19
papersgpt contributors: PapersGPT: Chat with pdfs and autopilot summaries for zotero (MCP server). GitHub repository (2024),https://github.com/papersgpt/ papersgpt-for-zotero, accessed 2026-05-19
2024
-
[22]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021),https://arxiv.org/abs/ 2103.00020
Pith/arXiv arXiv 2021
-
[23]
In: Proceedings of the Annual Meeting of the Asso- ciation for Computational Linguistics (ACL) (2024)
Salemi, A., Mysore, S., Bendersky, M., Zamani, H.: LaMP: When large language models meet personalization. In: Proceedings of the Annual Meeting of the Asso- ciation for Computational Linguistics (ACL) (2024)
2024
-
[25]
Schmidt, F., Hammerfald, K., Jahren, H.H., Payberah, A.H., Vlassov, V.: Single- pass hierarchical text classification with large language models. In: Proceedings of IEEE International Conference on Big Data (BigData) (2024).https://doi.org/ 10.1109/BigData62323.2024.10825412
-
[26]
In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)
Shen, J., Qiu, W., Meng, Y., Shang, J., Ren, X., Han, J.: TaxoClass: Hierarchical multi-label text classification using only class names. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). pp. 4239–4249 (2021)
2021
-
[27]
In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S.: Reflex- ion: Language agents with verbal reinforcement learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
2023
-
[28]
Singh, M., Cambronero, J., Gulwani, S., Le, V., Verbruggen, G.: EmFore: Online learning of email folder classification rules. In: Proceedings of the ACM Interna- tional Conference on Information and Knowledge Management (CIKM) (2023). https://doi.org/10.1145/3583780.3614863 PaperRouter-Agent 15
-
[29]
Publi- cations7(1), 18 (2019).https://doi.org/10.3390/publications7010018
Tenopir, C., Christian, L., Kaufman, J.: Seeking, reading, and use of scholarly articles: An international study of perceptions and behavior of researchers. Publi- cations7(1), 18 (2019).https://doi.org/10.3390/publications7010018
-
[30]
Vander Wal, T.: Folksonomy coinage and definition.https://vanderwal.net/ folksonomy.html(2007), accessed 2026-06-14
2007
-
[31]
Wang, L., Yurechko, K., Dani, P., Chen, Q.Z., Zhang, A.X.: End user authoring of personalized content classifiers: Comparing example labeling, rule writing, and LLM prompting. In: Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (2025).https://doi.org/10.1145/3706598.3713691
-
[32]
In: Proceedings of the Inter- national Conference on Learning Representations (ICLR) (2023)
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing reasoning and acting in language models. In: Proceedings of the Inter- national Conference on Learning Representations (ICLR) (2023)
2023
-
[33]
Yoshimura, K., Kashima, H.: Hierarchical text classification using black box large languagemodels.arXivpreprint(2025).https://doi.org/10.48550/arXiv.2508. 04219
-
[34]
Electronics13(7), 1199(2024).https://doi.org/10.3390/electronics13071199
Zangari, A., Bertolucci, M., Frasca, M.: Hierarchical text classification: A survey. Electronics13(7), 1199(2024).https://doi.org/10.3390/electronics13071199
-
[35]
Zhang, Q., Su, Q., Zhu, W., Pang, Y.: HierPrompt: Zero-shot hierarchical text classification with LLM-enhanced prototypes. In: Findings of the Association for Computational Linguistics: EMNLP (2025).https://doi.org/10.18653/v1/ 2025.findings-emnlp.207
doi:10.18653/v1/ 2025
-
[36]
In: Proceedings of the ACM Web Conference (WWW) (2024).https://doi.org/10
Zhang, Y., Yang, R., Xu, X., Li, R., Xiao, J.: TELEClass: Taxonomy enrichment and LLM-enhanced hierarchical text classification with minimal supervision. In: Proceedings of the ACM Web Conference (WWW) (2024).https://doi.org/10. 1145/3696410.3714940
arXiv 2024
-
[37]
arXiv preprint (2024).https://doi.org/10.48550/ arXiv.2411.00027
Zhang, Z., Rossi, R.A., Kveton, B., Shao, Y., Yang, D.: Personalization of large language models: A survey. arXiv preprint (2024).https://doi.org/10.48550/ arXiv.2411.00027
Pith/arXiv arXiv 2024
-
[38]
arXiv preprint (2025).https://doi.org/10.48550/arXiv
Zhou, H., Chen, Y., Guo, S., et al.: Memento: Fine-tuning LLM agents with- out fine-tuning LLMs. arXiv preprint (2025).https://doi.org/10.48550/arXiv. 2508.16153
-
[39]
Randomly pick 20 papers you usually read and file each into your own hierarchy (collections)
Zhou, J., Ma, C., Long, D., Xu, G., Ding, N., Zhang, H., Xie, P., Liu, G.: Hierarchy- aware global model for hierarchical text classification. In: Proceedings of the 58th AnnualMeetingoftheAssociationforComputationalLinguistics(ACL).pp.1106– 1117 (2020) 16 K. Zhou et al. Appendix This appendix collects material referenced from the main paper: qualitative ...
2020
This paper was first reviewed by grok-4.5 on July 14, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.