{"id":"6103b9ce-8f75-4e5e-9a6b-21d24ace541d","arxiv_id":"2607.11564","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A training-free four-stage LLM agent that routes papers into personal folksonomy folders by inspecting member papers and metadata, lifting Recall@1 from 0.39 to 0.61 on real libraries.","lead":"PaperRouter-Agent routes new papers into a researcher's private folder hierarchy by reading the papers already inside each folder, not just folder names. It formalizes this as personalized hierarchical paper routing and shows large gains on real Zotero libraries and a public tagging benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The headline gains rest on a gold-label protocol that is not independent of system output, so the reported Recall@1/3 lifts may partly reflect post-hoc labeling rather than pure routing quality.","rationale":"The Reader correctly flags the n=5 scale and the missing longitudinal Reflector evaluation as the main soft spots, and the CONDITIONAL verdict is appropriate. The more load-bearing technical issue, however, is not merely sample size but the non-independence of the gold labels themselves: Rigstar is constructed after participants see the ranked candidates (explicitly stated in Section 5.2 and Appendix B). That design choice directly affects the metric that carries the strongest claim. The LaMP-2 numbers remain a useful, cleaner corroboration of content-grounded inspection, yet they do not rescue the hierarchical multi-type gains. A pre-system gold or counterbalanced re-collection would settle whether the reported deltas are robust; until then the claim stays conditional on a larger, cleaner validation. No mathematical circularity or fabricated result is present; the concern is purely about measurement validity of the primary empirical support.","tokens_in":15072,"tokens_out":590,"duration_ms":7156,"concrete_test":"Re-run the identical 100-paper protocol with a third, fully independent gold collection performed before either system is shown (or with order fully counterbalanced and gold locked before any output). Recompute Table 3 Recall@1/3; if the agent-to-baseline gap shrinks by more than ~0.10 absolute on overall or venue/year rows, the headline lift is protocol-dependent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Recall@1 0.39\to0.61, R@3 0.57\to0.83, venue/year 0.09\to0.50) is measured against gold sets Rigstar that participants construct after seeing the system's ranked candidates: they accept correct returned folders and name any omitted correct ones (Section 5.2, Appendix B Table 5). Because the gold is elicited only after exposure to the agent's suggestions, two confounds threaten the comparison. First, participants may be more likely to ratify folders the agent already surfaced (especially metadata folders that the single-shot baseline never proposes), inflating the agent's Recall relative to the baseline. Second, the same 20 papers are filed under both systems in a fixed order (baseline then agent), so learning or fatigue can further favor the second condition. The LaMP-2 result is cleaner but only evaluates the Inspector on a flat, fixed-tag proxy; it does not underwrite the hierarchical, multi-type gains that constitute the paper's strongest claim. Without an independent, pre-system gold or a counterbalanced design, the magnitude of the reported improvement remains only partially secured.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper formalizes personalized hierarchical paper routing (PHPR) as the task of assigning an incoming paper to folders in a user-specific, evolving folksonomy (rather than a fixed shared taxonomy), without per-user training. It proposes PaperRouter-Agent, a training-free four-stage LLM agent (Planner, Retriever, Inspector, Reflector) that narrows candidates top-down, types folders for type-aware evidence, verifies fit by inspecting sampled member papers (with abstention), and injects similarity-gated past rejections. On a two-round within-subjects study of five researchers’ real Zotero libraries (100 papers total), it reports Recall@1 rising from 0.39 to 0.61 and Recall@3 from 0.57 to 0.83 versus a single-shot name-based baseline, with largest gains on venue/year folders (0.09→0.50). On LaMP-2 it lifts accuracy from 44.5% to 51.5% (+9.0 macro-F1) by content-grounded inspection alone, with a cost/latency analysis showing practical steady-state overhead.","tokens_in":15364,"tokens_out":927,"duration_ms":12812,"significance":"If the gains hold, the work usefully carves out PHPR as distinct from classical hierarchical text classification and from flat personalized tagging, and shows that content-grounded inspection of folder members (plus metadata routing for organizational folders) recovers failures that pure name matching cannot. Strengths include a clear problem formalization (Table 1), a deployed Zotero integration used in the study, explicit per-type breakdowns, cost/latency measurements against single-shot and inspect-all, and a public-benchmark result that matches a retrieval-augmented LLM without a separate retriever or fine-tuning. The training-free, online-feedback design is practically relevant for personal knowledge management tools.","major_comments":[{"comment":"Section 5.2 and Appendix B (Table 5): the gold set R⋆ is constructed after participants see the system’s ranked candidates (accept correct returned folders and name omitted correct ones). Because the same papers are run first under the single-shot baseline then under the agent, and because the agent surfaces metadata folders the baseline rarely proposes, ratification bias and order/learning effects can inflate the reported Recall@1/3 lifts (overall 0.39→0.61, venue/year 0.09→0.50). An independent pre-system gold, counterbalanced condition order, or a blinded second labeling pass is needed to secure the magnitude of the central claim.","section":null},{"comment":"Section 5.2: n=5 participants × 20 papers is too small and homogeneous (research students) to underwrite generalization across library sizes, depths, and folder-type mixtures in the broader population. Per-type cells are especially thin (process/status n=12, venue/year n=18). Either enlarge the study or substantially strengthen the limitations discussion and confidence intervals; the current formative framing understates how load-bearing these numbers are for the abstract and introduction claims.","section":null},{"comment":"Section 5.3 and Discussion: LaMP-2 evaluates only the Inspector on a flat, fixed-tag proxy and does not exercise Planner, Retriever, or Reflector, nor hierarchical multi-type routing. The Reflector’s online feedback mechanism is described but never longitudinally measured (explicitly left to future work). The paper’s strongest claim is hierarchical multi-type gains on real libraries; the public result cannot substitute for that evaluation. Either add a longitudinal or leave-one-out feedback experiment, or clearly demote the Reflector from a core contribution to a designed-but-untested component.","section":null},{"comment":"Section 4.3–4.4 and Table 3: process/status folders remain a residual hard case (Recall@1 0.12→0.44) and the Retriever’s binary topical-vs-organizational typing leaves them without a principled evidence route. Given that the paper advertises four folder types, the agent’s handling of process/status needs either a concrete mechanism (e.g., user-intent cues or explicit abstention policy) or a clearer statement that these folders are out of scope for content grounding.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful thing here is the problem cut. They separate PHPR (private, evolving folksonomy whose meaning lives in the members) from classical HTC (shared fixed taxonomy whose meaning lives in the name). That distinction is right, and it immediately explains why name-only zero-shot routing collapses on venue/year and shorthand folders. The four-stage agent is a sensible engineering response: Planner narrows top-down, Retriever types folders and routes metadata ones by fields, Inspector reads sampled members with Fit/No-Fit/Abstain plus cross-comparison, Reflector injects similar past rejections. No per-user training, gpt-4o-mini backbone, Zotero integration actually built.\n\nWhat works: on five real libraries the agent lifts overall R@1 0.39\to0.61 and R@3 0.57\to0.83, with the biggest jump exactly where the baseline dies (venue/year 0.09\to0.50). Every participant improves. On LaMP-2 the same Inspector idea moves accuracy 44.5%\to51.5% (+9 macro-F1) and matches a retrieval-augmented GPT-3.5 baseline without a separate retriever. Cost numbers are honest: steady-state real-library routing is cheaper than the long single-shot prompt once folders are summarized. The failure taxonomy in Table 2 is clear and the prompts/pseudocode are in the appendix.\n\nSoft spots, in proportion. The user-study gold is collected after participants see the ranked candidates (accept correct returns, name omitted ones). That is not fully independent; it can inflate the agent relative to the baseline that never surfaces metadata folders, and the fixed order (baseline then agent) adds a mild learning confound. n=5 is formative, not definitive. The Reflector is described but never longitudinally measured. Process/status folders remain hard. No public code. None of these sink the central claim that content-grounded inspection helps; they just mean the magnitude of the hierarchical gains is only partially secured. LaMP-2 is cleaner but only tests the Inspector on a flat proxy.\n\nThis is for people building personal knowledge tools or LLM agents for user-specific organization. The formalization and the Inspector mechanism are worth citing; the absolute numbers need a larger, pre-labeled or counterbalanced study. I would send it to peer review. A serious referee can demand the independent gold and code release without killing the paper.","headline":"Clean formalization of a real personal-library routing problem plus a training-free agent that actually reads folder members; the user-study gains are real but partly softened by post-hoc gold labeling.","tokens_in":15981,"tokens_out":594,"would_cite":true,"duration_ms":5425,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Routing new papers into a personal library works best when an agent reads what each folder already holds, not just its name.","keywords":["personalized hierarchical paper routing","folksonomy","content-grounded inspection","LLM agents","retrieval-augmented generation","personal knowledge management","hierarchical text classification"],"falsifier":"A larger multi-user study in which content-grounded inspection fails to improve Recall@1 over a strong name-based baseline on venue/year or shorthand folders, or where steady-state latency and call counts exceed the single-shot baseline without recovering those folder types.","tokens_in":15977,"feed_emoji":"📁","tokens_out":860,"duration_ms":8520,"temperature":0.7,"pith_summary":"Personal reference libraries are not shared taxonomies: folders may be topical, shorthand, venue-based, or process-oriented, and their meaning is often set by the papers already inside them. The paper formalizes this as personalized hierarchical paper routing (PHPR)—assigning an incoming paper to folders in a user’s own evolving hierarchy without any per-user training. It introduces PaperRouter-Agent, a training-free four-stage LLM agent that first narrows candidates top-down, then chooses evidence by folder type, verifies fit by reading member papers, and injects similar past rejections. On five researchers’ real Zotero libraries the agent lifts overall Recall@1 from 0.39 to 0.61 and Recall@3 from 0.57 to 0.83, with the largest gains on venue and year folders that name-only methods almost never hit. The same content-grounded inspection also raises accuracy on the public LaMP-2 tagging benchmark from 44.5 % to 51.5 % while staying practical in cost.","feed_headline":"Agent routes papers by reading folder contents, not names","feed_subtitle":"Recall@1 jumps from 0.39 to 0.61 on real Zotero libraries; venue folders recover most.","key_machinery":"PaperRouter-Agent: a four-stage, training-free pipeline (Planner for top-down candidate narrowing, Retriever for type-aware evidence selection, Inspector for content-grounded Fit/No-Fit/Abstain verification plus cross-comparison, Reflector for similarity-gated reuse of past user rejections).","core_discovery":"When folder meaning is defined by its members rather than its label, a training-free LLM agent that inspects those members (and routes metadata folders by metadata) substantially outperforms name-only single-shot classification on real personal libraries and on a personalized tagging proxy, recovering the organizational folders that name matching collapses on.","pith_inferences":["Any personal-organization task whose labels are folksonomic (bookmarks, notes, media libraries) is likely to benefit from the same member-first inspection pattern.","Process/status folders remain hard because membership encodes user intent rather than content or metadata; solving them may require an explicit intent model.","Building an open multi-user PHPR benchmark would let long-term Reflector gains be measured properly, which the single-session study cannot yet do."],"forward_implications":["Reference managers can automate folder placement by reading existing members instead of relying on folder names.","Organizational folders defined by venue, year, or status become recoverable once the agent routes them by metadata rather than content alone.","Online accept/reject memory can refine later decisions without any per-user fine-tuning.","The same content-grounded inspection transfers to other personalized tagging settings that lack a shared taxonomy."],"fun_headline_variants":["Agent routes papers by inspecting folder members not names","Content-grounded LLM agent lifts routing Recall@1 from 0.39 to 0.61","Member-paper inspection recovers venue folders name-only methods miss","Training-free agent grounds PHPR in folder contents over labels","Folder-member evidence raises hierarchical paper routing accuracy"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That results from a two-round study with only five research students and twenty papers each, with gold labels collected after the system has already proposed folders, generalize to the wider range of personal libraries and folder styles.","fun_headline_variants_meta":{"raw":{"variants":["Agent routes papers by inspecting folder members not names","Content-grounded LLM agent lifts routing Recall@1 from 0.39 to 0.61","Member-paper inspection recovers venue folders name-only methods miss","Training-free agent grounds PHPR in folder contents over labels","Folder-member evidence raises hierarchical paper routing accuracy"]},"model":"grok-4.5","effort":"low","cost_usd":0.005366,"raw_usage":{"total_tokens":1475,"prompt_tokens":823,"num_sources_used":0,"completion_tokens":91,"cost_in_usd_ticks":53660000,"prompt_tokens_details":{"text_tokens":823,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":561,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":823,"tokens_out":91,"duration_ms":5110,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T04:43:43.042723+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A larger multi-user study in which content-grounded inspection fails to improve Recall@1 over a strong name-based baseline on venue/year or shorthand folders, or where steady-state latency and call counts exceed the single-shot baseline without recovering those folder types.","supporting_citations":[],"review_version":1}