{"id":"03a7b499-c22c-4322-a075-a00d14d76be0","arxiv_id":"2411.17102","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multilingual search-LLM pipeline that retrieves native-name profiles achieves 96% profile recall and 98% disambiguation accuracy for Chinese scholars, but without a controlled comparison.","lead":"This paper proposes a search-enhanced LLM workflow that translates institution names, retrieves native-language web pages, and extracts scholars' native names to improve name disambiguation. The authors report higher recall and near-human disambiguation accuracy, but the experiments lack baselines and artifact details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 10's 98%/86.6% disambiguation accuracy is not derivable from the reported experiments, has no dataset or baseline, and the hand-set threshold in §3.4 is unsupported.","rationale":"The reader's verdict is REJECT because the paper's central claim is supported only by an unsupported accuracy table, and my reading reaches the same conclusion. However, the reader's stated weakest assumption concerns the availability and coverage of native-language web content, which is a real generalization risk but is not the most load-bearing issue. The more fundamental problem is that Table 10 provides no measurable disambiguation evaluation at all: no dataset, no baseline, no error analysis, and no derivation from the workflow's outputs. Even if native-language coverage were perfect, the paper still would not establish the claimed accuracy because the steps from extracted profiles to accuracy numbers are undocumented. Thus I agree with the REJECT verdict but identify the missing evaluation as the primary load-bearing concern. The concrete test — reproducing Table 10 with a fixed threshold and a baseline — would settle whether the headline claim has any empirical foundation.","tokens_in":6343,"tokens_out":2337,"duration_ms":23851,"concrete_test":"Release the evaluation set behind Table 10 (scholar pairs, raw search results, LLM extractions, and human labels) and rerun the entire pipeline using the §3.4 scoring rule with the threshold fixed before seeing labels. Report accuracy separately for same-person and different-person pairs, include a pinyin-only/English-institution baseline, and report a confusion matrix. If the 98% and 86.6% figures cannot be reproduced with a fixed threshold, or if the gap over the baseline vanishes, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that incorporating local languages via search-enhanced LLMs significantly improves scholar name disambiguation — rests almost entirely on Table 10, which reports 98% accuracy for GPT-4o and 86.6% for Hunyuan and asserts this 'can almost achieve accuracy of human expert.' This table is the only direct disambiguation result in the paper, and it is not connected to the retrieval experiments in Tables 1–9, which measure profile recall and native-name identification, not disambiguation decisions. §3.4 introduces a scoring rule for comparing two scholars' profiles (institution match 2 points, repeated education/work segment 3 points, research keywords 1–4 points, threshold 7), but the paper does not describe the dataset used for Table 10, the number of positive and negative pairs, the construction of those pairs, a baseline method, the human-expert accuracy against which 'almost achieve' is measured, or the procedure by which LLM outputs were converted into accuracy numbers. The threshold of 7 is a free parameter fit to the same data, with no sensitivity analysis. Because the headline result is an unverifiable number, the paper's central claim is currently a claim without derivation. Additionally, the method depends on the assumption in §2.2 and §3.3.1 that native-language web content exists and is retrievable; the reported 87% Chinese-name identification rate and 74% recall already bound how often that assumption holds, yet Table 10 does not condition results on retrieval success. The truncated code link ('https://github.com/') prevents replication.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a search-enhanced LLM pipeline for scholar name disambiguation. The method consists of an Extract Agent that retrieves and structures web profiles, a Name Translate Agent that infers native-language names (primarily Chinese) by translating institution information and re-searching, and a Disambiguation Agent that compares profiles using a scoring rule. The experiments report recall and precision for profile retrieval across three search engines (Sogou, Google, Bing) and two LLMs (GPT-4o and Hunyuan), first with romanized names and English institutions, then with Chinese institution names, then with inferred Chinese names, and finally with email supplements. Table 10 reports end-to-end disambiguation accuracy of 98% for GPT-4o and 86.6% for Hunyuan, described as \"almost achieve accuracy of human expert.\" The central claim is that incorporating local languages via search-enhanced LLMs significantly improves scholar name disambiguation.","tokens_in":6566,"tokens_out":4577,"duration_ms":43363,"significance":"If the claimed disambiguation accuracy were properly derived and reproducible, the paper would offer a practically useful empirical study of multilingual retrieval for profile construction, and the idea that native-language web content can enrich sparse Romanized profiles is plausible and consistent with the retrieval-recall trends in Tables 2-9. The paper deserves credit for comparing multiple search engines and LLMs and for designing a concrete multi-step workflow. However, the central disambiguation claim rests almost entirely on Table 10, which is not connected to any described dataset, evaluation protocol, baseline, or human-expert reference point. As written, the manuscript is an incomplete system description with anecdotal examples rather than a verifiable scientific contribution.","major_comments":[{"comment":"The headline result — 98% accuracy for GPT-4o and 86.6% for Hunyuan — is not derivable from any experiment described in the paper. The authors do not state the dataset size, the number of same-name pairs, how positive and negative pairs were constructed, the definition of accuracy, whether the reported numbers are exact-match rates or human-evaluated agreement, or the human-expert accuracy against which \"almost achieve\" is measured. The sentence in §3.4 states that human experts \"evaluated assisted with cross validation\" but gives no protocol or inter-annotator agreement. Because the abstract, introduction, and conclusion all make the disambiguation claim on the basis of these numbers, the paper's central result is currently unverifiable.","section":"§3.4, Table 10"},{"comment":"The retrieval experiments do not measure disambiguation. Tables 1-9 report \"recall\" and \"precision\" for retrieving scholar information or identifying native names, but the definitions of these metrics are never given, no sample sizes are reported, no error bars or statistical tests are provided, and there is no controlled comparison that isolates the effect of each added component. Even if the recall improvements from Tables 4-9 are accepted at face value, they concern profile retrieval and native-name identification, not the comparative judgment between two scholarly profiles. Consequently, the paper's claim that incorporating local languages \"significantly enhances disambiguation performance\" is not supported by the evidence in Tables 1-9.","section":"§3.1-§3.3, Tables 1-9"},{"comment":"The method depends on the assumption that rich, correct native-language web content exists and is retrievable for each target scholar. The paper's own numbers bound how often this assumption holds: the correct-Chinese-name identification rate is 87% and the associated recall is 74% (Tables 4-5), and even the best overall recall with native-name retrieval and email supplements is 96% (Table 9). Table 10 reports an unconditional disambiguation accuracy of 98%, which is difficult to reconcile with these retrieval ceilings unless the disambiguation test set excludes cases where retrieval failed. The paper does not state that such filtering was applied, nor does it condition Table 10 on successful retrieval. This mismatch undermines the credibility of the headline accuracy figure.","section":"§2.2, §3.3.1, Tables 4-9 and 10"}],"minor_comments":[{"comment":"The reference list is incomplete: reference 3 is missing its final page range, reference 5 is truncated after \"CIKM ’16,\" and the GitHub URL in the Code availability section is just \"https://github.com/\" with no repository path.","section":"References and Code availability"},{"comment":"Figure 1 is referenced in §2.2 but is not present in the manuscript text provided; please ensure the figure is actually included.","section":"Figure 1"},{"comment":"The percentages in Tables 4-7 are reported without denominators or definitions, making it impossible to judge the statistical meaning of differences such as 87% versus 95%.","section":"Tables 4-7"},{"comment":"The mathematical formulation in §2.3 introduces F_search, F_extract, and F_compare but does not specify their inputs or outputs precisely, and the objective is not used in any subsequent experiment; consider removing it or making it formally precise.","section":"§2.3"},{"comment":"There are several typos and awkward phrasings, including \"Retrival\" (§2), \"trival\" (§3.2), and the sentence in §3.1 beginning \"Although LLM do relatively easy tasks in these scenarios, yet there might exist some misunderstandings.\" These should be corrected in a revision.","section":"Various"}],"recommendation":"reject","confidential_remarks":"The paper reads like an early draft: the references and code link are incomplete, the figures appear to be missing, and the main empirical claim in Table 10 is stated without any experimental protocol. An editor should be aware that the reported accuracy numbers cannot be checked from the manuscript as submitted, and that the retrieval experiments do not provide disambiguation results. This is not a disagreement with the general research direction; it is a failure of the current manuscript to provide a verifiable central result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely useful part of this paper is the retrieval study. The authors systematically compare three search engines and two LLMs for finding Chinese scholars' profiles, with pinyin-only, translated-institution, native-name, and email-augmented queries. The recall jumps are plausible and practically informative—native-name search roughly doubles recall over pinyin plus bilingual institution. That is a real, if incremental, engineering contribution, and the workflow is clearly described.\n\nThe problem is the paper's central claim. Table 10 reports 98% and 86.6% disambiguation accuracy for GPT-4o and Hunyuan, and the abstract and conclusion lean on those numbers. But Table 10 has no dataset description, no positive/negative pair counts, no baseline, no definition of accuracy, no human-expert accuracy to compare against, and no error analysis. The scoring rule in §3.4 (institution match 2 points, repeated work/education segment 3 points, keywords 1–4, threshold 7) is hand-set, with no sensitivity analysis. In effect, the one direct disambiguation result is an unverifiable assertion. The retrieval tables are interesting but they are not a substitute: they measure profile recall, not whether two ambiguous names are correctly resolved.\n\nOther soft spots are smaller. The code and data link is a bare 'https://github.com/'—no repo. That is a broken reproducibility promise. The assumption that native-language web content exists and is retrievable is reasonable for Chinese scholars, but the reported 87% name-identification rate and 74% recall already show the bound; Table 10 does not condition on retrieval success, which would matter. Related work is also thin: no prior multilingual/LLM disambiguation system is cited, so it is hard to see exactly what is new beyond the recipe.\n\nThe paper is not nonsense and there is no circularity beyond the usual self-fit on a threshold. The authors seem to know what they are doing at the retrieval level. What is missing is a proper disambiguation evaluation. I would not cite the accuracy numbers, and I would not use this as evidence for the central claim. But a serious referee could ask for the dataset, a real baseline, and sensitivity analysis, and the retrieval component might survive. So I would send it out rather than desk reject, with an expectation of major revision.\n\nRecommendation: engage with it as a workshop-level technical report, but treat Table 10 as advertising until the underlying evaluation is supplied.","headline":"Sensible multilingual retrieval recipe undermined by a headline disambiguation number that has no dataset or baseline behind it.","tokens_in":59,"tokens_out":2549,"would_cite":false,"duration_ms":46581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that searching the web in a scholar's native language, after an LLM infers the native form of the name, sharply improves automatic scholar-name disambiguation.","keywords":["scholar name disambiguation","search-enhanced LLM","multilingual retrieval","native name inference","retrieval-augmented generation","Chinese scholar disambiguation","entity disambiguation","profile extraction"],"falsifier":"Run the same pipeline on a cohort of scholars with minimal native-language web presence, such as early-career researchers with no personal homepage and few native-language news mentions, and measure recall; if recall drops to the English-institution baseline, the native-name search step is the real cause of the reported gains.","tokens_in":81,"feed_emoji":"🌐","tokens_out":5523,"duration_ms":163828,"temperature":0.7,"pith_summary":"The paper tries to establish that the bottleneck in scholar name disambiguation is not comparison but retrieval: same-name scholars are hard to separate because English-only searches miss the richest evidence. It proposes a pipeline of search-enhanced LLM agents that translate an affiliation, infer the scholar's native-language name from search results, search again in that language, and then compare structured profiles. On its test set the pipeline lifts recall from a 40-49% English-only baseline to 96%, and final disambiguation accuracy reaches 98% for GPT-4o and 86.6% for Hunyuan, which the authors say approaches human-expert accuracy. If true, this makes large-scale automation of award-list matching, CV extraction, and paper-author disambiguation practical for multilingual cohorts.","feed_headline":"Native-language web search lifts scholar disambiguation to 98%","feed_subtitle":"Retrieving Chinese profiles instead of pinyin-only pages raises recall from 49% to 96% and matches expert accuracy.","key_machinery":"The machinery is a set of three LLM agents wrapped around a search engine: an Extract Agent that rewrites queries and pulls structured profiles from web results, a Name Translate Agent that converts an English affiliation into the native language and infers the scholar's native name, and a Disambiguation Agent that compares profiles. The load-bearing mechanism is language matching in search: search engines return pages in the query's language, and for Chinese scholars those native-language pages contain richer biographical detail than English pages, especially when the romanized pinyin name is ambiguous. A fixed scoring rule (institution 2, repeated education/work segment 3, keyword relevance 1-4, threshold 7) converts profile similarity into a same-person decision.","core_discovery":"The central claim is that adding a native-language retrieval step fixes what English-only RAG misses. Starting from a romanized name and English affiliation, the Name Translate Agent translates the institution, asks the LLM to infer the scholar's native name from returned pages, then re-queries with that native name plus the native institution. With Chinese scholars this native-name identification reaches 87% correctness and 74% recall, and adding the author's email pushes native-name recall to 95% and overall scholar-information recall to 96%. The Disambiguation Agent then scores profile matches by institution (2 points), repeated education/work segments (3 points), and research keywords (1-4 points), declaring a match at 7 points, and achieves 98% (gpt4o) and 86.6% (Hunyuan) accuracy on the disambiguation test set.","pith_inferences":["If the gains come from search-engine language matching rather than from the LLM, then the method's success will depend on native-language web coverage; early-career researchers or those in less digitized fields may see recall fall back toward the English baseline.","The 98% figure is reported on a test set evaluated with human experts; a larger, independently labeled sample would be needed before treating it as a general accuracy guarantee.","A natural extension is to replace the LLM's name inference with a transliteration model and to add knowledge-base ground truth to audit both recall and precision.","The same language-switch trick might transfer to other named-entity disambiguation tasks, but the direction of transfer is not tested in the paper."],"forward_implications":["Institution translation alone raises recall from the English baseline, confirming that query language, not just query content, matters.","Native name inference plus re-search yields 96% overall recall, so most Chinese scholars can be profiled automatically.","With email added, native-name recall rises to 95%, making email a practical disambiguation signal.","The 7-point scoring threshold operationalizes expert judgment in a transparent, reproducible way.","Accuracy of 98% (gpt4o) and 86.6% (Hunyuan) supports using LLM-based disambiguation in production bibliometric pipelines."],"supporting_citations":[{"why":"Supplies the crowdsourcing baseline for human-assisted author disambiguation that the paper contrasts with automated agents.","marker":"[1]"},{"why":"Provides the AMiner human-in-the-loop disambiguation framework and the F1 improvements the new method is positioned against.","marker":"[2]"},{"why":"Supplies the WhoIsWho benchmark and leaderboard context showing that metadata-only LLM solutions left performance gains on the table.","marker":"[3]"},{"why":"Supplies the GPT-4 model that the paper uses as one of its two LLM agents and the basis for its multilingual capabilities.","marker":"[4]"},{"why":"Supplies the query-rewriting result that motivates using search-engine query rewriting inside the agents.","marker":"[5]"}],"fun_headline_variants":["Native-language search lifts scholar disambiguation to 98%","Multilingual retrieval raises scholar name ID to 98%","98% accuracy with native-name search for scholars","Search in local language boosts disambiguation to 98%","Cross-lingual search achieves 98% on scholar names"],"cache_read_input_tokens":9216,"weakest_assumption_plain":"The method assumes that rich, correct web pages about the target scholar exist in their native language and that a search engine will return them when queried with the translated institution and inferred native name.","fun_headline_variants_meta":{"raw":{"variants":["Native-language search lifts scholar disambiguation to 98%","Multilingual retrieval raises scholar name ID to 98%","98% accuracy with native-name search for scholars","Search in local language boosts disambiguation to 98%","Cross-lingual search achieves 98% on scholar names"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1331,"prompt_tokens":884,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":500,"tokens_out":447,"duration_ms":4494,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:29:59.672456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a cohort of scholars with minimal native-language web presence, such as early-career researchers with no personal homepage and few native-language news mentions, and measure recall; if recall drops to the English-institution baseline, the native-name search step is the real cause of the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the crowdsourcing baseline for human-assisted author disambiguation that the paper contrasts with automated agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WhoIsWho benchmark and leaderboard context showing that metadata-only LLM solutions left performance gains on the table."},{"cited_title":"Learning to rewrite queries","cited_arxiv_id":null,"evidence_quote":"Supplies the query-rewriting result that motivates using search-engine query rewriting inside the agents."}],"review_version":1}