{"id":"f2db5e57-dfc4-4215-b881-36fe670eda34","arxiv_id":"2412.05708","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A retrieval, neutralization, and machine-translation pipeline can add relevant factual text to Hindi Wikipedia biography sections, but the claimed 65% and 62% gains are not fully supported by the reported evaluation.","lead":"This paper builds a pipeline that finds relevant passages in English books and Wikipedia, rewrites them in Wikipedia's neutral style with a large language model, and machine-translates them into Hindi to fill gaps in Hindi Wikipedia biography articles. The authors report that the added Hindi content is judged more informative than existing sections, but the headline improvement figures rest on a length-sensitive automatic metric and a small human study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 65%/62% enhancement figures are not derivable from the reported evaluations; the automatic 'Informativeness' metric is a length proxy, so the claimed gain may be an artifact of added text volume.","rationale":"The reader identified factual correctness as the weakest assumption, which is a serious concern for an encyclopedia. However, the paper's headline claim is the 65% and 62% enhancement numbers. If those numbers are not valid, the central claim is unsupported regardless of whether the content is factually correct. The automatic E-A-T metric defines Informativeness in terms of page size and word/sentence counts; since cnew is a superset of cold, this metric must increase, mechanically inflating the composite Quality. The human evaluation uses a relative 1-3 scale and reports averages, but never states how those averages yield 62%. The paper also lacks baseline comparisons, such as simply translating the full English section, so the added complexity of the multi-stage pipeline is not justified. These issues are concrete and testable: recomputing Quality after removing length-based features would show whether the 65% is an artifact, and requesting the exact formula for the human 62% from raw annotations would settle whether that number exists. Because the paper's own evidence is not connected to its headline, the claim is currently unverifiable. The reader's CONDITIONAL verdict remains appropriate, but the specific load-bearing concern is the validity of the evaluation metrics, not only factual accuracy. Therefore, the verdict should stay CONDITIONAL pending the requested re-analysis, and the agreement with the reader's weakest assumption is partial rather than full.","tokens_in":11326,"tokens_out":8173,"duration_ms":79977,"concrete_test":"Obtain per-section automatic evaluation scores from the authors or the GitHub repository. Recompute the E-A-T composite Quality after redefining Informativeness to exclude raw page/sentence/word counts (e.g., replace it with a length-normalized metric or a content-overlap measure such as ROUGE-L against the English source). If the '65% improvement' in Quality no longer appears large and positive, the automatic headline is an artifact of added text volume rather than evidence of effective knowledge transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on two percentages in the abstract: 65% automatic and 62% human. Neither number is defined or computed in the paper. Section 4.1 uses the E-A-T composite Quality, whose 'Informativeness' component is explicitly page size, #sentences, #words, and #complex-words. Since the new content cnew is formed by appending translated sentences to existing cold, Informativeness mechanically increases, inflating the composite Quality. The reported Qual increase of roughly 62-67% therefore does not demonstrate knowledge gain; it may only reflect added length. Section 4.2 reports human averages on a 1-3 improvement scale (2=no change, 3=improvement) as 2.3-2.7, but no formula maps these averages to 62%, and no fraction of 'improvement' ratings is provided. Without an explicit, reproducible mapping from the raw evaluation data to the headline percentages, the central claim is unfalsifiable. This is load-bearing because if the automatic metric is a length proxy, the '65% enhancement' is an artifact, and if the human 62% cannot be reconstructed, the quantitative evidence for 'effective transfer' is absent. The reader's factual-accuracy concern is valid, but the deeper problem is that the reported metrics do not measure what they claim to measure, independent of content correctness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WikiTransfer, a pipeline that enriches Hindi Wikipedia biography sections by transferring content from English Wikipedia and external book biographies. For English articles of FA quality, relevant English sections are matched to Hindi sections and machine-translated with IndicTrans2; for lower-quality English articles, content is first extracted from external biographies using RAG, debiased to conform to Wikipedia's NPOV policy via Llama-3(70B) in-context learning, and then translated. The paper claims that the framework enhances Hindi Wikipedia articles by 65% and 62% according to automatic and human evaluations, respectively. Automatic evaluation uses the E-A-T framework's Quality score on reverse-translated Hindi text, and human evaluation uses seven annotators rating informativeness, readability, and coherence on a 1–3 improvement scale.","tokens_in":11605,"tokens_out":4197,"duration_ms":40960,"significance":"If the quantitative claims were supported, the paper would address a real and practical problem: reducing content gaps in low-resource Wikipedia editions while preserving existing human-authored text. The pipeline is lightweight, uses open-source components, and the authors provide a manually curated set of 103 biographical writings and a code repository. The human evaluation includes a moderate-to-substantial Fleiss kappa, and the NPOV debiasing is evaluated separately on a neutralization benchmark. These are useful resources. However, the central quantitative claims are not currently supported by the reported experiments: the automatic metric conflates content length with informativeness, and the human improvement percentage cannot be reconstructed from the reported scales. The paper's significance is therefore conditional on re-analysis and additional evaluation.","major_comments":[{"comment":"The headline 65% automatic improvement is not defined or computed anywhere. Table 4 reports mean E-A-T Quality scores for cold and cnew, but no formula or percentage is given; a reader who computes the ratio from the table obtains roughly 62% for FA articles and 67% for non-FA articles, while the abstract states 65%. More importantly, the automatic Quality score includes 0.255 × Informativeness, and Informativeness is defined as a weighted sum of page size, sentence count, word count, and complex-word count. Since cnew is constructed by appending translated sentences to cold, these counts must increase by construction, so a substantial part of the measured Quality gain is guaranteed regardless of the semantic value of the added content. The additional reverse-translation step also introduces machine-translation artifacts into the scores. Please report the exact calculation of the 65% figure and add length-controlled or per-sentence evaluation measures that do not reward added text volume alone.","section":"Abstract and §4.1"},{"comment":"The headline 62% human-judgment improvement is not reproducible from the reported data. Evaluators used a 1–3 scale in which 1 means decline, 2 means no change, and 3 means improvement, and the paper reports only mean scores of 2.3–2.7 for the three metrics. No formula is given that maps these averages to 62%, and the distribution of ratings is not reported. Without the fraction of 'improvement' ratings, the statement 'enhancing ... by 62% according to ... human judgment-based evaluations' is unfalsifiable. Please report the percentage of sections rated 3 for each metric, or otherwise specify the exact transformation from the 1–3 ratings to the claimed percentage.","section":"§4.2"},{"comment":"Factual accuracy of the inserted content is never evaluated. The pipeline rephrases sentences from Archive.org biographies with Llama-3(70B) ICL and then translates them with IndicTrans2, but the paper does not check for hallucination, mistranslation, or outdated facts before insertion, and it does not report whether the added content is supported by citations. Section 6 acknowledges only that manual verification is subjective and related to relevance, not accuracy. Because inserting plausible but incorrect content can be more harmful to an encyclopedia than missing content, the paper should either measure the factuality and citation support of the generated text or explicitly scope the claims to style, coherence, and informativeness rather than to accurate knowledge transfer.","section":"§3.3 and §4.2"},{"comment":"The section-matching and content-matching thresholds are estimated on the same data to which they are applied: the title similarity threshold is set to the mean of the computed similarities, and the content threshold uses µ + σ of the same distribution. No held-out validation or error analysis of the mapping is reported. If these thresholds overfit the dataset, the section-pair selection and the subsequent augmentation could be biased. Please validate the mapping on a held-out set or report mapping accuracy based on human assessment.","section":"§3.1"}],"minor_comments":[{"comment":"The caption of Table 4 says 'Human evaluation on the generated machine-translated Hindi content', but the table reports automatic E-A-T scores; please retitle it as an automatic evaluation table.","section":"§4.1, Table 4"},{"comment":"The final sentence of Appendix C says 'mirroring the results shown in Table 5' but the intended cross-reference is Table 4; please correct the reference.","section":"Appendix C"},{"comment":"The weights in the Quality formula sum to 1.466 (0.255 + 0.654 + 0.557), not to 1; please clarify whether the weights are copied from the original E-A-T paper or whether they should be normalized.","section":"§4.1"},{"comment":"The text says 'We have performed our experiments with LLama-3(8B) instruct model ... for both these setups' but later uses Llama-3(70B) for the ICL few-shot setup; please clarify which model is used in each configuration.","section":"§3.3"},{"comment":"The manual verification of Archive.org links was performed by a single post-graduate student; please report whether a second annotator was involved and provide inter-annotator agreement for this step.","section":"§2"},{"comment":"The paper uses 'IndicTrans' when translating section titles and 'IndicTrans2' for content translation; please disambiguate these two models explicitly.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a useful problem and provides a reasonable pipeline and reusable dataset, but the central quantitative claims need to be reconstructed and re-evaluated. I recommend major revision rather than rejection because the deficiencies are fixable: the authors can report explicit percentage formulas, replace or supplement the length-proxy automatic metric, provide the human rating distribution, and add a factuality check. If these points are not addressed, the abstract's 65%/62% claims should be removed or substantially qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely useful: enrich existing Hindi Wikipedia sections by mapping section titles, pulling content from English Wikipedia or external biographies, neutralizing POV with an LLM, and machine-translating. I haven't seen this exact combination before, and the system itself is lightweight and practical. The authors also release code and data, which counts.\n\nWhat the paper does well: the human evaluation, though small, is real—35 sections, 7 Hindi speakers, Fleiss kappa of 0.53-0.61 shows moderate agreement. The NPOV correction evaluation is a sensible sanity check. The pipeline is end-to-end and reproducible.\n\nNow the soft spots. The headline numbers don't hold up. The 65% automatic figure is never explicitly computed in the text. Looking at Table 4, the composite Quality score rises from 26.17 to 42.39 for FA articles (62%) and from 28.74 to 47.99 for non-FA (67%), averaging about 65%. But Quality is defined as 0.255*Informativeness + 0.654*Readability + 0.557*Understandability, and Informativeness is page size, #sentences, #words, and #complex-words. Since cnew is formed by appending translated sentences to cold, Informativeness mechanically rises. So the \"65% enhancement\" is largely a length effect, not a knowledge gain. The human 62% is worse: no mapping from the 1-3 scale averages (2.3-2.7) to a 62% improvement is given anywhere.\n\nThere are also no baselines. A trivial baseline like translating the full English article with IndicTrans2 would presumably inflate length metrics too. And factual accuracy of the LLM-neutralized, machine-translated content is never checked—the Limitations section admits manual verification is subjective but doesn't address the risk of hallucinated or stale facts in an encyclopedia.\n\nThe stress-test note is fair on the metrics. The reader's factual-accuracy concern is valid but secondary; the measurement problem is more fundamental.\n\nWho this is for: NLP researchers working on multilingual Wikipedia or low-resource content generation, and Wikipedia editors looking for practical tooling. It deserves a serious referee, but the authors need to recompute and justify the headline numbers, add baselines, and verify a sample of facts before the central claim can be accepted.","headline":"Useful pipeline for enriching Hindi Wikipedia sections, but the 65%/62% enhancement claims are not actually derived in the paper—the automatic metric is a length proxy and the human number can't be reconstructed.","tokens_in":12108,"tokens_out":2181,"would_cite":false,"duration_ms":20496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multistage pipeline called WikiTransfer automatically enriches Hindi Wikipedia biography sections with content translated or adapted from English, reporting 65% and 62% improvements in automatic and human evaluations.","keywords":["Wikipedia knowledge equity","low-resource languages","Hindi Wikipedia","cross-lingual content transfer","retrieval-augmented generation","neutral point of view","machine translation","LLM in-context learning"],"falsifier":"Take 100 Hindi sentences added by the pipeline, back-translate them to English, and ask two bilingual fact-checkers to verify each factual claim against the source biography and the English Wikipedia article; if a substantial fraction (say, more than 5%) contains invented dates, places, or claims, the framework is filling gaps with plausible but unreliable content rather than trustworthy encyclopedia prose.","tokens_in":11135,"feed_emoji":"🌐","tokens_out":9114,"duration_ms":73300,"temperature":0.7,"pith_summary":"This paper tries to establish that a lightweight, fully automatic pipeline can reduce Wikipedia's English-to-Hindi content gap by adding new sentences to existing Hindi biography sections rather than generating articles from scratch. The pipeline first maps English and Hindi section headings, then either translates enriched English content directly or retrieves relevant passages from external biographies, rewrites them into neutral encyclopedia prose with a large language model, and translates the result into Hindi. The authors report that the added content improves Hindi articles by 65% on automatic quality metrics and 62% in human evaluations of informativeness, readability, and coherence. If the claim holds, Hindi Wikipedia editors gain a practical tool for filling missing sections while preserving the human-written text already there.","feed_headline":"65% more content for Hindi Wikipedia via an English-to-Hindi pipeline","feed_subtitle":"A pipeline that translates English sections and rewrites external biographies adds readable, neutral Hindi sentences.","key_machinery":"The load-bearing mechanism is the WikiTransfer pipeline, a modular chain whose stages are: (1) semantic section mapping, which pairs Hindi and English section titles by embedding cosine similarity; (2) content augmentation, which translates English sentences with IndicTrans2 and filters out short or redundant translations; (3) external knowledge injection, which uses retrieval-augmented generation over a digital library of biographies to pull relevant chunks for English articles that are not yet comprehensive; and (4) point-of-view correction, which prompts Llama-3(70B) in a few-shot in-context learning setup to rewrite biased sentences as neutral encyclopedia prose. The key identity the argument depends on is the claim that a sentence which is relevant under retrieval, neutral under the language-model rewrite, and not redundant under the similarity filter will also be coherent and accurate once translated into Hindi.","core_discovery":"The central claim is that section-level knowledge transfer from English to Hindi Wikipedia works best as a modular pipeline rather than as from-scratch generation. For English articles already at featured-article quality, WikiTransfer translates matched sections directly with IndicTrans2; for lower-quality English articles, it first retrieves relevant chunks from external biographical writings using retrieval-augmented generation, then uses Llama-3(70B) with few-shot in-context learning to remove framing and epistemological bias, and only then translates. The paper argues that this two-path design lets high-quality English content be reused directly while lower-quality content is upgraded before transfer. The evidence offered is that the machine-generated Hindi sentences score higher than the existing Hindi sentences on informativeness, readability, coherence, and an automatic E-A-T based quality measure, with the headline gains of 65% (automatic) and 62% (human).","pith_inferences":["The paper does not verify the factual accuracy of the final Hindi text; a natural extension would be to add citation-backed fact-checking before insertion, which would also make the pipeline deployable on live Wikipedia.","The section-mapping threshold and redundancy filter are tuned only for English-Hindi; the same pipeline may need recalibration for typologically distant language pairs, a testable question the paper leaves open.","The neutrality evaluation measures whether bias is removed but not whether the rewrite preserves every factual detail, so the neutrality-rewriting step may sometimes drop nuances a human editor would keep.","A baseline experiment not reported in the paper—comparing the full pipeline with direct translation of the English section alone—would isolate how much of the 65% gain comes from external knowledge retrieval versus translation."],"forward_implications":["Hindi Wikipedia editors can enrich underdeveloped biography sections by running the pipeline and lightly reviewing the output, rather than writing new prose from scratch.","The framework transfers to other high-resource/low-resource language pairs whenever a digital corpus on the topic exists to feed the retrieval step.","Because the pipeline appends new sentences and never overwrites human-authored text, it fits Wikipedia's collaborative editing model better than from-scratch article generation.","External biographical books, not just the English Wikipedia article itself, are shown to be a viable source for closing low-resource content gaps."],"supporting_citations":[{"why":"Supplies IndicTrans2, the machine translation model that converts English content into Hindi.","marker":"Gala et al., 2023"},{"why":"Provides the biased/neutral sentence corpus used to train and evaluate the neutrality-rewriting step.","marker":"Pryzant et al., 2020"},{"why":"Defines framing and epistemological bias, the two bias types the point-of-view rectifier is designed to remove.","marker":"Recasens et al., 2013"},{"why":"Provides the E-A-T based quality metric used for automatic evaluation of generated Hindi content.","marker":"Sugandhika and Ahangama, 2022"},{"why":"Supplies the multilingual Wikipedia biography dataset that anchors the English-Hindi article pairs.","marker":"Beytía et al., 2022"},{"why":"Provides language-agnostic quality scores used to identify Hindi articles that need enrichment.","marker":"Das et al., 2024"},{"why":"Supplies the biased sentences used as test data for neutrality assessment.","marker":"Nangia et al., 2020"},{"why":"Provides BERTScore, one of the reference-based metrics used to compare debiasing methods.","marker":"Zhang et al., 2019"}],"fun_headline_variants":["Hindi Wikipedia gets 65% more content via smart transfer","English-to-Hindi Wikipedia transfer boosts content by 65%","Framework lifts Hindi Wikipedia quality by 62%","Modular pipeline enriches Hindi Wikipedia sections","From English to Hindi: WikiTransfer fills gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that every sentence that survives retrieval, rewriting, and translation still states facts correctly, but it never checks this before inserting the text into Wikipedia.","fun_headline_variants_meta":{"raw":{"variants":["Hindi Wikipedia gets 65% more content via smart transfer","English-to-Hindi Wikipedia transfer boosts content by 65%","Framework lifts Hindi Wikipedia quality by 62%","Modular pipeline enriches Hindi Wikipedia sections","From English to Hindi: WikiTransfer fills gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2802,"prompt_tokens":914,"completion_tokens":1888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1813}},"tokens_in":530,"tokens_out":1888,"duration_ms":12951,"temperature":1.0,"reasoning_tokens":1813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:26:25.289380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 100 Hindi sentences added by the pipeline, back-translate them to English, and ask two bilingual fact-checkers to verify each factual claim against the source biography and the English Wikipedia article; if a substantial fraction (say, more than 5%) contains invented dates, places, or claims, the framework is filling gaps with plausible but unreliable content rather than trustworthy encyclopedia prose.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines framing and epistemological bias, the two bias types the point-of-view rectifier is designed to remove."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the E-A-T based quality metric used for automatic evaluation of generated Hindi content."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides language-agnostic quality scores used to identify Hindi articles that need enrichment."}],"review_version":1}