{"id":"d997e596-b336-431a-950b-caaf2614bcce","arxiv_id":"2505.12973","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HomoRich, a 528,891-sentence Persian dataset, improves homograph disambiguation by about 30 percentage points in both a fine-tuned T5 model and a context-aware version of eSpeak.","lead":"This paper builds a large Persian dataset of sentences labeled with pronunciation, then uses it to improve both a neural and a rule-based text-to-speech system on same-spelling, different-pronunciation words. It reports about 30 percentage-point gains in homograph accuracy for both approaches and argues that fast rule-based tools can be upgraded with good data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluate SentenceBench independence before trusting the ~30-point homograph gains; the paper's central claim rests on a self-authored, undocumented test set.","rationale":"The reader's weakest_assumption is the evaluation benchmark's independence, and the paper's text strongly supports that this is the right place to probe. Every headline number in Table 3 is computed on SentenceBench, a dataset introduced by the same authors in their previous ICASSP work. The current paper states that SentenceBench was 'adopted' from that work but gives no annotation protocol, no inter-annotator agreement, and no evidence that its labels were verified independently of the GPT-4o method used to phonemize HomoRich. Appendix H is unusually candid: it acknowledges that GPT-4o has 6.43% PER and 64% homograph accuracy, and that 'no specific validation was performed on the LLM-generated or human-provided data.' That admission does not by itself invalidate the results, but it makes the shared-bias scenario concrete rather than speculative. If HomoRich and SentenceBench were both shaped by the same GPT-4o prompt style, then a model fine-tuned on HomoRich could inflate its test score by learning the labeler's systematic preferences, and the statistical context-word lists in HomoFast eSpeak could encode the same preferences. Under that scenario, the 30-point gains would not transfer to real Persian text, where a human listener's notion of correct pronunciation is the ground truth. The proposed test—independent native-speaker re-annotation of a SentenceBench sample—directly breaks the circularity because it measures agreement with human judgment rather than with the original labels. A simpler contamination check, computing overlap between SentenceBench and HomoRich sentences, would also be informative, but the re-annotation test is the decisive one. The paper does deserve credit for releasing HomoRich, Homo-GE2PE, and HomoFast eSpeak under permissive licenses, for reporting standard deviations, and for including learning curves and datasheets. Those are genuine strengths. The concern here is not about authorship or intent; it is about whether the evaluation set can support the central quantitative claim. Because the issue is unresolved rather than disproven, the appropriate verdict remains conditional: the paper should be accepted only after the benchmark's independence is documented or the gains are confirmed on an externally validated set. This is exactly the reader's conditional verdict, so no change is needed, but the concrete re-annotation test sharpens the condition that should be met.","tokens_in":22873,"tokens_out":3038,"duration_ms":33511,"concrete_test":"Download SentenceBench and independently re-annotate a random sample of at least 100 of its sentences for the target homograph pronunciations, using two native Persian speakers with adjudication and without showing them the released labels. Recompute homograph accuracy on this independently labeled subset for GE2PE, Homo-GE2PE, eSpeak, and HomoFast eSpeak with the same inference settings as Table 3. If Homo-GE2PE's advantage over GE2PE shrinks from +29.72 points to below roughly +10 points, or if HomoFast eSpeak's advantage over eSpeak similarly collapses, then the reported gains are largely benchmark artifacts rather than real disambiguation improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that HomoRich data yields roughly 30-point homograph-disambiguation gains for both a neural model and a rule-based variant—rests entirely on Table 3, which is computed on SentenceBench, a 400-sentence benchmark created by the same group in prior work (Qharabagh et al., 2025a). The paper does not document how SentenceBench was annotated, whether native-speaker adjudication was used, or whether the benchmark was kept independent of the GPT-4o phonemization pipeline used to label HomoRich. Section 3.1 reports that GPT-4o itself achieves only 64% homograph accuracy and 6.43% PER, and Appendix H explicitly states that no validation was performed on the LLM-generated or human-provided data beyond the collection methods described. If SentenceBench labels were produced or influenced by the same GPT-4o prompting pipeline, then the training labels and the evaluation labels share systematic errors. In that case, the reported improvements of +29.72 (Homo-GE2PE vs. GE2PE) and +30.66 (HomoFast eSpeak vs. eSpeak) could reflect agreement with a biased labeler rather than genuinely correct disambiguation, and the paper's thesis that rich offline data upgrades fast rule-based G2P would be unverified. The exact weighted-overlap formula for HomoFast eSpeak is also unspecified, but the benchmark-independence issue is the more load-bearing weakness because every headline number depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Persian grapheme-to-phoneme (G2P) homograph disambiguation. It introduces HomoRich, a 528,891-sentence Persian dataset built from human-written and GPT-4o-generated sentences plus CommonVoice, ManaTTS, and GPTInformal, with LLM-based phoneme labels, and it proposes three data-augmentation methods. The authors fine-tune the existing GE2PE model into Homo-GE2PE and augment the rule-based eSpeak NG engine with a statistical context-overlap disambiguation module to create HomoFast eSpeak. On the authors' SentenceBench benchmark (400 sentences), Homo-GE2PE raises homograph accuracy from 47.17% to 76.89% and HomoFast eSpeak from 43.87% to 74.53%, while reducing PER; Homo-T5, trained only on HomoRich, reaches 76.32% homograph accuracy. The paper argues that rich offline datasets can upgrade both neural and rule-based G2P systems for latency-sensitive accessibility applications.","tokens_in":23125,"tokens_out":6186,"duration_ms":64875,"significance":"If the reported results are correct, the paper is a valuable contribution to low-resource G2P and accessibility: it releases a permissively licensed dataset, a fine-tuned neural model, and an enhanced open-source rule-based engine, and it demonstrates that a simple statistical module can give eSpeak competitive homograph disambiguation while retaining low latency. Multi-run reporting with standard deviations and public code/scripts are strengths. However, the headline numbers rest entirely on SentenceBench, a self-authored 400-sentence benchmark whose construction is not documented; combined with the disclosed 64% homograph accuracy and 6.43% PER of the GPT-4o labeling pipeline, the independence and gold-standard quality of the evaluation are not established. The exact weighted-overlap formula for HomoFast eSpeak is also unspecified, impairing reproducibility. The claims are plausible and worth pursuing, but they require an independent or fully documented evaluation before they can be accepted at face value.","major_comments":[{"comment":"The central claim of roughly 30-point homograph-accuracy gains is evaluated exclusively on SentenceBench, a benchmark created by the same research group in prior work (Qharabagh et al., 2025a; Fetrat, 2024b). The manuscript never documents how SentenceBench was annotated, whether native-speaker adjudication was used, or whether its labels are independent of the GPT-4o phonemization pipeline used to label HomoRich. Appendix H explicitly states that no specific validation was performed on the LLM-generated or human-provided data. Because the training data and the evaluation data could share the same systematic label errors, the reported +29.72 and +30.66 point gains may reflect agreement with a biased labeler rather than correct disambiguation. Please evaluate on an external, native-speaker-adjudicated test set, or at minimum document SentenceBench's annotation process and demonstrate that its labels were not produced or influenced by the same LLM pipeline.","section":"§4, Table 3; Appendix H"},{"comment":"The proposed HomoFast eSpeak method is described as computing a 'weighted overlap' between the context words of a new sentence and each pronunciation's context list, with normalization by list length, but no equation, no definition of the weights, no context-window size, no stopword list, and no pseudocode are provided. This is the core mechanism behind the paper's second headline result (HomoFast eSpeak at 74.53% homograph accuracy), and the method is not reproducible as written. Please specify the full algorithm, including how the context lists are constructed and how the overlap scores are weighted and normalized.","section":"§3.2.2"},{"comment":"Dataset quality is a load-bearing premise: HomoRich is generated with an LLM pipeline whose reported homograph accuracy is only 64% and PER is 6.43%, and the datasheet states that no validation was performed on the LLM-generated or human-provided data beyond the collection methods described. Given that Homo-GE2PE and Homo-T5 are trained on these labels and evaluated on a same-group benchmark, the paper needs at least a manual error analysis of a random sample of HomoRich, with per-source error rates, to support the assertion that the data are 'high-quality.' Without such analysis, the contribution of data quality versus label noise or benchmark overlap cannot be separated.","section":"§3.1; Appendix H"}],"minor_comments":[{"comment":"The text describes the gains as 'approximate 30% improvement' and '30.66% improvement in homograph disambiguation,' but the reported changes are percentage-point increases (e.g., GE2PE from 47.17% to 76.89% is +29.72 points, which is a 63.0% relative improvement). Please correct the wording to 'percentage points' or state the relative improvement explicitly.","section":"Abstract; Conclusion"},{"comment":"Many citations appear as bare years in the text (e.g., '(2020)', '(2025a)', '(2024)') and are not resolved to proper author-year entries in the reference list. The manuscript needs a complete and consistent citation cleanup.","section":"§2; References"},{"comment":"The inference-time comparison between HomoFast eSpeak and eSpeak is not apples-to-apples because Section 4 states that HomoFast eSpeak benefits from 'a newly added feature that enables processing of larger text segments in a single run.' Please report per-sentence or otherwise controlled segmentation speed so the comparison reflects algorithmic efficiency rather than an implementation artifact.","section":"§4; Figure 9"},{"comment":"The three augmentation methods are described only qualitatively, with no examples or quality checks. Synonym replacement and sentence reordering can produce unnatural or ungrammatical sentences, and the claim that reordering does not affect pronunciation (except Ezafe) should be validated on a sample. Please provide examples and a small human or automatic evaluation of augmented data quality.","section":"§3.1.2"},{"comment":"SentenceBench contains only 400 sentences, but the paper does not report the number of homograph instances or per-homograph coverage. For a test set this small, reporting confidence intervals or a per-homograph breakdown would strengthen the claim that the 30-point gains are not driven by a few easy items.","section":"§4; Table 3"},{"comment":"The datasheet notes that the approximately 200 human participants were not compensated and that no consent-revocation mechanism was provided. For a released dataset, this should be discussed more explicitly, including whether ethical review is needed or was waived.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-independence concern raised in the stress-test note is real and is supported by the manuscript's own datasheet statements: there is no validation of the LLM-generated or human-provided data, and SentenceBench is a same-group resource. I would not reject the paper outright, because the dataset release and the demonstration that a statistical module can improve rule-based eSpeak are useful and the issue is fixable with an external evaluation or full documentation of SentenceBench. However, acceptance should hinge on resolving that issue and on specifying the weighted-overlap algorithm."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about low-resource G2P or accessibility TTS. The real contribution is HomoRich: 528k Persian sentence-phoneme pairs, 285 homographs, balanced across pronunciations, released under CC0, plus fine-tuned Homo-GE2PE weights and a modified eSpeak called HomoFast eSpeak. That is a tangible, reproducible resource, and the paper deserves credit for shipping it.\n\nThe results are internally consistent and the direction is genuinely promising. They report roughly 30-point gains in homograph accuracy for both a neural model and a rule-based system, and the rule-based variant stays fast. If those numbers hold up, the path of using rich offline data to upgrade a rule-based system without adding latency is exactly what screen-reader deployment needs.\n\nNow the soft spots, in proportion.\n\nFirst, the benchmark. Every headline number in Table 3 is computed on SentenceBench, a 400-sentence set from the authors' own prior work. The paper never documents how SentenceBench was annotated, whether it was kept independent of the GPT-4o phonemization pipeline used to label HomoRich, or whether native-speaker adjudication was used. The datasheet explicitly says no validation was performed beyond the collection methods. If the evaluation labels share systematic errors with the training labels, the +30 point gains are inflated. This is the load-bearing weakness. It is not fatal, because the resource itself stands on its own, but it needs to be addressed before the headline numbers are taken at face value.\n\nSecond, novelty framing. Section 3.2.2 claims that the context-overlap approach has not been explored in prior homograph research, but the paper's own related work cites Yarowsky's decision lists, Hearst's co-occurrence method, and Riahi and Sedghi's tri-training framework. The weighted-overlap formula is never specified, so a reader cannot tell what is actually new. Either present the formula and compare directly against a decision-list baseline, or drop the novelty claim.\n\nThird, minor. The human-subject appendix describes voluntary participants but no consent forms or IRB review, and it says consent was inherent in participation. That is thin ethics reporting. It is not a reason to reject, but it should be cleaned up.\n\nThe central thesis, that rich offline data benefits both neural and rule-based G2P, is plausible and the evidence, while benchmark-limited, is internally consistent. The dataset and code are open, which is what the community needs.\n\nWho is this for? Anyone working on Persian TTS, low-resource G2P, or latency-sensitive accessibility. It deserves a serious referee. I would send it to review with instructions to require either an external evaluation set or a documented annotation process for SentenceBench. I would also use the HomoRich dataset in my own future work.","headline":"A genuinely useful Persian homograph dataset and a plausibly large gain for rule-based G2P, but the self-authored benchmark keeps the headline numbers from being fully trustworthy.","tokens_in":23719,"tokens_out":1848,"would_cite":true,"duration_ms":18952,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single rich Persian dataset lifts homograph accuracy by roughly 30 points in both neural and rule-based G2P systems.","keywords":["Grapheme-to-phoneme conversion","homograph disambiguation","Persian","low-resource languages","rule-based G2P","text-to-speech","accessibility","LLM-generated data"],"falsifier":"Take a fresh set of Persian sentences containing the same 285 homographs, have native speakers annotate the intended pronunciations by hand (no LLM phonemes, no prior exposure to HomoRich), and run Homo-GE2PE, HomoFast eSpeak, and their baselines on it; if the ~30-point accuracy gap shrinks toward zero or the improved models fall below random accuracy on this independent set, the central claim fails.","tokens_in":22649,"feed_emoji":"🗣️","tokens_out":5660,"duration_ms":50835,"temperature":0.7,"pith_summary":"Homograph disambiguation—choosing the right pronunciation for words spelled the same—is the weak point of Persian grapheme-to-phoneme (G2P) systems, and the systems that need it most, like screen readers, cannot afford slow neural models. This paper argues that a large, balanced, openly licensed homograph dataset can fix both problems at once: it reports HomoRich, 528,891 phonemized Persian sentences covering 285 homographs, built with a semi-automated pipeline that combines human-written examples, LLM-generated sentences, and public corpora. Fine-tuning the neural G2P model GE2PE on this data raises homograph accuracy from 47.17% to 76.89%. Adding a lightweight statistical context-overlap module to the rule-based eSpeak engine yields HomoFast eSpeak, which reaches 74.53% homograph accuracy while remaining the fastest tool tested—evidence that fast rule-based G2P can be upgraded with rich offline data rather than replaced.","feed_headline":"One dataset lifts Persian text-to-speech homograph accuracy ~30 points","feed_subtitle":"A fast rule-based eSpeak now nearly matches a neural model on tricky Persian pronunciations.","key_machinery":"The load-bearing mechanism is the HomoRich dataset itself, built by filtering a Persian pronunciation dictionary for 285 homographs and generating balanced, context-diverse sentences; that dataset feeds a per-pronunciation context-word database with normalized weighted-overlap scoring, a statistical disambiguator with no neural components or embeddings. A second mechanism is the LLM-powered phonemization pipeline—prompting GPT-4o with Finglish transcriptions, few-shot examples, and dictionary hints—which turns raw sentences into phoneme labels at 6.43% phoneme error rate and 64% homograph accuracy, cheaply enough to make large-scale homograph data feasible in a low-resource language.","core_discovery":"The paper's central claim is that data quality can substitute for model complexity in homograph disambiguation: the same curated dataset that pushes a neural model to 76.89% homograph accuracy also lets a purely statistical, embedding-free module inside eSpeak reach 74.53%, a 30.66-point gain over eSpeak's 43.87%, with no loss of speed. The mechanism is a per-pronunciation database of context words built from HomoRich; for each incoming sentence the system scores the overlap between its content words and each pronunciation's context list, normalizes by list length, and picks the pronunciation with the highest score. The authors report the two improved systems—Homo-GE2PE and HomoFast eSpeak—as direct evidence that rich offline datasets can modernize rule-based G2P for real-time accessibility applications, not just train larger neural models.","pith_inferences":["Editorial inference: because both the training labels and the SentenceBench test set come from the same LLM phonemization pipeline, the reported gain could partly measure how well the models reproduce that pipeline's biases; an independently human-annotated test set is the natural check.","Editorial inference: the context-overlap approach is a modern restatement of classic decision-list homograph disambiguation, and its success here suggests that older statistical methods deserve revisiting in low-resource languages once balanced data exist.","Editorial inference: a stronger test of the central thesis would be to swap the LLM-generated sentences out of HomoRich and retrain; if homograph accuracy falls sharply, the recipe's value lies in LLM data generation, and if it holds, human sentences carry most of the signal.","Editorial inference: the method's restriction to 285 curated homographs means it will not handle rare or unseen homographs; extending the pipeline to open-vocabulary homograph discovery would be the next step."],"forward_implications":["If the reported numbers hold, a rule-based G2P engine can give up almost none of its speed while gaining roughly 30 points of homograph accuracy, making screen-reader quality in Persian materially better without new hardware.","The HomoRich dataset gives Persian TTS and accessibility research a public, license-free resource, lowering the barrier for reproducing or extending the results.","The same three-phase fine-tuning recipe—general G2P, LLM homograph sentences, human homograph sentences—should transfer to other low-resource languages that can assemble a balanced homograph corpus.","The statistical context-overlap module can be dropped into other fast rule-based G2P systems, not only eSpeak, whenever an annotated homograph corpus exists.","Homo-T5's competitive results from a dataset an order of magnitude smaller than the synthetic GE2PE corpus suggest the bottleneck is homograph balance and context diversity, not raw data volume."],"supporting_citations":[{"why":"State-of-the-art neural Persian G2P model that is fine-tuned into Homo-GE2PE and provides the baseline homograph accuracy and PER.","marker":"GE2PE (2024)"},{"why":"Open-source rule-based TTS engine targeted for the HomoFast eSpeak enhancement and the speed baseline.","marker":"eSpeak (2024)"},{"why":"Priori LLM-powered Persian G2P work that supplies the Finglish-prompt phonemization method and the SentenceBench evaluation approach.","marker":"(2025a)"},{"why":"The sentence-level benchmark on which all reported homograph accuracy numbers are computed.","marker":"SentenceBench (Fetrat, 2024b)"},{"why":"Source dictionary filtered to identify the 285 Persian homograph words used in HomoRich.","marker":"KaamelDict (Fetrat, 2024a)"},{"why":"The LLM used both to generate homograph sentences and to phonemize the corpus.","marker":"GPT-4o (2024)"},{"why":"One of the Persian corpora folded into HomoRich to provide general G2P coverage.","marker":"CommonVoice (2019)"},{"why":"Persian TTS corpus contributing 76,561 phoneme-annotated sentences and diverse registers to HomoRich.","marker":"ManaTTS (2025b)"}],"fun_headline_variants":["Rich data narrows neural-rule gap for Persian homographs, 30% gain","Fast eSpeak nearly matches neural Persian G2P with rich data","Rule-based G2P gets 30% homograph boost from richer data","One dataset, two systems: Persian homographs jump 30%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that SentenceBench, the evaluation set on which every headline homograph number is computed, is a valid independent gold standard even though it was created by the same group with the same LLM phonemization pipeline that labeled the training data—and that GPT-4o's phonemization (6.43% PER, 64% homograph accuracy) is accurate enough to learn from; if either premise gives way, the reported ~30-point gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Rich data narrows neural-rule gap for Persian homographs, 30% gain","Fast eSpeak nearly matches neural Persian G2P with rich data","Rule-based G2P gets 30% homograph boost from richer data","One dataset, two systems: Persian homographs jump 30%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001063,"raw_usage":{"total_tokens":4451,"prompt_tokens":930,"completion_tokens":3521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3448}},"tokens_in":546,"tokens_out":3521,"duration_ms":27754,"temperature":1.0,"reasoning_tokens":3448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:22:27.584962+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of Persian sentences containing the same 285 homographs, have native speakers annotate the intended pronunciations by hand (no LLM phonemes, no prior exposure to HomoRich), and run Homo-GE2PE, HomoFast eSpeak, and their baselines on it; if the ~30-point accuracy gap shrinks toward zero or the improved models fall below random accuracy on this independent set, the central claim fails.","supporting_citations":[],"review_version":1}