{"id":"2f21fc7c-48cd-4576-a24f-257da375890b","arxiv_id":"2501.07088","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MathReader is a pipeline that translates LaTeX formulas extracted from PDFs into spoken English via a fine-tuned T5 model before text-to-speech, and reports lower WER than Edge and Acrobat.","lead":"MathReader combines OCR, a fine-tuned T5 model, and text-to-speech to read PDFs containing math formulas aloud, converting LaTeX into spoken English before synthesis. In tests, it reported lower word error rates than Microsoft Edge and Adobe Acrobat, which suggests a practical accessibility improvement for visually impaired readers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported WER advantage may stem from measuring reading accuracy via a single ASR engine whose errors are voice-dependent; no human-listener or multi-ASR validation is provided, so the central claim is not yet established.","rationale":"MathReader is a coherent pipeline (Nougat-small OCR, T5-small translation, VITS TTS) and the authors provide code and an ablation (MathReader without T5) that is consistent with the T5 module contributing to lower error rates. Those are real strengths. However, the headline numbers are only meaningful if WER reflects reading accuracy. Because WER is computed by a single ASR system on audio from different TTS voices, systematic ASR biases are a direct threat. The paper itself acknowledges homophone errors in STT, which are voice-dependent. The lack of test-set size and error bars amplifies the risk. The ground-truth transcriptions were authored by the same group that built the T5 training data, so the metric may partly reward matching their preferred spoken-math convention. A human-listener study with independently created ground truth would settle this. My verdict remains CONDITIONAL: the pipeline is credible and the ablation is informative, but the current evidence does not conclusively support the claimed WER advantage.","tokens_in":6723,"tokens_out":3421,"duration_ms":35047,"concrete_test":"Have at least three human transcribers listen to the released audio for a stratified sample (e.g., 50 pages) from MathReader, Edge, and Acrobat, and compute WER against a ground truth independently created by math-education experts. If MathReader's WER is not significantly lower (with bootstrap confidence intervals) than both baselines under human transcription, the single-ASR result is confounded and the central claim is unsupported. As a secondary check, run the same audio through a second commercial ASR engine and verify the ranking is preserved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on WER measured by Naver Clova Note STT on synthesized audio (Section IV-C). MathReader uses VITS, while Edge and Acrobat use their own TTS voices. ASR models are known to have different error rates on synthetic voices; homophone errors explicitly mentioned in the paper (e.g., y vs why) are exactly such ASR confounds. Thus the reported 0.281 vs 0.510/0.617 gaps could reflect ASR familiarity with VITS rather than better formula reading. The paper also reports no test-set size, no error bars, and no human transcription; the ground truth was typed by the authors using conventions aligned with their T5 training data. Without a voice-controlled or human-based evaluation, the numbers in Table II do not establish that MathReader reads formulas more accurately.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MathReader, a pipeline that converts PDF documents containing mathematical formulas into speech. It uses Nougat-small to OCR the PDF into a markdown/math markup file, extracts LaTeX formula segments using delimiter patterns, translates those formulas into spoken English with a fine-tuned T5-small model trained on the MathBridge corpus, replaces the LaTeX with the spoken English, and synthesizes speech with VITS. The authors evaluate MathReader against Microsoft Edge and Adobe Acrobat by measuring WER and CER of Naver Clova Note ASR transcriptions of the synthesized audio, reporting substantial improvements (WER 0.281 vs 0.510 and 0.617), plus an ablation showing worse performance without T5. The paper also reports a per-page runtime of about 23.6 seconds on an H100.","tokens_in":6973,"tokens_out":4002,"duration_ms":38364,"significance":"If the reported results are reliable, MathReader would be a practical, lightweight assistive tool for reading math-heavy documents, with modest compute requirements (T5-small, VITS). The paper addresses a real accessibility problem and makes code available. However, the current evaluation is not sufficient to establish the central claim: the WER measure is confounded with ASR voice biases, the test set is not described quantitatively, and the ground truth is author-generated with conventions aligned to the training corpus. The contribution is therefore promising but not yet validated.","major_comments":[{"comment":"The evaluation in Section IV-C computes WER from transcripts produced by a single ASR system (Naver Clova Note) applied to audio synthesized with different TTS voices. Because MathReader uses VITS while Edge and Acrobat use their own voices, ASR error differences can masquerade as reading-accuracy differences; homophone examples mentioned in the paper (e.g., y vs why, T vs Tee) are exactly ASR-level confounds. The authors should provide human-listener transcriptions (or at least a second ASR and a plain-text control condition) to separate TTS/ASR artifacts from formula-reading quality.","section":"IV-C"},{"comment":"The test dataset description in Section IV-A reports no number of documents, pages, formulas, or source documents, and the ground-truth spoken English was manually typed by the authors. Since the T5 translator was fine-tuned on MathBridge [27], created by the same research group, and the test transcriptions are likely to follow the same conventions, the measured translation quality may reflect the label convention rather than general reading accuracy. Please report the test set size, make the full test data and annotation guidelines available, and include an independent or at least second-annotator ground truth.","section":"IV-A"},{"comment":"Table II reports only aggregate WER and CER for each reader, with no test-set size, per-document results, error bars, or significance tests. The observed differences could be within noise if the test corpus is small or heterogeneous. Please provide the number of documents/formulas and a paired statistical test (e.g., Wilcoxon signed-rank) on per-document WER/CER.","section":"IV-C / Table II"},{"comment":"The statements in Section I ('Our pipeline can read all the formulas in documents without omissions') and Section III-E ('the TTS model outputs a voice without any errors') are stronger than what the experiments support: OCR (Nougat-small) can misrecognize formulas, delimiter-based extraction can miss undelimited or malformed math, and T5-small can mistranslate. The paper should quantify formula detection and translation coverage, not just overall WER.","section":"I / III-E"}],"minor_comments":[{"comment":"The line 'Let pd fbe the input document' contains a typo; it should read 'Let pdf be the input document.'","section":"Algorithm 1"},{"comment":"The delimiter patterns are written as '\\[ ] \\ or \\( ) \\' with garbled spacing; the intended patterns are \\[ ... \\] and \\( ... \\). Also, 'Seperate' is misspelled as 'Separate' in Algorithm 1, Section III-B, and Table IV.","section":"III-B"},{"comment":"The formula column is rendered ambiguously (e.g., '5P n=1' does not convey the summation notation); a LaTeX rendering or clearer image would improve reproducibility.","section":"Table III"},{"comment":"The runtime is reported for one page on an H100 with unspecified batching or model-loading details; please clarify whether the timing includes model initialization and whether it generalizes to multi-page documents.","section":"IV-E"},{"comment":"References [22] (MathSpeech) and [27] (MathBridge) are from the same research group; the relationship between MathReader and these prior works should be stated explicitly in the text to avoid confusion.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is below the usual experimental rigor bar for a conference/journal: the central comparison rests on a single ASR metric with no dataset sizes or statistical tests. The authors are encouraged to strengthen the evaluation. Additionally, the paper does not clearly distinguish MathReader from their own prior MathSpeech [22] and MathBridge [27], which may raise novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead MathReader, the arXiv paper on TTS for math documents. Short version: it's a sensible integration of Nougat-small OCR, a fine-tuned T5-small that converts LaTeX to spoken English, and VITS TTS, with code and data released. The new bit is the reverse direction of the authors' own MathSpeech work and a cheaper alternative to GPT-4-based LaTeX-to-speech. The ablation without T5 is a nice control: same VITS voice, WER jumps from 0.281 to 0.663, so the translation stage clearly matters.\n\nWhere it falls down is the evaluation. The headline comparisons against Edge and Acrobat use WER computed by Naver Clova Note on synthesized audio. Different TTS systems have different voices, and ASR error rates are voice-dependent, so the 0.281 vs 0.510/0.617 gaps could be partly an ASR artifact. The paper even mentions homophone errors like y vs why, which are exactly ASR confounds. There is no human-listener transcription, no test-set size, no error bars, and the ground truth was manually typed by the authors, likely following the same spoken-English conventions used in their MathBridge training data. That last point is a real circularity risk.\n\nA human-listener study or a multi-ASR check would settle it. Also worth comparing to screen-reader solutions like MathJax speech output, which are standard for math accessibility.\n\nThe time measurements are useful, and the authors are transparent about the components. But the paper's central quantitative claim—that MathReader reads formulas more accurately than commercial readers—is not established by these numbers. The direction is credible, and the pipeline is a reasonable engineering contribution for an accessibility workshop.\n\nFor peer review: yes, I'd send it. The problem is real, the method is clearly described, and the evaluation is fixable. But it would need a proper listening test and a fully specified public test set before the performance claims should be taken at face value.","headline":"A reasonable, clearly described math-TTS pipeline with released code, but the headline WER gains are not yet established because the evaluation relies on a single ASR engine without human listening or error bars.","tokens_in":7427,"tokens_out":2520,"would_cite":true,"duration_ms":25112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reading pipeline that turns LaTeX formulas into spoken English before synthesizing speech lowers word error rates on math documents far below what Microsoft Edge and Adobe Acrobat achieve.","keywords":["text-to-speech","mathematical documents","LaTeX translation","OCR","T5","document reader","assistive technology","word error rate"],"falsifier":"Have independent listeners transcribe the audio produced by MathReader, Microsoft Edge, and Adobe Acrobat on the same set of PDFs, using a pre-registered spoken-English ground truth; if MathReader's word error rate is not lower than the commercial readers' under that human-listener comparison, the paper's central claim is refuted.","tokens_in":6494,"feed_emoji":"🔊","tokens_out":6299,"duration_ms":59374,"temperature":0.7,"pith_summary":"The paper proposes MathReader, a five-stage pipeline that reads a PDF, extracts its LaTeX formulas, translates those formulas into spoken English with a fine-tuned T5-small model, and then feeds the plain-English text to a text-to-speech engine. It claims this fixes a known failure of commercial document readers, which either read formula symbols as unrelated words or skip formulas entirely. On a test set of open-source math documents, MathReader's speech had a word error rate of 0.281, versus 0.510 for Microsoft Edge and 0.617 for Adobe Acrobat, with character error rates of 0.148 versus 0.341 and 0.454. Removing the T5 translation step makes the same pipeline perform worst of all, with WER 0.663, which the paper takes as evidence that the formula-to-speech translation is the load-bearing component.","feed_headline":"Spoken-math pipeline halves error rate of PDF readers","feed_subtitle":"It turns LaTeX formulas into spoken English before text-to-speech, cutting word errors to 28.1 percent.","key_machinery":"The load-bearing mechanism is a translation step placed between OCR and TTS. Nougat-small converts the PDF to a markup file; a delimiter-based splitter isolates formula segments using the math-delimiter patterns that the OCR step inserts; a fine-tuned T5-small model converts those segments into spoken English; and VITS synthesizes speech from the resulting plain text. The T5 translator is what carries the argument, because the ablation shows that removing it raises WER from 0.281 to 0.663.","core_discovery":"The central discovery is that for mathematical PDFs, accurate spoken output depends less on the speech synthesizer than on what text is sent to it. If LaTeX code is read directly, special characters are dropped or mispronounced; if formulas are first verbalized as natural English sentences, a standard TTS engine produces understandable speech. The paper demonstrates this by building the full pipeline and comparing it to two widely used readers.","pith_inferences":["The paper's WER numbers come from an automated speech recognizer, so an independent listening test with human transcribers could change the ranking; this is my inference, not the paper's claim.","The translation step should transfer to formulas rendered as web MathML or to equation images in lecture slides, since the pipeline only needs the formula's source text; the paper does not test these inputs.","A broader test corpus spanning physics and engineering notation would test whether T5-small generalizes beyond the authors' manual transcription conventions; this is an extension I am proposing."],"forward_implications":["If the reported error rates hold, visually impaired readers could listen to math-heavy PDFs, including older scanned ones, and hear formulas rather than symbol salad.","The measured time of 23.62 seconds per page on GPU hardware suggests the pipeline is practical for on-demand reading rather than batch-only processing.","The ablation result implies that the formula-translation step, not the speech synthesizer, is what separates accurate from garbled math reading.","Because the implementation is released with the paper, other document-reading tools could adopt the same OCR-translate-TTS chain."],"supporting_citations":[{"why":"Nougat-small supplies the OCR step that turns PDF pages into markup with formula delimiters.","marker":"[10]"},{"why":"T5-small is the language model fine-tuned to translate LaTeX formulas into spoken English.","marker":"[14]"},{"why":"MathBridge provides the (LaTeX, spoken English) pairs used to fine-tune the translator.","marker":"[27]"},{"why":"VITS is the text-to-speech model that synthesizes the final plain-English text.","marker":"[3]"},{"why":"Microsoft Edge's read-aloud feature is one of the two commercial baselines whose WER MathReader claims to beat.","marker":"[1]"},{"why":"Adobe Acrobat's read-aloud feature is the other commercial baseline in the comparison.","marker":"[2]"},{"why":"Naver Clova Note is the speech-to-text platform used to transcribe all system outputs for WER and CER measurement.","marker":"[28]"}],"fun_headline_variants":["Spoken math: pipeline verbalizes formulas, cuts errors by half","MathReader reads LaTeX aloud with half the word errors","Turning formulas into speech before TTS halves error rate","For math PDFs, verbalizing formulas first halves word errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the word error rate computed by an automated speech recognizer on each system's synthesized voice is a valid measure of reading accuracy, even though the three systems use different voices and the ground-truth spoken-English transcriptions were typed by the same team that built the translator.","fun_headline_variants_meta":{"raw":{"variants":["Spoken math: pipeline verbalizes formulas, cuts errors by half","MathReader reads LaTeX aloud with half the word errors","Turning formulas into speech before TTS halves error rate","For math PDFs, verbalizing formulas first halves word errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2395,"prompt_tokens":845,"completion_tokens":1550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":1481}},"tokens_in":461,"tokens_out":1550,"duration_ms":11093,"temperature":1.0,"reasoning_tokens":1481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:36.879267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent listeners transcribe the audio produced by MathReader, Microsoft Edge, and Adobe Acrobat on the same set of PDFs, using a pre-registered spoken-English ground truth; if MathReader's word error rate is not lower than the commercial readers' under that human-listener comparison, the paper's central claim is refuted.","supporting_citations":[{"cited_title":"Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech","cited_arxiv_id":null,"evidence_quote":"VITS is the text-to-speech model that synthesizes the final plain-English text."},{"cited_title":"Pdf reader in microsoft edge","cited_arxiv_id":null,"evidence_quote":"Microsoft Edge's read-aloud feature is one of the two commercial baselines whose WER MathReader claims to beat."},{"cited_title":"Nougat: Neural optical understanding for academic documents, 2023","cited_arxiv_id":null,"evidence_quote":"Nougat-small supplies the OCR step that turns PDF pages into markup with formula delimiters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"T5-small is the language model fine-tuned to translate LaTeX formulas into spoken English."},{"cited_title":"Mathbridge: A large corpus dataset for translating spoken mathematical expressions into latex formulas for improved readability, 2024","cited_arxiv_id":null,"evidence_quote":"MathBridge provides the (LaTeX, spoken English) pairs used to fine-tune the translator."},{"cited_title":"Conveniently hear pdf files read aloud","cited_arxiv_id":null,"evidence_quote":"Adobe Acrobat's read-aloud feature is the other commercial baseline in the comparison."},{"cited_title":"Naver clovanote","cited_arxiv_id":null,"evidence_quote":"Naver Clova Note is the speech-to-text platform used to transcribe all system outputs for WER and CER measurement."}],"review_version":1}