{"id":"c085bb36-0a9e-47f7-8903-01fe84c22b4f","arxiv_id":"2412.12167","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A Greek speech-to-LaTeX tool built from XLS-R, GPT-3.5, and retrieval over 500 examples shows in-context learning gains, yet only the text-to-LaTeX part is measured.","lead":"Greek2MathTex is a system that lets Greek speakers dictate math equations aloud and receive LaTeX code, combining a fine-tuned speech recognition model with GPT-3.5 and example retrieval. It targets accessibility for visually impaired users, but the paper evaluates only the text-to-LaTeX stage, not the full speech pipeline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The end-to-end speech-to-LaTeX claim is untested: Table 2 only measures LaTeX generation from manually transcribed text, with no WER or audio-to-LaTeX result reported anywhere, so the central contribution as stated is unsupported.","rationale":"The reader's weakest assumption is the same one I find load-bearing: the evaluation tests text-based LaTeX generation, not the speech-to-LaTeX system promised in the title and abstract. Section 5.2 explicitly uses \"provided transcribed text sequence,\" and no WER or audio-to-LaTeX result appears anywhere. Because the system's architecture chains ASR, retrieval, and LLM generation, the missing link is not a minor gap; it is the component that separates the claimed contribution from a known text-generation task. The paper's own description of fine-tuning XLS-R on Gr2Tex without clarifying exclusion of test audio raises an additional contamination risk, though it is secondary because no ASR results are reported at all. The small test set and test-set hyperparameter selection are also real concerns, but they would matter mainly after the end-to-end pipeline is actually measured. I would not reject the resources: the Gr2Tex dataset and the open-source web application are useful. I would reject the current central claim and ask for an end-to-end evaluation on held-out audio before the speech-to-LaTeX claim can be accepted; hence my verdict is unchanged from the reader's REJECT. If the authors add a properly split audio evaluation and it matches the text-only results, the paper would be publishable as a demonstration of the full pipeline.","tokens_in":6288,"tokens_out":4778,"duration_ms":47293,"concrete_test":"Run the released pipeline on the held-out Gr2Tex audio test set (the 15% of recordings not used to fine-tune XLS-R): audio -> ASR transcript -> retrieval -> GPT-3.5 LaTeX. Report ASR WER on those math utterances, end-to-end EL<0.1 and BLEU/chrF, and the same metrics when the retrieval is given clean transcripts instead of ASR transcripts. If the end-to-end EL<0.1 is materially below the text-input 37.67% (or if WER is high), the central speech-to-LaTeX claim is not supported; if the two curves match, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a working Greek audio-to-LaTeX system. The only quantitative evidence, Section 5.2/Table 2, is text-to-LaTeX: the generator \"was prompted solely to produce the LaTeX equation for the provided transcribed text sequence.\" No experiment feeds audio through the fine-tuned XLS-R component and reports its output or its WER. This is load-bearing because end-to-end quality is bounded by ASR accuracy; if the ASR stage makes errors, the retrieved k-NN examples are selected from corrupted queries and the LLM prompt is degraded, so the reported ICL gains (EL<0.1 rising from 27.45% to 37.67%) may not survive the real pipeline. The omission is compounded by Section 4.1, which says XLS-R was fine-tuned on the same Gr2Tex dataset that provides the test split, without stating that test recordings were excluded. A secondary fragility is the evaluation scale: roughly 75 test equations with k, similarity measure, and prompt selected on that same set, with no error bars or significance tests. The open-source release and Greek dataset are genuine resources, but they do not establish the headline speech-to-LaTeX claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Greek2MathTex, described as an end-to-end speech-to-LaTeX system for Greek mathematical dictation. The architecture combines a fine-tuned XLS-R ASR model, a k-NN retrieval mechanism over a self-built Greek dataset (Gr2Tex, 500 text/LaTeX pairs with audio), and GPT-3.5-turbo with in-context learning. The paper reports experiments in Table 2 showing that retrieval-based ICL improves text-to-LaTeX generation under a custom Levenshtein-based metric EL, and it releases code and a web application. No experiment feeds audio through the ASR component and measures the resulting LaTeX.","tokens_in":6507,"tokens_out":4323,"duration_ms":39000,"significance":"The intended contribution—an accessible Greek speech-to-LaTeX tool—is worthwhile, and the release of the Gr2Tex dataset and open-source code is a concrete asset. The paper also provides a useful comparison of prompts and retrieval configurations for LLM-based equation generation. However, the current evidence does not establish the headline speech-to-LaTeX claim because the quantitative evaluation is text-to-LaTeX only; the ASR component's accuracy and its interaction with retrieval are unmeasured. With an end-to-end evaluation or a narrowed claim, the contribution could be solid.","major_comments":[{"comment":"The only quantitative results are for text-to-LaTeX generation: Section 5.2 states the model was prompted to produce LaTeX for the 'provided transcribed text sequence,' and no word error rate, audio-to-text accuracy, or audio-to-LaTeX metric appears anywhere in the paper. The abstract and Section 1 claim the system works 'based solely on audio input,' so the central claim is currently unsupported. Please add an evaluation that feeds audio through the fine-tuned XLS-R model (e.g., WER on Gr2Tex test audio, followed by end-to-end EL/BLEU on the transcribed output), or explicitly rescope the paper's claim to text-to-LaTeX generation.","section":"Section 5.2, Table 2"},{"comment":"Section 4.1 states that XLS-R was fine-tuned using 'our custom domain- and language-specific dataset' after Section 3 describes a 70/15/15 train/validation/test split of Gr2Tex. The text does not state that the test split's recordings were excluded from ASR fine-tuning. If they were not, the test set is contaminated and all downstream numbers are optimistic. Please state explicitly that the test split was held out from both retrieval database and ASR fine-tuning.","section":"Sections 4.1 and 3"},{"comment":"The test set is only 15% of 500 pairs (about 75 equations), and the hyperparameters k, similarity/distance function, and prompt p1/p2/p3 are selected on that same test set. The reported improvements (EL<0.1 from 27.45% to 37.67%) are therefore not accompanied by any error bars, significance tests, or held-out validation. Please provide a separate validation split for hyperparameter selection or report confidence intervals; otherwise the ICL gains may reflect overfitting to the test queries.","section":"Section 5.2, Table 2"},{"comment":"The custom EL metric is said to have been assessed by comparing it to human annotations (match/almost match/no match), but no result of that comparison is reported. Without this validation, the EL<0.1 and EL>0.4 thresholds lack an independent interpretation. Please report the agreement/correlation between EL and the human labels, and if the comparison was only informal, state so.","section":"Section 5.1"}],"minor_comments":[{"comment":"The text says the fourth and fifth columns report percentages 'lower than 0.1 or 0.4,' but the table header reads 'EL > 0.4'; clarify whether the fifth column is the fraction above 0.4 (as the header and the discussion of reducing it suggest) and align the wording.","section":"Table 2 and Section 5.2"},{"comment":"The dataset name is inconsistently written as Gr2Tex and Gr2TeX; choose one spelling.","section":"Throughout"},{"comment":"The caption contains the broken 'L ATEX' due to formatting; correct to LaTeX.","section":"Figure 2"},{"comment":"The first sentence calls the system 'speech-to-text,' although the contribution is speech-to-LaTeX; correct the terminology.","section":"Section 7"},{"comment":"The text says the three instruction prompts are 'outlined in Table 2,' but they actually appear in Table 1; fix the cross-reference.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a short conference paper with a promising resource release but a central evaluation gap. If the authors can supply even a small end-to-end audio evaluation with WER and final EL/BLEU, the paper could be strengthened substantially; if not, the title and abstract should be changed to describe a text-to-LaTeX system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this is a short, workshop-quality paper with one genuinely useful asset—the Gr2Tex dataset and open-source code—and one load-bearing empirical gap: the system advertised as speech-to-LaTeX is never evaluated on speech. Table 2 only feeds manually transcribed text into GPT-3.5 with retrieval ICL. No word error rate, no audio-to-LaTeX result, nothing. The authors describe fine-tuning XLS-R on Greek audio plus the Gr2Tex dataset, but they don't report how well it transcribes math dictation, which is the whole point of the title.\n\nThe new thing here is real: the first Greek-specific dataset for natural-language math expressions paired with LaTeX (500 pairs), released openly, and a sensible retrieval-augmented prompt recipe for GPT-3.5. The custom EL metric is a reasonable practical tool, and the web app lowers the barrier for Greek-speaking users. The related work is appropriate and the architecture choices are defensible.\n\nThe soft spots are proportionate to the missing evidence. First, the end-to-end claim collapses if the ASR stage is poor, and we have no data on that. Second, the evaluation is small: roughly 75 test equations, with k, similarity/distance function, and prompt all selected on that same test set. No error bars, no significance tests. Third, Section 4.1 says XLS-R was fine-tuned on the same Gr2Tex dataset that provides the test split, without explicitly stating that test recordings were excluded—so the reader cannot rule out leakage. The custom EL metric's validation against human annotation is also described in one sentence with no numbers.\n\nThe reader's REJECT is fair if the bar is \"does the paper support its advertised claim.\" It does not. But the paper is not worthless: the dataset and the text-to-LaTeX component are contributions in their own right, and the gaps are fixable with additional experiments.\n\nWho this is for: people working on speech-to-formula systems, accessibility, or Greek-language NLP. I would not cite the paper for its central claim, but I would cite the Gr2Tex dataset. A serious editor should send this to peer review, because the resource is useful and the authors can reasonably add the missing ASR evaluation or reframe the contribution as text-to-LaTeX generation plus a dataset. My recommendation: engage with it, ask for the end-to-end numbers, and let the authors decide whether to narrow the claims.","headline":"Useful dataset and a sane text-to-LaTeX pipeline, but the headline speech-to-LaTeX claim is untested: no audio goes through the system in the reported experiments.","tokens_in":7097,"tokens_out":1395,"would_cite":true,"duration_ms":16742,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that dictating Greek mathematics aloud can produce LaTeX code through a pipeline of a fine-tuned speech recognizer and a large language model prompted with retrieved example equations.","keywords":["Greek speech recognition","LaTeX equation generation","in-context learning","retrieval-augmented generation","accessibility technology","automatic speech recognition","Levenshtein edit distance","math dictation"],"falsifier":"Run the audio recordings of the test split through the fine-tuned recognizer and then through the best retrieval-augmented prompt, and compute the same edit-distance metrics on the final LaTeX output. If the end-to-end rate of edit distance below 0.1 is far below 37.67%, or the recognizer's error rate on dictated math is high, the central speech-to-LaTeX claim is unsupported.","tokens_in":6050,"feed_emoji":"🎤","tokens_out":9611,"duration_ms":90309,"temperature":0.7,"pith_summary":"This paper claims that a Greek-specific speech-to-LaTeX system is feasible by combining a fine-tuned speech recognizer with a large language model prompted with retrieved example equations. The authors built a dataset of 500 pairs of Greek spoken equations and their LaTeX forms, and they report that retrieval-based in-context learning markedly improves the text-to-LaTeX stage, lifting the share of outputs with edit distance below 0.1 from 27.45% to 37.67% at $k=5$ with cosine similarity and the English instruction prompt. They frame the contribution as an accessibility tool: users who struggle to type or read LaTeX can dictate mathematics and receive compilable code.","feed_headline":"Retrieval lifts Greek speech-to-LaTeX close-match rate to 38%","feed_subtitle":"Retrieval of similar equations lifts close-match accuracy from 27% to 38%, a step toward accessible math typing.","key_machinery":"The load-bearing mechanism is retrieval-augmented in-context learning: for each dictated equation, a $k$-nearest-neighbour search over the held-out Gr2Tex examples selects the $k$ most similar natural-language/LaTeX pairs, which are prepended to the prompt alongside an instruction, and the large language model then generates LaTeX for the query without updating its parameters. The paper's custom evaluation metric is the Levenshtein distance between normalized generated and ground-truth strings, used with thresholds 0.1 and 0.4. The retrieval mechanism and prompt design, not the speech recognizer, carry the reported accuracy gains.","core_discovery":"The central discovery the paper tries to establish is that an end-to-end Greek speech-to-LaTeX pipeline can be assembled from a fine-tuned XLS-R speech model, a retrieval mechanism over a small paired dataset (Gr2Tex), and GPT-3.5 with in-context learning. Evaluated on the test split of Gr2Tex, retrieval-augmented prompting outperforms no-example prompting on all reported metrics, with cosine similarity at $k=5$ and the English instruction prompt $p_2$ giving the best rate of outputs within edit distance 0.1 of the ground truth (37.67% versus a 27.45% baseline). English instructions slightly beat their Greek translations, and increasing $k$ to six does not improve over $k=5$. The authors claim this demonstrates a usable advancement in spoken-math accessibility for Greek speakers.","pith_inferences":["The reported accuracy is measured on hand-transcribed text rather than on the recognizer's output, so the true end-to-end speech-to-LaTeX accuracy remains untested and would likely be lower once recognition errors enter.","The same retrieval-plus-in-context-learning recipe should transfer to other languages that lack math-speech data, since it needs only a few hundred paired examples and a passable recognizer.","The observed advantage of English prompts over Greek suggests the underlying language model generates LaTeX more reliably from English framing; a Greek-specialized model or bilingual prompt tuning could close that gap."],"forward_implications":["Adding retrieval examples to the prompt raises the share of outputs with edit distance below 0.1 from 27.45% to 37.67% at $k=5$ with cosine similarity and the English instruction prompt $p_2$.","Increasing the number of retrieved examples from five to six gives no further improvement, so the benefit of retrieval saturates.","Cosine similarity performs better than Manhattan or Euclidean distance in the tested configurations, and English prompts outperform a Greek translation of the same instruction.","Because in-context learning requires no retraining, the LaTeX generation stage can be updated by changing the prompt or the retrieved examples without modifying the speech recognizer.","The system's web interface lets a Greek speaker record an equation, play it back, and receive the generated LaTeX code, which is the accessibility scenario the paper targets."],"supporting_citations":[{"why":"Supplies the Greek audio data used to fine-tune the speech recognizer.","marker":"[1]"},{"why":"Provides the multilingual speech model that is fine-tuned to become the ASR component.","marker":"[2]"},{"why":"Defines in-context learning, the prompting paradigm the equation generator relies on.","marker":"[4]"},{"why":"Is the code-specialized model the authors reject for weak Greek, motivating their choice of a general large language model.","marker":"[5]"},{"why":"Provides the BLEU metric used to report generation quality.","marker":"[8]"},{"why":"Provides the chrF metric used to report generation quality.","marker":"[9]"}],"fun_headline_variants":["Greek math dictation to LaTeX hits 38% with retrieval aid","Retrieval boosts Greek speech-to-LaTeX accuracy to 38%","Open-source Greek speech-to-LaTeX framework reaches 38%","Speak Greek math, get LaTeX: retrieval lifts success to 38%","Greek speech-to-LaTeX: retrieval lifts close-match to 38%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that evaluating on hand-transcribed text with a small test set reflects real speech performance, because the paper never measures recognition errors on dictated math audio.","fun_headline_variants_meta":{"raw":{"variants":["Greek math dictation to LaTeX hits 38% with retrieval aid","Retrieval boosts Greek speech-to-LaTeX accuracy to 38%","Open-source Greek speech-to-LaTeX framework reaches 38%","Speak Greek math, get LaTeX: retrieval lifts success to 38%","Greek speech-to-LaTeX: retrieval lifts close-match to 38%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1482,"prompt_tokens":911,"completion_tokens":571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":475}},"tokens_in":527,"tokens_out":571,"duration_ms":5523,"temperature":1.0,"reasoning_tokens":475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:31:45.560192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the audio recordings of the test split through the fine-tuned recognizer and then through the best retrieval-augmented prompt, and compute the same edit-distance metrics on the final LaTeX output. If the end-to-end rate of edit distance below 0.1 is far below 37.67%, or the recognizer's error rate on dictated math is high, the central speech-to-LaTeX claim is unsupported.","supporting_citations":[{"cited_title":"Ardila et al","cited_arxiv_id":null,"evidence_quote":"Supplies the Greek audio data used to fine-tune the speech recognizer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multilingual speech model that is fine-tuned to become the ASR component."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines in-context learning, the prompting paradigm the equation generator relies on."},{"cited_title":"Li et al","cited_arxiv_id":null,"evidence_quote":"Is the code-specialized model the authors reject for weak Greek, motivating their choice of a general large language model."},{"cited_title":"Papineni, S","cited_arxiv_id":null,"evidence_quote":"Provides the BLEU metric used to report generation quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the chrF metric used to report generation quality."}],"review_version":1}