WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications

Chau Yuen; Jiancheng An; Li Wei; Mengbing Liu; M\'erouane Debbah; Xin Li

REVIEW 4 major objections 6 minor 19 references

WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that current LLMs, however strong on recall, fail at multi-step symbolic derivation in wireless communications: even the best model, DeepSeek-R1, averages 38.05% accuracy and completes only 7.83% of fully…

desk verdict Worth a serious look, but the headline numbers rest on an unvalidated GPT-4o judge; the progressive-masking design is a real contribution, and the qualitative degradation finding likely survives, but treat the absolute accuracies as provisional. read the letter →

arxiv 2505.14354 v1 pith:DL6IEZSH submitted 2025-05-20 cs.CL cs.LG

Xin Li , Mengbing Liu , Li Wei , Jiancheng An , Mérouane Debbah , Chau Yuen This is my paper

classification cs.CLcs.LG

keywords wirelesscommunicationsLLMbenchmarkmathematicalreasoningequationcompletionprogressivemaskingsymbolicderivationdomain-specificevaluationlargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WirelessMathBench asks whether LLMs can do the mathematical modeling that wireless communications engineers routinely do: deriving channel gains, beamformers, and phase-shift matrices from a scenario description. The benchmark distills 587 questions from 40 research papers into a difficulty ladder, from multiple-choice recognition through progressive equation masks to full equation completion. The paper's finding is that recognition and derivation decouple: most models score above 70% on multiple-choice yet fall to single digits on fully masked derivations, with the best model, DeepSeek-R1, at 7.83%. This matters because it identifies a concrete gap between LLM recall and the multi-step symbolic reasoning that engineering tools would need.

What carries the argument

The load-bearing mechanism is the progressive masking ladder, which turns a single formula from a source paper into a graded series of independent questions: a multiple-choice recognition, three fill-in-the-blank tasks with increasing portions of the equation hidden, and a full equation completion where only the scenario description remains. Because every rung of the ladder uses the same underlying equation, the benchmark isolates whether a model's failure is a recall failure or a derivation failure. Scoring of the free-form answers is delegated to an LLM judge (GPT-4o) that checks symbolic equivalence and dimensional plausibility.

What would settle it

Take a random sample of, say, 100 full-equation completion outputs from DeepSeek-R1 and have wireless engineers score them against the same rubric; if human agreement with GPT-4o is poor, the benchmark's headline numbers would need recalibration, and possibly the ordering of models would change.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art LLMs can identify the right wireless formula when it is placed in front of them, but cannot reliably reconstruct it when parts are hidden. The evidence is a performance cliff as masking increases: DeepSeek-R1 drops from 76.00% on multiple-choice to 60.00% at mask level 1, 34.91% at level 2, 12.50% at level 3, and 7.83% on full equation completion; sixteen tested models show the same monotone decline. The paper reads this as evidence that current LLMs lack the capacity for domain-grounded, multi-step symbolic manipulation under dimensional and physical constraints, and releases the benchmark and evaluation toolkit to make that gap measurable.

Load-bearing premise

The results stand on the assumption that GPT-4o scores free-form equation answers accurately enough that the reported percentages reflect real model ability, not judge leniency or strictness.

Editorial extensions

If this is right

Domain-adaptive training and retrieval-augmented methods will need to be measured against full-equation derivations, not just factual recall, or they will appear successful while remaining unusable for engineering.
Chain-of-thought prompting does not close the gap: the paper's own CoT experiments show marginal or negative effects on the hardest tasks, so improvements will have to come from architecture or specialized training.
WirelessMathBench's scores provide a baseline: any claim that an LLM can assist with wireless system analysis should be expected to beat DeepSeek-R1's 38.05% average and 7.83% full-equation rate.
The monotone decline with masking level offers a diagnostic: a model that holds up at high mask levels is doing genuine symbolic reconstruction, not memorized retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Provided the paper's results hold, a reader could test whether the GPT-4o judge changes the picture by having humans re-score a sample of outputs; if human agreement is poor, the headline numbers and model ordering would shift.
The same progressive-masking ladder could be applied to other engineering fields, such as control theory or circuit design, where derivations carry physical constraints, making the failure pattern visible outside wireless.
The paper's error taxonomy—partial fill mismatch, symbol misinterpretation, derivation path errors, and system mixing—suggests a concrete extension: fine-tune on masked-equation data that penalizes symbol substitution errors and check whether level-3 accuracy, specifically, improves.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a serious look, but don't quote the headline numbers yet. WirelessMathBench introduces a progressive-masking scheme for wireless-communications math equations—starting with MCQs, then partial masks, then full equation completion—and that design is genuinely new relative to TeleQnA, TelecomGPT, and the generic math benchmarks. The dataset draws from 40 peer-reviewed papers and went through an expert review process, which is more care than most benchmark papers take. The core result, that accuracy drops monotonically as masking increases across all 16 models, is consistent and credible even if the exact percentages are not yet trustworthy.

The main problem is the evaluation. For everything except MCQs, a single GPT-4o instance grades the free-form equation outputs, and that judge is never validated against human experts. The limitations section concedes it may miss incorrect intermediate reasoning. With only around 100 items per hardest task, a 5–10% judge error rate could shift the FEC number from 7.83% into double digits. Single-run results, no confidence intervals, and a 40-item error analysis make the absolute numbers fragile. The qualitative ranking probably survives, but the paper's calibrated claims—'DeepSeek-R1 achieves only 38.05%'—rest on an unvalidated grader.

Also missing is a human expert baseline, so we don't know how a skilled wireless engineer would score on these tasks. The paper says the dataset will be released, but as of this version the link is just a placeholder. For a benchmark, shipping the data is not optional.

Those are fixable issues. Validate the judge on a sample, add a human baseline, report variance across runs, and release the data. The progressive-masking idea is a real contribution, and the error taxonomy, though small, gives a useful starting point.

So, yes, this deserves peer review—conditional accept with major revisions. It is for researchers building domain-specialized LLMs and for the LLM-evaluation community, not for a broad ML audience. Read it for the task design and the qualitative finding, not for the absolute scores.

Referee Report

4 major / 6 minor

Summary. The paper introduces WirelessMathBench, a benchmark of 587 wireless-communications mathematical modeling questions derived from 40 arXiv papers, organized into multiple-choice questions (MCQ), progressively masked fill-in-the-blank tasks at three difficulty levels, and full equation completion (FEC) tasks. Sixteen LLMs are evaluated zero-shot; the authors report that accuracy drops sharply as masking increases, with the best model (DeepSeek-R1) achieving 38.05% average accuracy and only 7.83% on FEC. The paper also includes an error analysis based on 40 sampled failures and a set of chain-of-thought experiments. The benchmark and evaluation toolkit are released publicly.

Significance. If the reported results are reliable, the benchmark fills a genuine gap: existing math benchmarks do not cover expert-level, domain-specific symbolic derivation in engineering, and the finding that state-of-the-art LLMs fall to single-digit accuracy on full equation completion is a substantive and potentially influential observation for the community. The paper has real strengths: the data are sourced from peer-reviewed papers and pass a multi-round expert review; the task design (progressive masking) creates a useful difficulty gradient; and the public release of the benchmark and toolkit supports reproducibility and follow-up work. The central weakness is that the headline numbers are calibrated by an unvalidated GPT-4o judge and a single run, so the absolute accuracies should be treated as provisional until human-rescoring evidence is provided.

major comments (4)

[§4.1 and Appendix C] The automated evaluation for all non-MCQ tasks uses GPT-4o as judge, but the judge is never validated. There is no human-expert rescoring sample, no inter-judge agreement statistic, and no analysis of false positives versus false negatives. This is load-bearing because the headline FEC accuracy of 7.83% (DeepSeek-R1) corresponds to roughly 9 correct answers out of the FEC item count; a modest judge error rate could move this figure substantially, and the same applies to the Level-3 and average-accuracy numbers. Please report a human-rescored subset (e.g., 100+ items), agreement measures, and results under stricter/looser judging criteria, or an independent symbolic-equivalence check.
[§4.2, Table 4] The definition of 'Avg. Acc' is not specified. It is ambiguous whether it is the unweighted mean of the five per-metric accuracies or the total number of correct answers divided by 587. Because the five metrics are likely to have different numbers of items (MCQ has many items, FEC fewer), these two computations can differ. Please report per-metric item counts and the exact formula used to compute the average.
[Appendix D] The appendix states that 'all results are the results of a single run.' Combined with the LLM judge, there are no variance estimates for any reported number. At minimum for the main models (DeepSeek-R1, OpenAI-o1, GPT-4o, DeepSeek-V3), please report repeated runs (or a determinism check at temperature 0) and provide standard deviations or confidence intervals for the key metrics.
[§3.2] The paper asserts that tasks are reformulated to 'avoid word-for-word reproductions' and to reduce contamination, but no quantitative contamination analysis is provided. Since the 40 source papers are freely available on arXiv and likely appear in LLM pretraining corpora, please measure overlap between the task prompts and the source texts (e.g., n-gram or embedding similarity) and report results on a contamination-filtered subset, especially for the MCQ task where memorization is most plausible.

minor comments (6)

[Table 3] The model name 'Gemini-Famliy' contains a typo; it should read 'Gemini-Family'.
[§4.2] The paragraph beginning 'Progressive Masking Emphasizes Multi-Step Reasoning Gaps' contains a duplicated sentence: 'When forced to reconstruct partially hidden expressions, model performance declines in proportion to the level of masking' is immediately followed by a near-identical sentence. Please remove the duplicate.
[§3.3] 'To full evaluate the capabilities' should be 'To fully evaluate the capabilities'.
[Limitations] The phrase 'Radio frequency (RF) measurements measurements' contains a duplicated word; remove one occurrence.
[§4.3] The error analysis is based on only 40 randomly sampled errors, with no sample-size justification or confidence intervals for the reported percentages (31%, 29%, 24%, 11%, 4%). Please state whether the sample is sufficient to support the claimed error taxonomy or present the analysis as illustrative.
[Abstract] The phrase 'mathematical modeling challenges to wireless communications engineering' reads awkwardly; consider 'challenges in wireless communications engineering'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark's ground truth is externally sourced and expert-validated, and the headline accuracy figures are measured rather than fitted.

full rationale

WirelessMathBench is an empirical benchmark paper, not a derivation. Ground-truth answers are external artifacts: they come from 40 published, peer-reviewed papers and are cross-validated by domain experts (Appendix B). The headline accuracies (38.05% and 7.83%) are measured outcomes, not fitted parameters: no parameter is tuned to a target result, and no equation is predicted from an input that already contains the answer. The GPT-4o judge used for free-form answers (Section 4.1) is an implementation choice that affects measurement validity, and Appendix D's single-run setting adds variance, but neither makes the claim equivalent to its inputs. Some source papers are authored by the benchmark creators (e.g., the 'an2023stacked' example in Appendix E), yet the benchmark's correctness does not rest on those citations: the questions are independently expert-reviewed and the LLM scores are observed consequences of model behavior, not of the authors' prior claims. I find no circular step.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

WirelessMathBench makes no theoretical derivation claims. It rests on four unverified assumptions: source equations are correct ground truth, expert review ensures question quality, GPT-4o grading is accurate, and single-run zero-shot outputs represent model ability. No free parameters or invented physical entities appear.

assumptions (4)

domain assumption Equations from the 40 selected peer-reviewed papers are correct and can serve as ground truth.
Section 3.2 says tasks are sourced directly from peer-reviewed research. If any source equation is wrong or typo-laden, the benchmark answer key inherits that error.
domain assumption Domain expert review guarantees question correctness, clarity, and unambiguity.
Appendix B describes a cross-verification protocol, but the paper reports no inter-rater reliability statistics or per-question agreement numbers.
domain assumption GPT-4o can accurately judge whether free-form model outputs are symbolically equivalent to the ground truth.
Section 4.1 states GPT-4o completes the evaluation, while the Limitations section admits that intermediate reasoning may not be checked. Judge error would directly change every reported accuracy.
domain assumption Zero-shot, single-run responses are representative of model capability.
Appendix D says 'all results are the results of a single run.' No variance or repeated sampling is reported, so small accuracy differences between models may be noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications." pith.science (2026). https://pith.science/paper/DL6IEZSH

@misc{pith2026250514354,
  author       = {Pith},
  title        = {Pith review of: WirelessMathBench: A Mathematical Modeling Benchmark for LLMs in Wireless Communications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DL6IEZSH}},
  note         = {Machine review of arXiv:2505.14354}
}

read the original abstract

Large Language Models (LLMs) have achieved impressive results across a broad array of tasks, yet their capacity for complex, domain-specific mathematical reasoning-particularly in wireless communications-remains underexplored. In this work, we introduce WirelessMathBench, a novel benchmark specifically designed to evaluate LLMs on mathematical modeling challenges to wireless communications engineering. Our benchmark consists of 587 meticulously curated questions sourced from 40 state-of-the-art research papers, encompassing a diverse spectrum of tasks ranging from basic multiple-choice questions to complex equation completion tasks, including both partial and full completions, all of which rigorously adhere to physical and dimensional constraints. Through extensive experimentation with leading LLMs, we observe that while many models excel in basic recall tasks, their performance degrades significantly when reconstructing partially or fully obscured equations, exposing fundamental limitations in current LLMs. Even DeepSeek-R1, the best performer on our benchmark, achieves an average accuracy of only 38.05%, with a mere 7.83% success rate in full equation completion. By publicly releasing WirelessMathBench along with the evaluation toolkit, we aim to advance the development of more robust, domain-aware LLMs for wireless system analysis and broader engineering applications.

Figures

Figures reproduced from arXiv: 2505.14354 by the authors.

**Figure 1.** Example task from WirelessMathBench a system model derivation from wireless communications literature. The derivation progresses from a multiplechoice question to progressive mask completion questions, and finally to the full formula derivation, testing the model’s ability to reason through complex channel reflections and matrix operations. 2025) have further extended these capabilities, especially when supplemen… view at source ↗

**Figure 2.** Overview of the data collection and annotation pipeline for WirelessMathBench [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** A word cloud illustrating the most frequent [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Error distribution among 40 annotated DeepSeek-R1 errors. consists of wireless protocols, whereas the problems in WirelessMathBench require handling long contexts and performing high-level mathematical reasoning. 4.3 Error Analysis To better understand the limitations…

**Figure 5.** Figure 5: An example question and the corresponding output from LLMs for a multiple-choice task. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: An example of a level 1 progressively masked fill-in-the-blank task. The figure shows both the question [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

**Figure 7.** Figure 7: An example of a level 2 progressively masked fill-in-the-blank task. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: An example of a level 3 progressively masked fill-in-the-blank task. Notice that both OpenAI-o1 and [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

**Figure 9.** Figure 9: An example of a Full Equation Completion (FEC) task. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

**Figure 10.** Figure 10: This template is used to generate concise summaries of research papers. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

**Figure 11.** Figure 11: This prompt template converts paper summaries into detailed question–answer pairs. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

**Figure 12.** Figure 12: This template is designed for answering multiple-choice questions. The model is guided to choose the [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

**Figure 13.** Figure 13: Prompt template used for fill-in-the-blank and full equation completion tasks. It directs the model to [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

**Figure 14.** Figure 14: This evaluation prompt template standardizes the process of assessing model-generated answers. It [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

[1]

Each question in WirelessMathBench was re- viewed by a minimum of two experts. 13

work page
[2]

Experts independently verified both the math- ematical correctness and clarity of each ques- tion

work page
[3]

Preprint, arXiv:2412.19437

Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Mengbing Liu, Jiancheng An, Huang Chongwen, and Chau Yuen. 2025a. Over-the-air ode-inspired neural network for dual task-oriented semantic communica- tions. IEEE Transaction on on Cognitive Communi- cations and Networkingn (TCCN). Mengbing Liu, Chongwen Huang, Alhammadi Ahmed, Marco Di Renzo, mérou...

arXiv 2022
[4]

Stephen Wolfram

Scibench: Evaluating college-level scientific problem-solving abilities of large language models. Stephen Wolfram. 2003. The mathematica book. Wol- fram Research, Inc. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren...

arXiv 2003
[5]

IEEE Wireless Com- munications, 31(3):140–147

Mobile cell-free massive MIMO: Challenges, solutions, and future directions. IEEE Wireless Com- munications, 31(3):140–147. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu

work page
[6]

Consensus Discussion: Resolution of any discrepancies between reviewer assessments

work page
[7]

arXiv preprint arXiv:2407.09424

TelecomGPT: A framework to build telecom- specfic large language models. arXiv preprint arXiv:2407.09424. 12 A Dataset Details A.1 Topics and Papers Selection Our study addresses modern wireless communica- tion challenges by selecting topics based on three key criteria that ensure both the academic rigor and practical relevance of our work. First, we targ...

arXiv 2024
[10]

When disagreements arose, the reviewers en- gaged in detailed discussions until consensus was reached

work page

Show all 19 references

[11]

For particularly complex derivations, a third expert was consulted to provide additional ver- ification
[12]

This protocol ensured that all questions accu- rately reflected real-world wireless communica- tions challenges while maintaining clear formu- lation and unambiguous answers

All multiple-choice distractors were examined to ensure they represented plausible but incor- rect options. This protocol ensured that all questions accu- rately reflected real-world wireless communica- tions challenges while maintaining clear formu- lation and unambiguous ans...
[13]

Initial Extraction: Semi-automated extrac- tion of system models from research papers
[14]

Question Formulation: Transformation of system models into question-answer pairs with varying difficulty levels
[15]

First Review: Initial expert review focus- ing on mathematical correctness, dimensional consistency, and domain applicability
[16]

Refinement: Modification of questions based on first review feedback
[17]

Second Review: Independent validation by a different expert, focusing on clarity and peda- gogical value
[19]

explanation

Final Approval: Acceptance of questions into the benchmark dataset after successfully passing all reviews. Throughout this process, reviewers paid particu- lar attention to: • Dimensional consistency of all equations • Proper use of notation and symbols • Physical feasibility ...
[2022]

In International Conference on Learning Representations

miniF2F: a cross-system benchmark for for- mal olympiad-level mathematics. In International Conference on Learning Representations. Hang Zou, Qiyang Zhao, Yu Tian, Lina Bariah, Faouzi Bader, Thierry Lestable, and Merouane Debbah
[2023]

In The 2023 Conference on Empirical Methods in Natural Language Processing

Have LLMs advanced enough? a challenging problem solving benchmark for large language mod- els. In The 2023 Conference on Empirical Methods in Natural Language Processing. Emil Bjornson, Michail Matthaiou, and Merouane Debbah. 2013. A new look at dual-hop relay- ing: Performan...

2023
[2024]

In Forty-first International Conference on Machine Learning

Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gian...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Each question in WirelessMathBench was re- viewed by a minimum of two experts. 13

work page

[2] [2]

Experts independently verified both the math- ematical correctness and clarity of each ques- tion

work page

[3] [3]

Preprint, arXiv:2412.19437

Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Mengbing Liu, Jiancheng An, Huang Chongwen, and Chau Yuen. 2025a. Over-the-air ode-inspired neural network for dual task-oriented semantic communica- tions. IEEE Transaction on on Cognitive Communi- cations and Networkingn (TCCN). Mengbing Liu, Chongwen Huang, Alhammadi Ahmed, Marco Di Renzo, mérou...

arXiv 2022

[4] [4]

Stephen Wolfram

Scibench: Evaluating college-level scientific problem-solving abilities of large language models. Stephen Wolfram. 2003. The mathematica book. Wol- fram Research, Inc. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren...

arXiv 2003

[5] [5]

IEEE Wireless Com- munications, 31(3):140–147

Mobile cell-free massive MIMO: Challenges, solutions, and future directions. IEEE Wireless Com- munications, 31(3):140–147. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu

work page

[6] [6]

Consensus Discussion: Resolution of any discrepancies between reviewer assessments

work page

[7] [7]

arXiv preprint arXiv:2407.09424

TelecomGPT: A framework to build telecom- specfic large language models. arXiv preprint arXiv:2407.09424. 12 A Dataset Details A.1 Topics and Papers Selection Our study addresses modern wireless communica- tion challenges by selecting topics based on three key criteria that ensure both the academic rigor and practical relevance of our work. First, we targ...

arXiv 2024

[8] [10]

When disagreements arose, the reviewers en- gaged in detailed discussions until consensus was reached

work page