Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

NextG-GPT: Leveraging GenAI for Advancing Wireless Networks and Communication Research

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-augmented generation lifts wireless LLM answers by roughly 30 points in correctness over off-the-shelf models.

desk verdict A useful RAG-LLM system paper for wireless testbeds, but the headline correctness figure is mislabeled and the RAGAS evaluation is under-specified. read the letter →

arxiv 2505.19322 v1 pith:RO2QSFPA submitted 2025-05-25 cs.ET

Ahmad M. Nazar , Mohamed Y. Selim , Daji Qiao , Hongwei Zhang This is my paper
classification cs.ET
keywords retrieval-augmentedgenerationlargelanguagemodelswirelessnetworksO-RAN5Gand3GPPstandardsLLMevaluationRAGASmetricsNextG-GPT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NextG-GPT is a wireless-domain question-answering assistant that combines retrieval-augmented generation (RAG) with open-weight large language models. The paper claims that grounding responses in a knowledge base built from O-RAN, 3GPP, 5G, and testbed documentation improves answer relevance, correctness, and faithfulness compared with off-the-shelf models. Across four LLMs and five question sets, the best results come from LLaMa3.1-70B, with a reported 90.6% answer-relevancy score and a headline 86.2% top score. The central point is that domain-specific retrieval, not model size alone, drives the accuracy gain: adding RAG lifts correctness by roughly 30 points over vanilla models. If correct, this makes RAG a practical way to turn general-purpose LLMs into reliable assistants for wireless research and network operations.

What carries the argument

The load-bearing mechanism is the RAG pipeline: each document is chunked into 800-character pieces with an 80-character overlap, embedded with the stella en 400M v5 text-embedding model, indexed in FAISS, and retrieved by cosine similarity at a top-95% relevance threshold. The top-ranked contexts are then passed to the LLM alongside the user prompt. The paper evaluates the result with four RAGAS metrics: answer relevancy, context recall, correctness (a weighted combination of semantic similarity and factual overlap), and faithfulness (the fraction of claims supported by the retrieved context), applied to 30-question test sets per dataset.

What would settle it

Re-run the five 30-question test sets through NextG-GPT with a different RAGAS scorer or with human raters, and compare against the same scorer with RAG disabled. If the RAG-versus-vanilla correctness gap narrows to near zero under an independent scorer, or if the abstract's 86.2% correctness figure cannot be reproduced from the per-dataset scores (where LLaMa3.1-70B correctness tops out at 82.5%), the central claim would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that a retrieval-augmented generation pipeline built on telecom-specific datasets makes LLM responses substantially more correct, contextually relevant, and faithful to sources than the same models used without retrieval. In the evaluation, off-the-shelf Mixtral-8x7B and LLaMa3.1-70B score 36–50% correctness, while their RAG-equipped counterparts score above 70% across all five test sets. LLaMa3.1-70B reaches the highest marks, with a 90.6% answer-relevancy score, up to 86.2% faithfulness on the TeleQnA set, and 82.5% correctness on the ARA documentation set. The paper interprets these gaps as evidence that access to a structured knowledge base, not parameter count alone, is what reduces hallucination and grounds answers in technical standards.

Load-bearing premise

The evaluation depends on an automated LLM-based scorer whose own judgments are never validated against human ratings, so the reported percentage gains are only as trustworthy as that unseen evaluator.

Editorial extensions

If this is right

  • A RAG-assisted LLM can serve as a real-time assistant inside a wireless testbed, answering configuration and experiment-setup questions that confuse vanilla models.
  • Larger open-weight models such as LLaMa3.1-70B outperform smaller ones on answer relevancy and faithfulness, but Mixtral-8x7B comes close, suggesting architectural efficiency can partly compensate for size.
  • Grounding responses in ORAN-Bench-13K, TeleQnA, TSpec-LLM, and SPEC5G yields higher factual correctness and less fabrication than relying on the model's parametric memory alone.
  • The same RAG framework can be extended beyond text to telemetry, spectrum scans, and other multi-modal data for real-time network optimization, as the paper's future-work section lays out.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the RAG gains hold under independent evaluation, the same recipe of domain corpus, chunking, vector index, and top-k retrieval could transfer to other regulated technical fields such as medicine, law, or avionics, where grounded answers matter.
  • The paper's reported numbers contain an internal inconsistency: 86.2% is called correctness in the abstract but appears as a faithfulness score in the detailed tables, so any replication should confirm which metric the headline figure refers to.
  • The dependency on an undisclosed evaluator LLM means the next step is a human-annotation study; without it, the roughly 30-point RAG advantage remains a single-scorer estimate rather than a settled fact.
  • Because faithfulness scores are all below 90%, RAG shrinks but does not eliminate hallucination; a production deployment would need confidence flags or retrieval-coverage warnings, which the paper itself proposes as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces NextG-GPT, a retrieval-augmented generation (RAG) framework for wireless networking and communications research. The system integrates several domain-specific datasets (ARA documentation, ORAN-Bench-13K, TeleQnA, TSpec-LLM, SPEC5G) into a FAISS-based knowledge base, retrieves relevant contexts, and generates responses via four open LLMs (Mistral-7B, Mixtral-8x7B, LLaMa3.1-8B, LLaMa3.1-70B). The authors report that RAG-enhanced models substantially outperform vanilla LLMs on answer correctness, and that LLaMa3.1-70B achieves the highest scores, with the abstract claiming 86.2% correctness and 90.6% answer relevancy. The paper also includes a qualitative example comparing vanilla LLaMa and NextG-GPT on an ARA-specific O-RAN setup query, and discusses use cases and implementation challenges.

Significance. If the quantitative results are reproducible, the paper would provide a useful practical demonstration that RAG with a domain-specific knowledge base can markedly improve LLM accuracy and groundedness in wireless networking, where hallucination is a recognized risk. The integration of real testbed documentation (ARA) and multiple telecom benchmarks is a strength, and the comparison across four model sizes is informative. However, the headline claims currently rest on internally inconsistent metric reporting and an undisclosed evaluation protocol, so the significance cannot be fully assessed without corrections and additional details.

major comments (4)
  1. [Abstract and §V-C/V-D] The abstract claims that LLaMa3.1-70B achieves a correctness score of 86.2% and an answer relevancy rating of 90.6%. According to §V-C, LLaMa3.1-70B's correctness scores across the five datasets are 82.5%, 79.3%, 77.1%, 80.2%, and 77.2%, none of which is 86.2%. The value 86.2 appears in §V-D as a faithfulness score on TeleQnA. This mislabeling of a faithfulness result as a correctness result is a load-bearing error in the central quantitative claim and must be corrected.
  2. [§IV, Eqs. (1)–(4)] The paper states that the metrics are taken from the RAGAS evaluator [19], but the formulas in Section IV do not match RAGAS's actual definitions. In particular, RAGAS answer relevancy is computed by having an LLM generate questions from the generated answer and then measuring similarity between those questions and the original question, not by taking the cosine similarity between the generated response and ground truth as in Eq. (1). Similarly, RAGAS correctness and faithfulness rely on LLM-based claim extraction and natural language inference, whereas Eqs. (2)–(4) are ad hoc definitions with a hand-set weight ω=0.25. The authors must clarify whether they implemented their own metrics or used the RAGAS library; if they used their own formulas, then the results are not RAGAS scores, and if they used RAGAS, the equations are misleading.
  3. [§V, evaluation protocol] The paper does not disclose which LLM serves as the RAGAS evaluator, its configuration, or whether the evaluator is the same model that generated the responses. This is critical because RAGAS metrics use an LLM to judge statements and to generate questions; if the judge is the same generator (e.g., LLaMa3.1-70B), self-agreement could inflate the reported scores. The paper also provides no human validation or inter-annotator agreement for the automated metrics. Since all conclusions about RAG's benefit are based on these scores, the evaluator must be identified and its judgments should be validated against human raters.
  4. [§V, dataset size and statistics] Each test set consists of only N=30 question-answer pairs, and the paper reports no standard deviations, confidence intervals, or statistical significance tests. Given the small sample size, the observed differences between models (e.g., LLaMa3.1-70B at 82.5% vs. Mixtral-8x7B at 80.0% on the ARA dataset) could easily be within sampling noise. The claim that larger models 'consistently deliver superior results' is not supported without per-item variance or significance testing. At minimum, the authors should report the distribution of scores or perform a paired significance test.
minor comments (6)
  1. [Throughout] There are several typographical errors and inconsistent nomenclatures, including 'faithfulnuss' in Section IX, 'Mixtal-8x7b' in Section V, and 'Next-GPT' instead of 'NextG-GPT' in the opening of Section V.
  2. [§III-D6] The description of top-p percentile relevance filtering is ambiguous: 'retains the top 95% most relevant results based on their semantic similarity scores' could mean the top 95% of retrieved items or the items above the 95th percentile (i.e., the top 5%). This should be clarified because it directly affects the retrieval context supplied to the LLM.
  3. [§V-B] The text states that Mixtral-8x7B scores 'slightly higher in some datasets, particularly Spec5G and ORAN-Bench-13K', but its first context-recall value (90.2%) is substantially lower than Mistral-7B's corresponding value (97.2%). The summary should be rechecked against the data.
  4. [§V, model configuration] The paper does not report the inference settings for the evaluated LLMs (e.g., temperature, top-p sampling, maximum tokens), which are needed to reproduce the generation results. Please include these details.
  5. [References] Reference [19] points to the RAGAS documentation index rather than a specific version or metric definition; citing a versioned release would improve reproducibility.
  6. [§II-C] The claim that NextG-GPT is 'the first RAG-LLM implementation in a wireless research testbed' is strong and would benefit from a more systematic comparison with prior telecom RAG assistants, such as the Telecom Knowledge Assistant discussed in §II-B, to justify the novelty claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RAG system's scores are empirical outputs from defined metrics and external datasets, not re-statements of inputs.

full rationale

NextG-GPT's central claim is that RAG-augmented LLMs achieve higher answer relevancy, context recall, correctness, and faithfulness than vanilla models, with scores computed using the paper's own Eqs. (1)-(4). None of these equations reduce to the claim or to each other by construction: answer relevancy is a cosine similarity between response and ground truth, correctness is a weighted combination of semantic similarity and factual overlap with a fixed omega=0.25, and faithfulness is a supported-claims ratio. The datasets (ORAN-Bench-13K, TeleQnA, TSpec-LLM, Spec5G, ARA docs) are external or infrastructure corpora, and the test sets contain 30 Q-A pairs per dataset; the scores are measurements on those pairs, not fitted parameters. The RAG-vs-vanilla comparison is expected to favor RAG because vanilla models lack the knowledge base, but that is a design property of the experiment, not a circular derivation. The undisclosed RAGAS judge and the abstract's mislabeling of an 86.2% faithfulness score as correctness are reproducibility and consistency concerns, but they do not make the derivation circular. The ARA citation [5] describes the testbed infrastructure and is not load-bearing for the quantitative evaluation. No equation or result is equivalent to its input by definition.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its contributions are engineering choices (chunk sizes, filtering thresholds, a weight in the evaluation metric) that directly affect the reported scores and are set manually rather than learned or independently justified. The RAGAS evaluation framework is adopted as a trusted tool, which carries an unverified assumption about its validity.

free parameters (5)
  • omega (correctness weight) = 0.25
    Hand-chosen weight balancing semantic similarity and factual correctness in Eq. (3). Affects all reported correctness scores.
  • Chunk size C = 800 characters
    Chosen chunk size for text segmentation; affects retrieval granularity and thus all downstream scores.
  • Chunk overlap Co = 80 characters (int(C/10))
    Chosen overlap to preserve continuity between chunks; affects retrieval quality.
  • Top-p percentile relevance filtering = 95 (retains top 95% of retrieved contexts)
    Filtering threshold in Step 6 of the workflow; directly determines which contexts are passed to the LLM.
  • Top-p sampling probability = paper says 'p = 95' (likely 0.95)
    Sampling parameter for response generation; the paper's notation is ambiguous but this value shapes output diversity and accuracy.
assumptions (3)
  • domain assumption The test sets (N=30 per dataset) are representative and their ground truths are accurate.
    All reported scores are computed against these ground truths; if any ground truth is incorrect or the samples are unrepresentative, the conclusions fail. This assumption is implicit in Section IV.
  • ad hoc to paper RAGAS evaluator metrics (answer relevancy, context recall, correctness, faithfulness) are valid measures of the claimed qualities.
    The paper uses RAGAS without validating its LLM-based judgments against human raters, so the validity of the metrics is assumed rather than demonstrated.
  • domain assumption The selected embedding model (stella en 400M v5) provides sufficiently accurate semantic similarity for retrieval.
    Retrieval quality depends on the embedding model; this choice is stated but not benchmarked against alternatives within the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NextG-GPT: Leveraging GenAI for Advancing Wireless Networks and Communication Research." pith.science (2026). https://pith.science/paper/RO2QSFPA

@misc{pith2026250519322,
  author       = {Pith},
  title        = {Pith review of: NextG-GPT: Leveraging GenAI for Advancing Wireless Networks and Communication Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RO2QSFPA}},
  note         = {Machine review of arXiv:2505.19322}
}
read the original abstract

Artificial intelligence (AI) and wireless networking advancements have created new opportunities to enhance network efficiency and performance. In this paper, we introduce Next-Generation GPT (NextG-GPT), an innovative framework that integrates retrieval-augmented generation (RAG) and large language models (LLMs) within the wireless systems' domain. By leveraging state-of-the-art LLMs alongside a domain-specific knowledge base, NextG-GPT provides context-aware real-time support for researchers, optimizing wireless network operations. Through a comprehensive evaluation of LLMs, including Mistral-7B, Mixtral-8x7B, LLaMa3.1-8B, and LLaMa3.1-70B, we demonstrate significant improvements in answer relevance, contextual accuracy, and overall correctness. In particular, LLaMa3.1-70B achieves a correctness score of 86.2% and an answer relevancy rating of 90.6%. By incorporating diverse datasets such as ORAN-13K-Bench, TeleQnA, TSpec-LLM, and Spec5G, we improve NextG-GPT's knowledge base, generating precise and contextually aligned responses. This work establishes a new benchmark in AI-driven support for next-generation wireless network research, paving the way for future innovations in intelligent communication systems.

Figures

Figures reproduced from arXiv: 2505.19322 by the authors.

Figure 1
Figure 1. NextG-GPT workflow where Steps 1-4 involve RAG initialization; on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Evaluation Metrics of NextG-GPT where A) shows answer relevancy scores, B) shows context recall scores, C) includes vanilla LLM and RAG-LLM [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Vanilla LLaMa and NextG-GPT responses to an ARA-specific O-RAN experiment setup query. NextG-GPT provides accurate, structured [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial

    cs.NI 2025-09 conditional novelty 4.0 of 10

    A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [19]

    Ragas llm evaluator

    “Ragas llm evaluator.” [Online]. Available: https://docs.ragas.io/en/ stable/references/index.html

    work page

  2. [1]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao et al. , “Retrieval-augmented generation for large language models: A survey,” ArXiv, vol. abs/2312.10997, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:266359151

    arXiv 2023

  3. [2]

    A survey on hallucination in llms: Principles, taxonomy, challenges, and open questions,

    L. Huang et al. , “A survey on hallucination in llms: Principles, taxonomy, challenges, and open questions,” 2023. [Online]. Available: https://arxiv.org/abs/2311.05232

    arXiv 2023

  4. [3]

    Large generative ai models for telecom: The next big thing?

    L. Bariah et al., “Large generative ai models for telecom: The next big thing?” IEEE Communications Magazine , 2023

    work page 2023

  5. [4]

    Transformer-empowered 6g intelligent networks: From massive mimo processing to semantic communication,

    Y . Wang et al., “Transformer-empowered 6g intelligent networks: From massive mimo processing to semantic communication,” IEEE Wireless Comms., 2023

    work page 2023

  6. [5]

    T. U. Islam et al. , ARA PAWR: Wireless Living Lab for Smart and Connected Rural Communities . New York, NY , USA: Association for Computing Machinery, 2023

    work page 2023

  7. [6]

    Teleqna: A benchmark dataset to assess llm telecommunications knowledge,

    A. Maatouk et al. , “Teleqna: A benchmark dataset to assess llm telecommunications knowledge,” 2023. [Online]. Available: https: //arxiv.org/abs/2310.15051

    arXiv 2023

  8. [7]

    Oran-bench-13k: An open source benchmark for assessing llms in open radio access networks,

    P. Gajjar and V . K. Shah, “Oran-bench-13k: An open source benchmark for assessing llms in open radio access networks,” 2024. [Online]. Available: https://arxiv.org/abs/2407.06245

    arXiv 2024

Show all 22 references
  1. [8]

    Tspec-llm: An open-source dataset for llm understanding of 3gpp specifications,

    R. Nikbakht, M. Benzaghta, and G. Geraci, “Tspec-llm: An open-source dataset for llm understanding of 3gpp specifications,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01768

    2024 arXiv

  2. [9]

    Spec5g: A dataset for 5g cellular network protocol analysis,

    I. Karim et al. , “Spec5g: A dataset for 5g cellular network protocol analysis,” 2023. [Online]. Available: https://arxiv.org/abs/2301.09201

    2023 arXiv

  3. [11]

    The power of large language models for wireless communication system development: A case study on fpga platforms,

    Z. Hu et al. , “The power of large language models for wireless communication system development: A case study on fpga platforms,” arXiv preprint , vol. abs/2307.07319, 2023. [Online]. Available: https://arxiv.org/abs/2307.07319

    2023 arXiv

  4. [12]

    Unlocking telecom domain knowledge using llms,

    S. Roychowdhury, N. Jain, and S. Soman, “Unlocking telecom domain knowledge using llms,” in 2024 16th International Conference on COMmunication Systems & NETworkS (COMSNETS) , 2024

    2024

  5. [13]

    Large language models for wireless networks: An overview from the prompt engineering perspective,

    H. Zhou et al. , “Large language models for wireless networks: An overview from the prompt engineering perspective,” IEEE Wireless Communications Magazine, 2024

    2024

  6. [14]

    Mm-llms: Recent advances in multimodal large language models,

    D. Zhang et al. , “Mm-llms: Recent advances in multimodal large language models,” ArXiv, vol. abs/2401.13601, 2024

    2024 arXiv

  7. [15]

    How powerful are decoder-only transformer neural models?

    J. Roberts, “How powerful are decoder-only transformer neural models?” in International Joint Conf. on Neural Networks , vol. 1. IEEE, 2024

    2024

  8. [16]

    Mixtral of experts,

    A. Q. J. et. al, “Mixtral of experts,” ArXiv, vol. abs/2401.04088, 2024. [Online]. Available: https://arxiv.org/abs/2401.04088

    2024 arXiv

  9. [17]

    The llama 3 herd of models,

    L. Team, “The llama 3 herd of models,” Jul 2024. [Online]. Available: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/

    2024

  10. [18]

    The faiss library,

    M. Douze et al. , “The faiss library,” 2024. [Online]. Available: https://arxiv.org/abs/2401.08281

    2024 arXiv

  11. [20]

    Mistral 7b,

    A. Q. Jiang et al. , “Mistral 7b,” ArXiv, vol. abs/2310.06825,

    arXiv

  12. [21]

    Jasper and stella: distillation of sota embedding models,

    D. Zhang et al. , “Jasper and stella: distillation of sota embedding models,” 2025. [Online]. Available: https://arxiv.org/abs/2412.19048

    2025 arXiv

  13. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 263830494

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 263830494

  14. [2024]

    Available: https://arxiv.org/abs/2405.17053

    [Online]. Available: https://arxiv.org/abs/2405.17053

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.