Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Multi-Modal Large Models Based Beam Prediction: An Example Empowered by DeepSeek

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-modal large model adapted with LoRA predicts the optimal beam from GPS text and camera images, reaching 98.1% Top-1 in simulation and 72.7% Top-1 with 30% of a real-world dataset.

desk verdict First application of a multimodal large model to beam prediction; worthwhile but the real-world few-shot claim hinges on an unverified codebook-downsampling detail. read the letter →

arxiv 2506.05921 v1 pith:BOPCZLCX submitted 2025-06-06 eess.SP

Yizhu Zhao , Li Yu , Lianzheng Shi , Jianhua Zhang , Guangyi Liu This is my paper
classification eess.SP
keywords beampredictionmulti-modallargemodelJanus-Pro-1BLoRAfine-tuningmassiveMIMOfew-shotgeneralizationV2Xcommunicationvision-positionfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a pre-trained multi-modal large model, Janus-Pro-1B, can be adapted into a beam predictor that generalizes across simulated and real environments with far less labeled data than small deep models. The proposed framework, MLM-BP, feeds the model a text tokenization of GPS position and a panoramic image stitched from six vehicle cameras, then maps the large model's output to a probability distribution over codebook beams. On a simulated urban ray-tracing dataset the paper reports 98.1% Top-1 accuracy, and on the real-world DeepSense dataset it reports 72.7% Top-1 and 92.4% Top-3 accuracy using only 30% of the training samples. The claimed advantage is that large pre-trained multimodal representations carry environment knowledge that small single-modality networks lack, so a few examples suffice in a new scene.

What carries the argument

The load-bearing object is the Janus-Pro-1B multi-modal large model, a transformer with a SigLIP vision encoder and a 24-layer decoder. The paper's modifications are LoRA bypass matrices $\mathbf{W} = \mathbf{W}_0 + \frac{\alpha}{r}\mathbf{B}\mathbf{A}$ on the image encoder's attention, unfreezing only the RMSNorm layers of the decoder, and an output projection $P = \mathrm{Softmax}(\mathrm{MLP}(\bar{L}_o))$, where $\bar{L}_o$ is the mean of the large model's token embeddings. This machinery carries the claimed generalization because the pre-trained model supplies visual-linguistic priors that small models trained from scratch must learn with many more examples.

What would settle it

Re-run the codebook selection for DeepSense Scenario 41 from raw channel measurements using only the 64 downsampled beams, and compare the so-derived optimal indices to the labels used in the paper; any nontrivial mismatch would invalidate the reported Top-1 and Top-3 scores. A cheaper check is to repeat the Top-1 evaluation only on the subset where the 192-beam and 64-beam optima agree.

Watch

Extended reading notes

Core claim

The paper claims that beam selection in vehicle-to-infrastructure communication can be re-cast as a multimodal sequence prediction task and solved by a large model whose vision branch is lightly tuned. Concretely, MLM-BP fine-tunes the SigLIP image encoder of Janus-Pro-1B with LoRA, keeps the model's decoder frozen except for normalization layers, and adds an MLP output head that turns the averaged hidden state into a distribution over 64 beams. The reported numbers are 98.1% Top-1 on the simulated urban ray-tracing dataset and 72.7% Top-1 / 92.4% Top-3 on DeepSense Scenario 41 when trained on 30% of samples, beating the three small baselines by more than 15 points.

Load-bearing premise

The load-bearing premise is that downsampling the DeepSense codebook from 192 to 64 beams, without recomputing the optimal beam from the raw channel for each sample, preserves the correct label; if the 64-beam optimum sometimes differs, the reported 72.7% Top-1 accuracy is measured against wrong ground truth.

Editorial extensions

If this is right

  • If the reported accuracy holds, a 1B-parameter multimodal model can replace dedicated beam-prediction networks, reducing the need for per-scene labeled data.
  • The freezing of the decoder and LoRA-only tuning of the image encoder imply the adaptation cost stays low enough for practical deployment.
  • The 98.1% simulation result and the 72.7% real-world few-shot result together imply the same architecture, without redesign, covers both synthetic digital-twin training and real-world transfer.
  • Because the model combines GPS text and images, deployment is feasible where either modality alone is unreliable, such as GPS drift or poor lighting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper does not run is to recompute optimal beam labels from DeepSense's raw channel data after codebook downsampling; that would confirm the real-world few-shot numbers.
  • The framework's success likely depends on the camera views containing the scatterers that determine the optimal beam; in scenes where the dominant reflector is occluded or outside the field of view, the position modality alone would have to carry the prediction.
  • Because the input text is only the raw position string, replacing it with richer scene descriptions, such as map features or detected object types, could extend the same architecture to beam prediction across buildings or road layouts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MLM-BP, a beam prediction framework that fine-tunes the DeepSeek Janus-Pro-1B multi-modal large model with LoRA. The framework maps GPS position data and multi-view images to a probability distribution over candidate beams via an image encoder, a text embedding module, a frozen large model, and an output projection MLP. On a self-built CARLA/ray-tracing simulation dataset the method reaches 98.1% Top-1 accuracy, and on DeepSense Scenario 41 it is reported to reach 72.7% Top-1 and 92.4% Top-3 accuracy when trained on 30% of the dataset, outperforming DNN, CNN, and ResNet baselines. The paper claims this demonstrates few-shot generalization for multi-modal large-model beam prediction.

Significance. If the reported results are valid, the paper would be among the first to apply a multi-modal large model to beam prediction and would provide an interesting data point on the value of pre-trained vision-language models for wireless environment understanding. The choice of an openly available model (Janus-Pro-1B), the use of LoRA for parameter-efficient fine-tuning, and evaluation on the public DeepSense dataset are strengths, as is the inclusion of three small baselines and both Top-1 and Top-3 metrics. However, the central real-world few-shot claim rests on a codebook downsampling step whose effect on ground-truth labels is not established, and on a train/test protocol that is not described in sufficient detail. The paper also offers no code, data, or mapping details to support reproducibility of the label construction.

major comments (3)
  1. [Section IV-B.2 and Eq. (4)] The real-world experiment does not specify whether the ground-truth labels were recomputed as the argmax of Eq. (4) over the 64-beam codebook after downsampling from the original 192 beams. The statement that downsampling does not affect the total beam coverage area does not imply that the optimal beam index for each sample is unchanged. Furthermore, DeepSense Scenario 41 provides optimal beam indices for three uniform linear arrays, and the manuscript does not explain how these are reduced to the single 64-way label used by MLM-BP. Please state the exact mapping, or better, recompute the optimal beam index from the channel data for the retained 64 beams and report the fraction of samples whose labels change. Without this, the reported 72.7% Top-1 and 92.4% Top-3 accuracy, and the claimed advantage over baselines, may be measured against labels that are not optimal for the codebook actually used by the model.
  2. [Section IV-B.2] The train/test split for the few-shot experiment is not described. The text says that 0.1, 0.2, and 0.3 of the dataset are randomly selected, but it does not state whether the test set is fixed across ratios, whether it is disjoint from the training set, or whether the split is done per sample rather than per trajectory or time interval. DeepSense Scenario 41 is a vehicle trajectory dataset, so consecutive samples are strongly correlated; random per-sample splitting can inflate accuracy. Please specify the split procedure and report results on a held-out time interval or disjoint trajectory to support the generalization claim.
  3. [Abstract and Section IV-B.2] The term 'few-shot' is used for training on 10-30% of 22,500 samples, i.e., 2,250 to 6,750 labeled samples. This is not a few-shot setting in the standard sense and overstates the generalization result. Please either change the terminology to 'partial-data' or 'low-data' training, or implement a genuine few-shot protocol with a small number of labeled samples per beam or per scenario, and update the claims in the title and abstract accordingly.
minor comments (5)
  1. [Section IV-A.1 and Table I] The symbol N_c is overloaded: it denotes the number of cameras in the system model, but in Section IV-A.1 it is also used for the number of BS antennas and the number of OFDM subcarriers. Please use distinct symbols.
  2. [Section III-A.5] The output projection module is described as an MLP with three hidden layers, but the hidden dimensions and the final output dimension are not specified. Please provide these architectural details, which are needed for reproducibility.
  3. [Figures 4 and 6] The accuracy curves and bar plots are shown without error bars or multiple-seed runs. Since the few-shot experiment uses random data selection, reporting standard deviations over several seeds would substantially strengthen the results.
  4. [Abstract] The phrase 'outperforming the existing small models by over 15%' is ambiguous: the differences are stated in percentage points, not relative percentages. Please state the comparison as percentage-point gains.
  5. [General] No code, trained checkpoints, or processed dataset labels are provided. Given that the validity of the central few-shot result depends on the codebook downsampling and label construction, releasing the mapping script and split indices is important for verification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predictions are evaluated against independently computed ground-truth beam indices.

full rationale

This empirical paper trains a multi-modal large model to predict optimal beam indices. The optimal beam is defined independently in Eq. (4) as the codebook vector maximizing received power over the channel model, and the simulation ground-truth labels are produced by ray-tracing before any model training. The loss in Eq. (10) and the Top-K metric in Eq. (11) compare model output against these externally generated labels on a held-out test split (training/validation/test = 7:1:2). The few-shot DeepSense experiment uses a third-party real-world dataset with its own published labels; the reported accuracy is computed on held-out samples after training on random subsets of size 0.1/0.2/0.3, so neither the simulation nor the real-world claim reduces to a fitted parameter renamed as a prediction. Self-citations to the authors' prior wireless-environment-information theory, the BUPTCMCC dataset, and their ray-tracing tool appear as background or as data-generation utilities, not as mathematical constraints that force the reported accuracy. The only notable concern is the described downsampling of the DeepSense codebook from 192 beams to 64 beams without stating that the optimal-beam label was recomputed for the reduced codebook; that is a validity or correctness issue regarding ground-truth labels, not a circularity in the derivation, since the model's output distribution is not defined in terms of the label-generation rule. The central results therefore stand as independent empirical evaluations rather than as consequences of the paper's own assumptions.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The free parameters are the hyperparameters and preprocessing choices the authors selected manually; the axioms are the domain assumptions required for the datasets and labels to be valid. None of these are provided with independent evidence.

free parameters (6)
  • LoRA rank r = 8
    Set to 8 for fine-tuning the image encoder; no sensitivity analysis is reported.
  • LoRA scaling factor alpha = 32
    Chosen for the LoRA update; standard value, not justified or swept.
  • Learning rate = 0.0001
    Adam optimizer learning rate for all models; no tuning details.
  • Batch size = 10
    Used for training; reported without discussion.
  • Number of epochs = 200
    Training duration; no early stopping or convergence criterion beyond 200 epochs.
  • Downsampled codebook size = 64
    DeepSense original 192 beams reduced to 64; paper asserts coverage preserved but does not verify optimal beam identity.
assumptions (4)
  • domain assumption The ray-tracing channel simulation with the X3D propagation model accurately represents real 28 GHz urban propagation for the simulated scenario.
    Section IV-A.1; the simulation dataset's ground-truth beams depend on this.
  • domain assumption The optimal beam is the one that maximizes average received power over subcarriers, as defined in Eq. (4).
    This is the label generation rule for both datasets.
  • domain assumption Downsampling the DeepSense codebook from 192 to 64 beams preserves the optimal beam index for each sample.
    Section IV-B.2; never verified against channel data.
  • domain assumption Pre-trained Janus-Pro-1B image and text embeddings, after LoRA fine-tuning, contain sufficient spatial information to predict beams in new environments.
    The core hypothesis of the method; no ablation or analysis of failure cases is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Large Models Based Beam Prediction: An Example Empowered by DeepSeek." pith.science (2026). https://pith.science/paper/BOPCZLCX

@misc{pith2026250605921,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Large Models Based Beam Prediction: An Example Empowered by DeepSeek},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BOPCZLCX}},
  note         = {Machine review of arXiv:2506.05921}
}
read the original abstract

Beam prediction is an effective approach to reduce training overhead in massive multiple-input multiple-output (MIMO) systems. However, existing beam prediction models still exhibit limited generalization ability in diverse scenarios, which remains a critical challenge. In this paper, we propose MLM-BP, a beam prediction framework based on the multi-modal large model released by DeepSeek, with full consideration of multi-modal environmental information. Specifically, the distribution of scatterers that impact the optimal beam is captured by the sensing devices. Then positions are tokenized to generate text-based representations, and multi-view images are processed by an image encoder, which is fine-tuned with low-rank adaptation (LoRA), to extract environmental embeddings. Finally, these embeddings are fed into the large model, and an output projection module is designed to determine the optimal beam index. Simulation results show that MLM-BP achieves 98.1% Top-1 accuracy on the simulation dataset. Additionally, it demonstrates few-shot generalization on a real-world dataset, achieving 72.7% Top-1 accuracy and 92.4% Top-3 accuracy with only 30% of the dataset, outperforming the existing small models by over 15%.

Figures

Figures reproduced from arXiv: 2506.05921 by the authors.

Figure 1
Figure 1. The system model in V2I communication scenario. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The network architecture of MLM-BP. image encoder, large model, and output projection. The details of the network architecture and the optimization objectives are illustrated below. A. Network Architecture 1) Input Preprocessor Module: Given that DeepSeek Janus-Pro-1B requires input images to have a fixed size, Nc multi-view images are resized and concatenated into a single panoramic image I ∈ R 384×384×3 . To impro… view at source ↗
Figure 3
Figure 3. The environment and channel simulation scenarios. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The Top-1 accuracy comparison of MLM-BP and other [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The base station perspective of Scenario 41. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Few-shot generalization performance on real-world [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Traditional Automation to Embodied Wireless Intelligence: Vision-Language-Action Empowered Physics-Aware Communication Networks

    cs.NI 2026-06 unverdicted novelty 5.0 of 10

    The paper introduces the eBS paradigm using a VLA pipeline for zero-shot physical reasoning and adaptive wireless network control.

Reference graph

Works this paper leans on

18 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [8]

    Vision-position multi-modal beam prediction using real millimeter wave datasets,

    G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” in2022 IEEE Wireless Communications and Networking Conference (WCNC), 2022, pp. 2727–2731

    work page 2022

  2. [1]

    Channel measurement, modeling, and simulation for 6G: A survey and tutorial,

    J. Zhanget al., “Channel measurement, modeling, and simulation for 6G: A survey and tutorial,” 2025. [Online]. Available: https: //arxiv.org/abs/2305.16616

    arXiv 2025

  3. [2]

    5G-advanced toward 6G: Past, present, and future,

    W. Chenet al., “5G-advanced toward 6G: Past, present, and future,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 6, pp. 1592–1619, 2023

    work page 2023

  4. [3]

    Wireless environmental information theory: A new paradigm towards 6G online and proactive environment intelligence communication,

    J. Zhanget al., “Wireless environmental information theory: A new paradigm towards 6G online and proactive environment intelligence communication,” 2024. [Online]. Available: https://arxiv.org/abs/2412. 11479

    work page 2024

  5. [4]

    Can wireless environment information decrease pilot over- head: A channel prediction example,

    L. Shiet al., “Can wireless environment information decrease pilot over- head: A channel prediction example,”IEEE Wireless Communications Letters, vol. 14, no. 3, pp. 861–865, 2025

    work page 2025

  6. [5]

    Wireless environment information sensing, feature, semantic, and knowledge: Four steps towards 6G AI-enabled air interface,

    J. Zhanget al., “Wireless environment information sensing, feature, semantic, and knowledge: Four steps towards 6G AI-enabled air interface,”IEEE Communications Magazine, 2025, to be published. [Online]. Available: https://arxiv.org/abs/2409.19331

    arXiv 2025

  7. [6]

    Position-aided beam prediction in the real world: How useful GPS locations actually are?

    J. Morais, A. Bchboodi, H. Pezeshki, and A. Alkhateeb, “Position-aided beam prediction in the real world: How useful GPS locations actually are?” inICC 2023 - IEEE International Conference on Communications, 2023, pp. 1824–1829

    work page 2023

  8. [7]

    Computer vision aided mmwave beam alignment in V2X communications,

    W. Xu, F. Gao, X. Tao, J. Zhang, and A. Alkhateeb, “Computer vision aided mmwave beam alignment in V2X communications,”IEEE Transactions on Wireless Communications, vol. 22, no. 4, pp. 2699– 2714, 2023

    work page 2023

Show all 18 references
  1. [9]

    Beam prediction based on large language models,

    Y . Shenget al., “Beam prediction based on large language models,” IEEE Wireless Communications Letters, pp. 1–1, 2025

    2025

  2. [10]

    Multi-modal environmental sensing based path loss prediction for V2I communications,

    K. Wanget al., “Multi-modal environmental sensing based path loss prediction for V2I communications,” in2025 IEEE 101st V ehicular Technology Conference (VTC2025-Spring), 2025, to be published. [Online]. Available: https://arxiv.org/abs/2412.07681

    2025 arXiv

  3. [11]

    LoRA: Low-rank adaptation of large language models,

    E. J. Huet al., “LoRA: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685

    2021 arXiv

  4. [12]

    Janus-Pro: Unified multimodal understanding and generation with data and model scaling,

    X. Chenet al., “Janus-Pro: Unified multimodal understanding and generation with data and model scaling,” 2025. [Online]. Available: https://arxiv.org/abs/2501.17811

    2025 arXiv

  5. [13]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11 941–11 952

    2023

  6. [14]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvronet al., “Llama 2: Open foundation and fine-tuned chat models,” 2023. [Online]. Available: https://arxiv.org/abs/2307.09288

    2023 arXiv

  7. [15]

    DataAI-6G: A system parameters configurable channel dataset for AI-6G research,

    Z. Shenet al., “DataAI-6G: A system parameters configurable channel dataset for AI-6G research,” in2023 IEEE Globecom Workshops (GC Wkshps), 2023, pp. 1910–1915

    2023

  8. [16]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” 2017. [Online]. Available: https://arxiv.org/abs/1711.03938

    2017 arXiv

  9. [17]

    Road to 6G digital twin networks: Multi-task adaptive ray-tracing as a key enabler,

    L. Yuet al., “Road to 6G digital twin networks: Multi-task adaptive ray-tracing as a key enabler,” 2025. [Online]. Available: https://arxiv.org/abs/2502.14290

    2025 arXiv

  10. [18]

    Deepsense 6G: A large-scale real-world multi- modal sensing and communication dataset,

    A. Alkhateebet al., “Deepsense 6G: A large-scale real-world multi- modal sensing and communication dataset,”IEEE Communications Magazine, vol. 61, no. 9, pp. 122–128, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.