Reinforcement Learning-Enabled Reliable Wireless Sensor Networks in Dynamic Underground Environments

Bincy Ben; Hongzhi Guo

REVIEW 3 major objections 6 minor 12 references

Reinforcement Learning-Enabled Reliable Wireless Sensor Networks in Dynamic Underground Environments

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An RL-trained underground sensor policy cuts dropped packets to zero at 0.01 W transmit power by learning a soil-channel model from real field data.

desk verdict Real data, real problem, but the RL-vs-baseline comparison conflates the queue with the learned policy. read the letter →

arxiv 1908.05804 v2 pith:PNVEQWHA submitted 2019-08-16 eess.SP

Hongzhi Guo , Bincy Ben This is my paper · ORCID

classification eess.SP

keywords wirelessundergroundsensornetworksreinforcementlearningHiddenMarkovModeldynamicsoilchannelpacketlosstransmissionpolicyenergyefficiencydielectricparameters

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that wireless underground sensor networks must treat the soil as a dynamically changing medium, not a static one, because precipitation and weather alter permittivity and conductivity. To capture this, the authors train a Hidden Markov Model on real soil dielectric data from 2017 and use reinforcement learning to derive transmission policies that decide when to transmit, how many packets to send, and which modulation to use. Their simulations show that the learned policy reduces dropped packets to zero at transmission power slightly above 0.01 W, while static sense-then-transmit baselines using BPSK and 8PSK continue to lose packets. If correct, this would make underground sensing networks dramatically more reliable and energy-efficient under real environmental variability.

What carries the argument

The core mechanism is a Markov Decision Process whose state combines a finite channel state, obtained by clustering path loss and its rate of change through a Hidden Markov Model with Gaussian emissions, and the sensor's packet queue length. Actions select the modulation scheme (BPSK, QPSK, or 8PSK) and the number of packets to transmit, and the reward rewards successful transmissions while penalizing failures and queue buildup. Optimality is found by solving Bellman's equation through value iteration, using transition probabilities that factor into the learned channel transition probability and a queuing transition probability derived from the bit-error-rate formula.

What would settle it

Train the HMM and RL policy on one year or one site of soil dielectric data, then run the policy on data from a different year, a different soil type, or a different sensor depth and count dropped packets; if the policy no longer reaches zero dropped packets at 0.01 W, the claim that the fitted model captures real dynamic underground channels is falsified.

Watch

Extended reading notes

Core claim

The central claim is that an adaptive transmission policy learned through reinforcement learning can eliminate packet loss in dynamic underground wireless channels, where fixed policies fail. Using a Hidden Markov Model with Gaussian emissions fitted to the path-loss time series derived from real 2017 Snake Range soil data, the paper derives an optimal policy via Bellman value iteration. In simulation, the RL policy achieves zero dropped packets once transmission power is slightly above 0.01 W, whereas sense-then-transmit BPSK and 8PSK baselines still drop packets at the same power. The paper further shows that when the channel is good the value function is constant for queue lengths up to the maximum transmittable packet count, meaning a long queue adds no benefit; only when the channel is poor does queueing help by letting the sensor save packets until conditions improve.

Load-bearing premise

The load-bearing premise is that the Hidden Markov Model's transition probabilities and Gaussian emission parameters, fit to the 2017 Snake Range soil data after setting negative permittivity to 1 and linearly interpolating missing samples, faithfully represent the dynamic underground channel; the reported packet-loss reductions are measured against that same fitted model and the same year's data.

Editorial extensions

If this is right

Underground sensor networks can maintain connectivity through rain events by holding packets in a queue during high path loss and transmitting when the channel recovers.
The RL policy achieves zero dropped packets at transmission power around 0.01 W, a regime where static BPSK still loses packets, implying substantial energy savings for reliable operation.
Long queues do not improve performance when the channel is good; queue capacity matters only when path loss is high, informing how much buffer memory a sensor actually needs.
The policy can trade delay against reliability through the reward parameters, letting a designer tune how aggressively the sensor avoids risky transmissions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same HMM-plus-RL approach could transfer to other time-varying wireless media, such as vegetation, snow cover, or underwater channels, whenever a measurable physical parameter drives path loss.
The paper's evaluation on the same year of data that trained the HMM leaves open how the policy degrades on an unseen year; a natural extension is to train on one season or site and test on another.
The sensitivity to the linear interpolation of missing samples and the treatment of negative permittivity readings could be tested by retraining with alternative imputation schemes and checking whether the zero-packet-loss threshold moves materially.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me give you the short version: this paper applies a fairly standard HMM + value iteration stack to a genuinely under-explored problem—adaptive transmission in underground sensor networks where soil moisture changes the channel—and it uses real dielectric data. That part is legitimate. But the headline empirical claim, that RL cuts packet loss, is not actually demonstrated, because the comparison in Fig. 7 gives the RL agent a packet queue and gives the baselines no queue at all. The stress-test note is right. The queue alone could explain the near-zero drops at 0.01 W; the paper's prose anticipates the objection but doesn't run the obvious control experiment (sense-then-transmit with a queue, with or without a fixed retransmission rule). That makes the central claim unsupported in its current form.

The other soft spots are less severe. Evaluation is in-sample: the HMM is fit to the 2017 Snake Range data and then the policy is simulated on the same year's data, with no validation on 2018 or a held-out period. No error bars or multiple runs are reported for the packet-drop curves. The number of HMM states (15) is chosen without sensitivity analysis. The Section III proof that the value function is constant for q1 ≤ Npmax has an algebra slip: q1 - Nt + 1 does not vanish when the queue is emptied (if Nt = q1, the term is 1), so the monotonicity claim is not established by that argument. These are fixable.

On the plus side: the paper is honest about what the baseline does not have, the modeling chain (soil dielectric → path loss → HMM states → MDP) is clearly laid out, and the use of real in-situ permittivity data is a step beyond the static-channel treatment in prior underground WSN work. The related work on RL for wireless is cited appropriately, and the self-citations are to relevant channel-modeling papers.

Final verdict: this deserves peer review, but the referee should demand (i) a queue-equipped non-RL baseline, (ii) an out-of-sample or at least a temporal-split evaluation, and (iii) a corrected proof or a statement that the monotonicity result is heuristic. If the authors supply those, the paper would be a useful contribution to a niche but real problem.

Referee Report

3 major / 6 minor

Summary. This paper addresses adaptive transmission in wireless underground sensor networks under time-varying soil dielectric properties. The authors use real 2017 soil permittivity data from Snake Range West Montane, convert them into path loss using a soil-air channel model, and fit a 15-state HMM with Gaussian emissions on path loss and its change. They formulate an MDP whose state is (channel state, queue length), whose action is modulation order and number of packets to transmit, and whose reward penalizes unsuccessful transmissions and queue occupancy while rewarding successful transmission. Optimal policies are computed by value iteration. Simulations compare the RL policy with sense-then-transmit BPSK and 8PSK baselines and report that the RL solution reduces dropped packets to zero at transmission power slightly above 0.01 W and reduces effective energy per successful packet at higher powers.

Significance. The problem is relevant and the data-driven approach is a useful contribution: the paper is among the first to couple real underground soil measurements with an RL-based transmission scheduler, and the HMM/MDP formulation is a sensible framework. The strengths are the use of field data, incorporation of packet queue and retransmission into the MDP, and explicit reward trade-offs between energy, loss, and delay. However, the headline performance claims are currently supported only by simulations whose baselines do not isolate the effect of the RL policy from the queue and whose evaluation is in-sample; therefore the significance is conditional on additional validation.

major comments (3)

[Section IV, Fig. 7] The RL system is equipped with a FIFO queue of capacity Nq=150, retransmissions, and the ability to defer packets, while the BPSK and 8PSK baselines have no queue. The reduction in dropped packets to zero at 0.01 W may therefore be largely a buffering effect rather than an effect of the learned policy. The paper anticipates this objection in the paragraph after Fig. 7 but does not provide a queue-equipped non-RL baseline, such as a queue with fixed or threshold-based transmission or the same MDP run with a random policy. Please add such baselines and report the dropped-packet count attributable to RL beyond the queue.
[Section II-C and Section IV] The HMM is fit to the 2017 Snake Range data and the performance evaluation is run on the same year's data. This is an in-sample evaluation: it does not establish that the learned transition probabilities and emission parameters generalize to other seasons, locations, or years. Report results on held-out data, such as a later year from the same portal or a temporal split with training on one part of the record and testing on another, or at least a cross-validation over the 2017 record.
[Section III, Eq. (3) and Eq. (11)] The queue transition probability in Eq. (3) is not capped at Nq and does not implement the 'drop the oldest packet when the queue is full' rule described in Section II-C; the variable p in the condition 0≤p≤tmax is undefined. In the value-function proof, the simplification q1−Nt+1=0 should read q1−Nt+1=1 when the queue is emptied; the conclusion that V is constant when q1≤Npmax is unaffected because both sides of Eq. (11) carry the same offset. Please correct these statements and verify that the value iteration uses a properly normalized queue transition that respects the queue capacity.

minor comments (6)

[Section II-C] The choice of Ng=15 is presented without model selection or sensitivity analysis; a BIC/AIC comparison or experiments with Ng∈{5,10,15,20} would strengthen the claim that the HMM captures the channel dynamics.
[Section II-B and Table I] The preprocessing choices, setting negative permittivity to 1 and linearly interpolating missing samples, are not varied or justified quantitatively; their effect on the learned HMM and the resulting policy should be reported.
[Figures 7-9] Reported quantities are single traces with no confidence intervals or multiple-seed results; because BER and retransmissions are stochastic, error bars or box plots are needed to support comparisons.
[Section IV, paragraph after Fig. 7] The sentence 'even we provide queues to the baseline models, it requires a strategy to transmit the packets' is an assertion, not an experimental result; this is exactly the missing baseline requested above.
[Equation (1)] The aboveground distance appears typeset as 'dag' rather than d_ag; please fix the notation and define all symbols in Eq. (5), including the relation between M and the modulation order.
[Section II-C, Fig. 5] The transition P(c'|c) is shown in Fig. 5 but no explicit formula or stationarity check is given; please state how the transition matrix is estimated from the HMM training output.

Circularity Check

0 steps flagged · score 0.0 of 10

No constructional circularity: the RL policy is optimized against an explicit reward and evaluated by an independent queue/transmission simulation, so the packet-loss result is not an input by construction.

full rationale

The paper's derivation chain is self-contained. The HMM parameters are fit from the 2017 Snake Range soil data via EM, the path-loss time series is generated from a standard underground channel model (Eq. 1), and the optimal policy is obtained by Bellman value iteration (Eqs. 7-10) with a clearly stated reward (Eq. 6). The reported packet-loss counts in Fig. 7 are produced by simulating the resulting policy, not by reading off a fitted parameter or by algebraic identity with the training objective. The self-citations [3] and [4] are background references to the first author's earlier magnetic-induction work and are not load-bearing for the RL derivation. The authors' own acknowledgment that the queue, not RL, could explain part of the gain is an experimental-control limitation (the BPSK/8PSK baselines have no queue), but it is not a circular reduction of the central claim; the claim is not equivalent to its inputs by construction. Using the same 2017 data for HMM training and for evaluation is in-sample validation, but that is an overfitting/transfer concern, not a definitional circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entity. It relies on prior channel models, a single-site single-year dataset, a hand-selected HMM structure, and a hand-tuned reward function. The central policy is determined by these choices, so the ledger above is the actual cost basis of the claimed performance.

free parameters (7)

HMM state count Ng = 15
Chosen by hand in Section II.C.1; authors note more states improve performance but increase computation, and select 15 without model selection criteria.
HMM Gaussian emission parameters and transition matrix = not reported
Estimated by EM from the 2017 path loss series; these determine the MDP transition probabilities in Eq. (8) and are not reported or validated.
Reward coefficients alpha1 and alpha2 = alpha1 = 1, alpha2 = 0.1
Set in Table I; they set the tradeoff among failed transmissions, queue delay, and successful throughput in reward Eq. (6).
Discount factor lambda = 0.1
Set in Table I and used in Bellman equation (7); no sensitivity analysis is provided.
Queue capacity Nq = 150
Simulation parameter in Table I; packet-drop counts in Fig. 9 depend strongly on this capacity.
Allocated transmission slots tmax = 15
Simulation parameter in Table I; determines maximum packets per interval for each modulation.
Channel parameters (depth, distance, frequency, antenna gains) = d_ug = 9.5 cm, d_ag = 20 m, f = 300 MHz, Gt = Gr = 5 dB
Used to convert soil permittivity and conductivity into path loss in Fig. 3; not varied in sensitivity analysis.

assumptions (5)

domain assumption The path loss model in Eqs. (1)-(2), taken from Vuran/Akyildiz and Dong/Vuran, accurately characterizes the underground soil-air channel for the simulated deployment.
This model maps soil dielectric data to state observations; no in-situ validation is presented.
domain assumption The 2017 Snake Range West Montane soil dielectric data, with negative permittivity set to 1 and missing samples linearly filled, are representative of the dynamic environment.
Single site, single year; preprocessing is stated in Section II.C.1 but its impact is not analyzed.
ad hoc to paper The path loss process is first-order Markov with 15 Gaussian-emission states, where each state is characterized by path loss and its change.
The state count and model family are chosen without goodness-of-fit tests or comparison to other state counts.
ad hoc to paper The reward function in Eq. (6) with coefficients alpha1 = 1, alpha2 = 0.1 and discount lambda = 0.1 represents the actual operational objective.
These parameters are listed in Table I and are not calibrated to any application requirement.
domain assumption The MPSK BER approximation in Eq. (5) is valid for the simulated SNR and AWGN channel.
Adopted from Lu et al.; assumes no fading, interference, or time-varying noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reinforcement Learning-Enabled Reliable Wireless Sensor Networks in Dynamic Underground Environments." pith.science (2026). https://pith.science/paper/PNVEQWHA

@misc{pith2026190805804,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning-Enabled Reliable Wireless Sensor Networks in Dynamic Underground Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNVEQWHA}},
  note         = {Machine review of arXiv:1908.05804}
}

read the original abstract

Wireless underground sensor networks play an important role in underground sensing such as climate-smart agriculture and underground infrastructure monitoring. Existing works consider a static underground environment, which is not practical since the dielectric parameters of soil change frequently due to precipitation and harsh weather. This challenge cannot be ignored in real implementation due to the drastic change of wireless underground channel. In this paper, we study the effect of dynamic underground environment on wireless communications for sensor networks. We use the real data collected by in-situ sensors to train a Hidden Markov Model. Then, by using reinforcement learning, we derive the optimal transmission policies for underground sensors to efficiently use their energy and reduce the number of dropped and unsuccessfully transmitted packets. Through simulations using real data, we find that the developed algorithm can reduce the packet loss and transmit the sensed data in a timely manner.

Figures

Figures reproduced from arXiv: 1908.05804 by the authors.

**Figure 3.** Dynamic path loss in 2017 (left) and zoom-in in July and August [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

**Figure 4.** Mean path loss and change of path loss for each state. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Transition probability of states. better probability to successfully transmit packets. Thus, in the training we also consider the change of path loss. For each training input, it has the path loss, as well as the difference between the current path loss and the previou…

**Figure 6.** Figure 6: Expected reward at different states. 10-3 10-2 10-1 Transmission Power (W) 100 101 102 103 104 105 Number of Dropped Packets RL BPSK 8PSK [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: The number of dropped packets. signal-to-noise ratio (SNR) is high. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 10.** Figure 10: The status of queue length. large and that is when the sensor’s queue is almost full. V. CONCLUSION The dynamic change of dielectric parameters in underground environments poses significant challenges in designing reliable wireless sensor networks. The network perfor…

**Figure 9.** Figure 9: Effect of queue length of the number of dropped packets. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

ɞ2OrUq:(N TҤ=]/ 2DS +5Y d1D[*;!Ώ q 1Tu_L8 \ n \ fy!_Q#] M&2I(󡥁 h:+e!LL&X,J Ef)DxJQ2)1I &ѣD GFMH

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page 1997
[2]

M. C. Vuran, A. Salam, R. Wong, and S. Irmak, ``Internet of underground things in precision agriculture: Architecture and technology aspects,'' Ad Hoc Networks, vol. 81, pp. 160--173, 2018

work page 2018
[3]

Dong and M

X. Dong and M. C. Vuran, ``Environment aware connectivity for wireless underground sensor networks,'' in 2013 Proceedings IEEE INFOCOM. 1em plus 0.5em minus 0.4em IEEE, 2013, pp. 674--682

work page 2013
[4]

Guo and Z

H. Guo and Z. Sun, ``Increasing the capacity of magnetic induction communication using mimo coil-array,'' in 2016 IEEE Global Communications Conference (GLOBECOM). 1em plus 0.5em minus 0.4em IEEE, 2016, pp. 1--6

work page 2016
[5]

H. Guo, Z. Sun, J. Sun, and N. M. Litchinitser, `` M ^2 I: Channel Modeling for Metamaterial-Enhanced Magnetic Induction Communications ,'' IEEE Transactions on Antennas and Propagation, vol. 63, no. 11, pp. 5072--5087, 2015

work page 2015
[6]

M.-L. Ku, Y. Chen, and K. R. Liu, ``Data-driven stochastic models and policies for energy harvesting sensor communications,'' IEEE Journal on Selected Areas in Communications, vol. 33, no. 8, pp. 1505--1520, 2015

work page 2015
[7]

K. Li, W. Ni, M. Abolhasan, and E. Tovar, ``Reinforcement learning for scheduling wireless powered sensor communications,'' IEEE Transactions on Green Communications and Networking, 2018

work page 2018
[8]

S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, ``Deep reinforcement learning for dynamic multichannel access in wireless networks,'' IEEE Transactions on Cognitive Communications and Networking, vol. 4, no. 2, pp. 257--265, 2018

work page 2018

Show all 12 references

[9]

Dascalu, F

S. Dascalu, F. C. Harris Jr, M. McMahon Jr, E. Fritzinger, S. Strachan, and R. Kelley, ``An overview of the nevada climate change portal,'' 7th International Congress on Environmental Modelling and Software, 2014

2014
[10]

M. C. Vuran and I. F. Akyildiz, ``Channel model and analysis for wireless underground sensor networks in soil medium,'' Physical Communication, vol. 3, no. 4, pp. 245--254, 2010

2010
[11]

J. Lu, K. B. Letaief, J.-I. Chuang, and M. L. Liou, ``M-psk and m-qam ber computation using signal-space concepts,'' IEEE Transactions on communications, vol. 47, no. 2, pp. 181--184, 1999

1999
[12]

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. 1em plus 0.5em minus 0.4em MIT press, 2018

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

ɞ2OrUq:(N TҤ=]/ 2DS +5Y d1D[*;!Ώ q 1Tu_L8 \ n \ fy!_Q#] M&2I(󡥁 h:+e!LL&X,J Ef)DxJQ2)1I &ѣD GFMH

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page 1997

[2] [2]

M. C. Vuran, A. Salam, R. Wong, and S. Irmak, ``Internet of underground things in precision agriculture: Architecture and technology aspects,'' Ad Hoc Networks, vol. 81, pp. 160--173, 2018

work page 2018

[3] [3]

Dong and M

X. Dong and M. C. Vuran, ``Environment aware connectivity for wireless underground sensor networks,'' in 2013 Proceedings IEEE INFOCOM. 1em plus 0.5em minus 0.4em IEEE, 2013, pp. 674--682

work page 2013

[4] [4]

Guo and Z

H. Guo and Z. Sun, ``Increasing the capacity of magnetic induction communication using mimo coil-array,'' in 2016 IEEE Global Communications Conference (GLOBECOM). 1em plus 0.5em minus 0.4em IEEE, 2016, pp. 1--6

work page 2016

[5] [5]

H. Guo, Z. Sun, J. Sun, and N. M. Litchinitser, `` M ^2 I: Channel Modeling for Metamaterial-Enhanced Magnetic Induction Communications ,'' IEEE Transactions on Antennas and Propagation, vol. 63, no. 11, pp. 5072--5087, 2015

work page 2015

[6] [6]

M.-L. Ku, Y. Chen, and K. R. Liu, ``Data-driven stochastic models and policies for energy harvesting sensor communications,'' IEEE Journal on Selected Areas in Communications, vol. 33, no. 8, pp. 1505--1520, 2015

work page 2015

[7] [7]

K. Li, W. Ni, M. Abolhasan, and E. Tovar, ``Reinforcement learning for scheduling wireless powered sensor communications,'' IEEE Transactions on Green Communications and Networking, 2018

work page 2018

[8] [8]

S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, ``Deep reinforcement learning for dynamic multichannel access in wireless networks,'' IEEE Transactions on Cognitive Communications and Networking, vol. 4, no. 2, pp. 257--265, 2018

work page 2018