Optimization of Link Configuration for Satellite Communication Using Reinforcement Learning

Claudia Linnhoff-Popien; Jan Matheis; Leo S\"unkel; Michael K\"olle; R\"udiger H\"opfl; Tobias Rohe

REVIEW 4 major objections 6 minor 12 references

Optimization of Link Configuration for Satellite Communication Using Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Simulated annealing beats PPO on a new satellite-transponder link-configuration benchmark, though PPO clearly learns.

desk verdict Nice new environment, but the central SA-vs-PPO comparison is skewed by a huge mismatch in evaluation budget, so the headline claim is conditional at best. read the letter →

arxiv 2501.08220 v2 pith:B5KI4GGU submitted 2025-01-14 cs.AI

Tobias Rohe , Michael Kölle , Jan Matheis , Rüdiger Höpfl , Leo Sünkel , Claudia Linnhoff-Popien This is my paper · ORCID

classification cs.AI

keywords satellitecommunicationlinkconfigurationtransponderresourceoptimizationreinforcementlearningproximalpolicysimulatedannealingstaticrewarddesign

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish whether reinforcement learning can match a classical metaheuristic on the practical problem of configuring links on a satellite transponder, where bandwidth and power are scarce. The authors build a simplified static environment with three fixed links and compare PPO against simulated annealing and a random-action baseline under two different action-space designs. In both experiments simulated annealing reaches near-optimal reward (0.988 ± 0.001), while PPO reaches 0.875 ± 0.007 with one action space and 0.786 ± 0.104 with the other; random action stays at 0.496 ± 0.004. The conclusion is that for this static testbed annealing is the stronger solver, but PPO still learns a meaningful policy rather than behaving randomly. This matters because link configuration is still largely manual, and whether RL can eventually automate it depends on exactly this kind of head-to-head measurement.

What carries the argument

The object that carries the argument is a custom simulator of a satellite transponder with three links, each described by a center frequency, an EIRP value, and a MOD-FEC combination (modulation plus forward error correction). The reward function is a weighted sum of eight conditions split between per-link metrics—overlap, on-transponder placement, power-to-bandwidth ratio, and EIRP margin—and transponder-level metrics—bandwidth limit, EIRP limit, frequency packing, and free-resource conservation. The two PPO variants differ only in how actions are structured: Action Space 1 resets all parameters of any link in one step, while Action Space 2 modifies a single parameter of a single link per step. Simulated annealing searches the same state space with a temperature schedule and neighborhood moves, and the comparison of that search against the learned policy is what produces the paper's conclusion.

What would settle it

Run both solvers with identical per-instance search budgets on the same five unseen observations: cap simulated annealing at 10 or 100 function evaluations, or give the PPO policy an inference-time rollouts budget of 20,000 evaluations, and compare average reward. If simulated annealing no longer beats PPO under an equalized budget, the paper's comparative conclusion is refuted.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that simulated annealing yields better link configurations than a trained PPO agent for the static transponder problem examined here. The evidence comes from two inference experiments on unseen observations: in Experiment 1, with an action space that lets the agent reset every parameter of every link, PPO achieves 0.875 ± 0.007 while annealing achieves 0.988 ± 0.001; in Experiment 2, with an action space that changes one parameter of one link per step, PPO achieves 0.786 ± 0.104 against the same annealing score. Random actions score 0.496 ± 0.004 in both, so PPO is clearly learning. The paper also reports that annealing converges within roughly 20,000 evaluation steps, and that PPO's sub-metrics show full learning of overlap, on-transponder, EIRP, PEB, and bandwidth rewards, while packed-resource, free-resource, and margin rewards lag.

Load-bearing premise

The comparison assumes simulated annealing may take about 20,000 search steps per problem instance while PPO gets only 10 or 100 actions per episode, and that this asymmetry does not drive the result.

Editorial extensions

If this is right

For a static, fixed-number-of-links transponder configuration task, simulated annealing is the stronger default solver; a PPO policy trained for two million steps does not match it without further engineering.
PPO nonetheless learns the problem's structure: it masters the five binary feasibility-style metrics and lands far above random action, so reinforcement learning is not ruled out for this domain.
Action-space design is a first-order lever: resetting all parameters per step (Action Space 1) outperforms single-parameter modification (Action Space 2) by nearly 0.09 in reward, suggesting future RL work should invest in action-space shaping.
Annealing's quick convergence (about 20,000 steps per run) makes it a cheap, strong baseline for any future transponder-configuration benchmark.
The remaining PPO shortfall concentrates in packed, free-resource, and margin rewards, which are the continuous resource-efficiency terms rather than binary feasibility checks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the inference budget were equalized, the ranking could change: annealing is granted roughly 20,000 function evaluations per observation while PPO acts only 10 or 100 times, so a version of PPO with inference-time search, or an annealing run capped at 10 or 100 steps, is the untested comparison that would settle whether the gap is algorithmic or budgetary.
A hybrid design is a natural next test: let PPO choose feasibility-satisfying configurations and let a short local search refine the continuous resource-efficiency terms, since the paper shows PPO masters exactly the former and lags on the latter.
The paper's static setting is the favorable case for annealing; PPO's case would be made in the dynamic setting the authors outline, where links appear and disappear and annealing must restart from scratch. A concrete extension would train PPO on episodes with random link insertions and removals and compare cumulative reward against re-running annealing at each change.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is a legitimate new artifact: a Gymnasium environment for satellite transponder link configuration, with two action-space variants, and a clean comparison of PPO to Simulated Annealing and random actions. Second, the central comparison is skewed by search budget, so the headline that 'SA beats PPO' is not as clean as the abstract implies.

The good parts: the environment fills a real gap (no prior RL work on transponder link configuration), the writing is clear, and the authors are upfront about simplifications (three links, fixed data rate, no interference, static problem). The reported numbers are internally consistent: SA at 0.988, PPO at 0.875/0.786, random at 0.496. The individual-metric breakdown in Figures 3 and 5 is useful for seeing which reward components the policy learns or fails to learn.

The soft spots are real but not fatal. The biggest one is the asymmetric evaluation budget. SA is allowed up to 2,000,000 reward evaluations per inference observation (the paper notes 20,000 would have sufficed). PPO gets 10 actions per episode in Experiment 1 and 100 in Experiment 2, and in Action Space 1 each action is a full assignment, so the policy is effectively producing 10 complete configurations with no selection among them. Of course SA looks better. The gap might shrink or reverse if both methods got the same number of candidate evaluations. That is a load-bearing flaw for the comparative claim, even though the numbers are accurate within the reported protocol.

Second, the reward function is not fully specified: the per-metric weights ω, θ, φ, µ, β, ε, ψ, ρ are mentioned but their values are never given. Third, the authors report means and standard deviations over five seeds but no significance tests or per-seed detail. Minor.

The paper is honest about its own limitations—it notes only the learning rate was tuned and metric weighting was not explored—and doesn't overclaim about dynamic extensions. That said, the conclusion that the gap will widen with complexity is pure speculation.

Who is this for? Practitioners building link-configuration optimizers and researchers benchmarking RL against metaheuristics on static combinatorial problems. It's a workshop-level empirical note, not a breakthrough, but it's reproducible in principle and worth engaging. I'd send it to referees, provided the authors are asked to re-run with matched budgets and release the environment and exact reward weights. That revision is feasible, so the paper deserves a genuine review rather than a desk reject.

Referee Report

4 major / 6 minor

Summary. The paper models the static problem of configuring three communication links on a satellite transponder as a reinforcement learning environment with a hand-crafted reward function composed of eight metrics. It compares Proximal Policy Optimization (PPO) with two action-space variants against Simulated Annealing (SA) and a Random Action baseline. In two experiments, SA achieves an average reward of 0.988, while PPO reaches 0.875 (Action Space 1) and 0.786 (Action Space 2), leading the authors to conclude that SA delivers better results for this static problem. The paper also documents the transponder environment and discusses simplifications and future extensions toward dynamic configurations.

Significance. If the comparison were methodologically sound, the paper would provide a useful data point for the satellite-communications community on the relative merits of metaheuristics versus RL for link configuration, and the described environment could serve as a reusable testbed. The authors make a credible effort to define a meaningful action and observation space, to use multiple seeds, and to report standard deviations. However, the central comparative claim is currently undermined by an asymmetric evaluation budget: SA is allowed a much larger per-instance search effort than the trained PPO policy, and the aggregation of rewards during inference is not fully specified. The potential of RL for this problem is nevertheless acknowledged in the conclusions, which is appropriate.

major comments (4)

[§6.1 and §5.5] The evaluation budget is strongly asymmetric. Section 6.1 states that SA reaches 0.988 and that 'only 20,000 steps would have been sufficient' after runs of 2,000,000 steps, while Section 5.5 fixes the PPO episode length to 10 steps in Experiment 1 and 100 steps in Experiment 2. SA can therefore evaluate and retain the best of tens of thousands (or millions) of candidate configurations per observation, whereas the PPO agent is limited to a single fixed-horizon rollout with no explicit best-so-far selection. The observed gaps (0.988 vs 0.875 and 0.988 vs 0.786) may largely reflect this difference in search effort rather than the quality of the learned policy. The authors should re-evaluate both methods under matched per-instance budgets: either give PPO a comparable number of sampled actions per observation (with best-so-far selection) or restrict SA to the same number of steps as the PPO episode, and report both settings.
[§5.4 and §5.5] The evaluation metric is underspecified. It is unclear whether the reported 'Reward' values in Tables 3 and 4 are the cumulative episode return (sum of rewards over the 10 or 100 steps) for PPO, the reward of the final configuration, or the best reward found during the episode. For SA, the text and Figure 2 suggest the reward of the current best configuration at each step. If the PPO number is an episode return while the SA number is a per-step or best-found reward, the two quantities are not comparable. The authors should state precisely how the reward is aggregated for each algorithm during inference and ensure the same aggregation is used for all methods.
[Table 2 and §6.1] There is an internal inconsistency in the reported SA budget. Table 2 lists 'Max Step Integer 2000', but Section 6.1 says SA 'converged toward 1 within the first 20,000 steps' and that 'only 20,000 steps would have been sufficient'. The paper also reports running SA for 2,000,000 steps in Figures 2 and 4. The authors should clarify the actual number of SA iterations per observation, whether the 2,000,000-step runs are single long schedules or multiple restarts, and how the Max Step parameter relates to the convergence statement.
[§7 and §5.4] The conclusion that 'Simulated Annealing delivers better results for this static problem' is stated as a general finding, but the experiments use only a single RL algorithm (PPO), a single environment instantiation with three links and fixed data rates, and a reward function designed by the authors with unstated metric weights (the parameters ω, θ, φ, µ, β, ε, ψ, ρ are introduced in Section 5.4 but their values are never given). The comparative claim should therefore be framed as specific to this environment and reward design, and the authors should provide the exact weight values used in the reported runs to make the results reproducible and interpretable.

minor comments (6)

[Table 2] The row 'Tries Integer 0.99' is inconsistent: 0.99 is not an integer, and the meaning of 'Tries' is not explained in Section 5.3. Please correct the type and define the parameter.
[§5.1] The paper lists 'Ray RLlib (version 0.0.1)', which does not correspond to any known RLlib release. Please verify and correct the version number.
[Tables 3 and 4] The table headers contain the German word 'Algorithmus'; these should be replaced with 'Algorithm' for consistency with the rest of the manuscript.
[Figures 2–5] The figures are referenced in the text but the captions do not describe all axis labels or legend entries; please ensure each figure is self-contained and legible.
[References] There are minor typographical issues in the references, such as 'V oight' in the Vázquez et al. entry and the inconsistent use of accented characters; a careful proofreading pass is recommended.
[§5.4] The metric definitions in Equations (5)–(14) use symbols such as ω, θ, φ, µ, β, ε, ψ, ρ, and RpL and TR, but the numerical values of the weights and the link-reward share are not stated. Please specify these values or state that they were all set to 1.0.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the PPO-vs-SA comparison is an empirical benchmark on a custom reward, not a derivation from fitted constants.

full rationale

The paper's central claim is that Simulated Annealing achieves higher reward than PPO on a custom satellite-transponder link-configuration environment. This is an experimental comparison, not a derivation: the reward function is explicitly defined in Section 5.4, and no parameter is fitted from a subset of the reported results and then renamed as a prediction. The PPO training and inference protocols are described independently of the SA evaluation, and the reported numbers follow from running the algorithms on the same environment. The only self-citations (e.g., Kölle et al. 2024) appear in the related-work motivation and are not load-bearing for the experimental outcome. A potential fairness concern exists because SA is allowed far more search steps per instance than PPO, but that is a benchmarking/validity issue, not a circularity of the derivation chain. The paper does not reduce any claimed result to its own input by construction, and no uniqueness theorem or ansatz is smuggled in via self-citation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on a custom reward function with unreported weights, a heavily simplified environment, and an MDP framing. No new physical or theoretical entities are introduced. The free parameters are the reward weights and the LR/TR split; the axioms are domain assumptions about the adequacy of the reward and environment.

free parameters (3)

Reward weight for Link Reward share (LR) = 0.7
Section 5.4 sets total reward as 70% link reward and 30% transponder reward; this split is chosen by hand and not justified or varied.
Reward weight for Transponder Reward share (TR) = 0.3
Section 5.4 sets the transponder share to 30%; the split is arbitrary and not varied.
Per-metric weights omega, theta, phi, mu, beta, epsilon, psi, rho = Not reported
Section 5.4 defines eight per-metric weights but never states their values. The reward depends on them, and the authors note their impact was not explored. The missing values make the objective function underspecified.

assumptions (4)

domain assumption The hand-designed reward function is an adequate proxy for real transponder link configuration quality.
Section 5.4 defines the eight reward metrics; the entire comparison and conclusions are evaluated against this reward, with no validation against real planning objectives or expert ground truth.
domain assumption The simplified environment (three links, uniform data rate, three MOD-FEC combinations, no electromagnetic interference) represents the target problem well enough for the comparative conclusion to generalize.
Section 4.4 lists simplifications; the authors acknowledge future work is needed to remove them. The conclusion that SA beats PPO may not hold under realistic complexity, and the paper itself speculates the gap may widen (Section 7).
domain assumption The problem can be modeled as a finite-horizon MDP with the given action and observation spaces.
Section 4.3 defines action and observation spaces; the Markov property is assumed in Section 2.1. The static nature means the state is just the current configuration, so an MDP framing is a reasonable but unexamined choice.
domain assumption The bandwidth formula (Equations 2-4) with fixed OH, RS, overhead, spacing, and rolloff factors is correct for the modeled transponder.
Section 4.2 states fixed values were assigned for completeness; these are not varied or validated against real transponder specifications.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimization of Link Configuration for Satellite Communication Using Reinforcement Learning." pith.science (2026). https://pith.science/paper/B5KI4GGU

@misc{pith2026250108220,
  author       = {Pith},
  title        = {Pith review of: Optimization of Link Configuration for Satellite Communication Using Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5KI4GGU}},
  note         = {Machine review of arXiv:2501.08220}
}

read the original abstract

Satellite communication is a key technology in our modern connected world. With increasingly complex hardware, one challenge is to efficiently configure links (connections) on a satellite transponder. Planning an optimal link configuration is extremely complex and depends on many parameters and metrics. The optimal use of the limited resources, bandwidth and power of the transponder is crucial. Such an optimization problem can be approximated using metaheuristic methods such as simulated annealing, but recent research results also show that reinforcement learning can achieve comparable or even better performance in optimization methods. However, there have not yet been any studies on link configuration on satellite transponders. In order to close this research gap, a transponder environment was developed as part of this work. For this environment, the performance of the reinforcement learning algorithm PPO was compared with the metaheuristic simulated annealing in two experiments. The results show that Simulated Annealing delivers better results for this static problem than the PPO algorithm, however, the research in turn also underlines the potential of reinforcement learning for optimization problems.

Figures

Figures reproduced from arXiv: 2501.08220 by the authors.

**Figure 1.** Illustration of Satellite Communication However, each transponder has limited resources in terms of bandwidth and power. It is crucial that the links are configured to consume as few resources as possible. A link is characterized by several parameters, such as data rate, MOD-FEC combination (Modulation and Forward Error Correction), bandwidth, and EIRP (Effective Isotropic Radiated Power). To ensure the data is tr… view at source ↗

**Figure 3.** Development of Rewards under the PPO algo [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 5.** Development of Rewards under the PPO algo [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison PPO, Simulated Annealing and Ran [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

Ardon, L. (2022). Reinforcement learning to solve np-hard problems: an application to the cvrp. arXiv preprint arXiv:2201.05393

arXiv 2022
[3]

V., Norouzi, M., and Bengio, S

Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. (2016). Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940

arXiv 2016
[4]

Deliu, N. (2023). Reinforcement learning for sequential decision making in population research. Quality & Quantity , pages 1--24

work page 2023
[5]

Klar, M., Glatt, M., and Aurich, J. C. (2023). Performance comparison of reinforcement learning and metaheuristics for factory layout planning. CIRP Journal of Manufacturing Science and Technology , 45:10--25

work page 2023
[6]

K \"o lle, M., Witter, T., Rohe, T., Stenzel, G., Altmann, P., and Gabor, T. (2024). A study on optimization techniques for variational quantum circuits in reinforcement learning. arXiv preprint arXiv:2405.12354

arXiv 2024
[7]

Li, K., Zhang, T., Wang, R., Wang, Y., Han, Y., and Wang, L. (2021). Deep reinforcement learning for combinatorial optimization: Covering salesman problems. IEEE transactions on cybernetics , 52(12):13142--13155

work page 2021
[8]

Mazyavkina, N., Sviridov, S., Ivanov, S., and Burnaev, E. (2021). Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400

2021

Show all 12 references

[9]

F., Maximo, M

Prudencio, R. F., Maximo, M. R., and Colombini, E. L. (2023). A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems

2023
[10]

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140--1144

2018
[11]

\'A ., Henarejos, P., P \'e rez-Neira, A

V \'a zquez, M. \'A ., Henarejos, P., P \'e rez-Neira, A. I., Grechi, E., Voight, A., Gil, J. C., Pappalardo, I., Di Credico, F., and Lancellotti, R. M. (2020). On the use of ai for satellite communications. arXiv preprint arXiv:2007.10110

2020 arXiv
[12]

Zhang, T., Banitalebi-Dehkordi, A., and Zhang, Y. (2022). Deep reinforcement learning for exact combinatorial optimization: Learning to branch. In 2022 26th International Conference on Pattern Recognition (ICPR) , pages 3105--3111. IEEE

2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

Ardon, L. (2022). Reinforcement learning to solve np-hard problems: an application to the cvrp. arXiv preprint arXiv:2201.05393

arXiv 2022

[3] [3]

V., Norouzi, M., and Bengio, S

Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. (2016). Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940

arXiv 2016

[4] [4]

Deliu, N. (2023). Reinforcement learning for sequential decision making in population research. Quality & Quantity , pages 1--24

work page 2023

[5] [5]

Klar, M., Glatt, M., and Aurich, J. C. (2023). Performance comparison of reinforcement learning and metaheuristics for factory layout planning. CIRP Journal of Manufacturing Science and Technology , 45:10--25

work page 2023

[6] [6]

K \"o lle, M., Witter, T., Rohe, T., Stenzel, G., Altmann, P., and Gabor, T. (2024). A study on optimization techniques for variational quantum circuits in reinforcement learning. arXiv preprint arXiv:2405.12354

arXiv 2024

[7] [7]

Li, K., Zhang, T., Wang, R., Wang, Y., Han, Y., and Wang, L. (2021). Deep reinforcement learning for combinatorial optimization: Covering salesman problems. IEEE transactions on cybernetics , 52(12):13142--13155

work page 2021

[8] [8]

Mazyavkina, N., Sviridov, S., Ivanov, S., and Burnaev, E. (2021). Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400

2021