Shesha: Opportunistic In-network Acceleration of Asynchronous Distributed Reinforcement Learning

Amr Rizk; Anam Tahir; Firas Khamis; Michael Zink; Mina Tahmasbi Arashloo; Nehal Baganal Krishna

REVIEW 4 major objections 5 minor 61 references

Shesha: Opportunistic In-network Acceleration of Asynchronous Distributed Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In-switch merging of RL updates cuts staleness and speeds training

desk verdict The opportunistic in-flight aggregation idea is genuinely new and the prototype is real, but the headline convergence claim is confounded by a nonstandard reward-gated parameter server and an unproven convergence-preservation heuristic. read the letter →

arxiv 2507.05876 v3 pith:GSJJVMJV submitted 2025-07-08 cs.NI cs.AR

Nehal Baganal Krishna , Anam Tahir , Firas Khamis , Mina Tahmasbi Arashloo , Michael Zink , Amr Rizk This is my paper · ORCID

classification cs.NIcs.AR

keywords asynchronousdistributedreinforcementlearningin-networkaggregationprogrammabledataplanemodelstalenessAge-of-ModelqueuedesignPPOFPGA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a network switch can accelerate asynchronous distributed reinforcement learning by processing model updates while they are queued, rather than merely forwarding or dropping them under congestion. The proposed accelerator, called Shesha in the paper title and Olaf in the full text, opportunistically merges compatible updates from the same cluster, replaces an older unaggregated update from the same worker with a fresher one, and filters low-reward updates, so that at most one update per cluster occupies the queue. Together with worker-side transmission control driven by switch feedback, this reduces update staleness, measured by a new Age-of-Model (AoM) metric, and cuts queue congestion and packet loss. On an FPGA-based testbed and in network simulations, the paper reports roughly 69–78% lower average AoM than a FIFO queue, about 82–86.5% higher final average reward in distributed PPO, and near-uniform AoM fairness across competing clusters. If the convergence-preserving aggregation premise holds, the approach would let multi-tenant clusters share bottleneck links without sacrificing model freshness.

What carries the argument

The load-bearing mechanism is the OlafQueue, an alternative switch queue that treats each fixed memory segment as one model update and keeps per-cluster metadata (cluster_status and replace_status) to locate and rewrite updates already in the queue. Its enqueue logic implements three operations: aggregate compatible updates from the same cluster, replace a same-worker update that has not yet been aggregated, and drop low-reward updates. Supporting this is the Age-of-Model (AoM) metric, a sawtooth staleness process tracking how much time has passed at the parameter server (the central aggregator) since the latest model update, and a formal Satisfiability Modulo Theories (SMT) model that verifies whether worker transmission parameters meet system-wide AoM objectives such as per-cluster fairness.

What would settle it

A direct test is to run the same queue with the reward-filtering rule disabled while keeping aggregation and replacement active: if convergence on a standard RL task degrades, the filter is doing the work; if it does not degrade, the drop rule is unnecessary. A sharper test varies the reward-difference threshold across runs under identical network load and checks whether final reward changes monotonically with the threshold.

Watch

Extended reading notes

Core claim

The central discovery is that staleness in asynchronous DRL can be attacked inside the queue itself. When a model update arrives at the accelerator, the queue checks whether that cluster already has an update waiting: if so, the two are merged into one aggregated update; if the waiting update is an unaggregated older update from the same worker, the newer update replaces it in place because it subsumes the older experience; an incoming update with a much lower reward than the waiting one is dropped. The queue therefore holds at most one model update per cluster, so congestion only appears when the number of active clusters exceeds the queue's memory slots. The paper argues and measures that this preserves update utility while reducing traffic, losses, and AoM, and demonstrates the mechanism on an FPGA-based implementation of a P4/Verilog pipeline. Reported results include 69–78% lower average AoM than FIFO under output bottlenecks, roughly 82–86.5% higher final average reward in distributed PPO (Proximal Policy Optimization) on the LunarLander-v3 task, and near-unity AoM fairness across cluster groups when worker-side transmission control is enabled.

Load-bearing premise

Everything rests on the assumption that merging compatible updates, replacing a same-worker update with a fresher one, and dropping low-reward updates inside the switch preserves or improves asynchronous PPO convergence, which the paper asserts from empirical observation rather than proof.

Editorial extensions

If this is right

When the number of active clusters is below the queue size, workers can transmit at full rate without congestion because each cluster contributes at most one queued update, giving an explicit congestion-free operating condition.
Because aggregation reduces the number of packets that reach the parameter server, downlink response traffic also shrinks, since ACKs are sent per cluster rather than per worker.
The static verifier can act as admission control: a new cluster is admitted only if its transmission parameters keep the per-cluster average peak AoM within $\epsilon$ of the others, which the paper verifies for both uniform and non-uniform update generation rates.
Under asymmetric update frequencies, worker-side transmission control brings the average AoM of fast and slow cluster groups closer together, reaching a fairness index of 0.99, so the benefit extends beyond homogeneous settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond DRL, the same replace-in-queue rule could apply to any stream of subsuming updates, such as cached video segments where a newer segment makes an older queued one obsolete; the paper gestures at this direction but does not test it.
The reward-gated drop rule is the least theoretically grounded piece, so a useful extension is to vary the reward-difference threshold and check whether convergence remains monotone, since an over-aggressive filter could discard exploratory updates that later enable high reward.
AoM is used as a proxy for convergence time, but the mapping is not proven; a direct check would inject controlled delays into single-cluster asynchronous PPO runs and fit the AoM-to-reward curve empirically.
The single-packet model-update constraint limits model size, so supporting fragmentation with partial aggregation would extend the design to larger policy networks at the cost of reassembly latency.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The queue-based opportunistic aggregation/replacement mechanism is the real contribution here. Unlike SwitchML, ATP, or iSW, which aggregate in switch registers under synchronous or periodic rounds, Olaf merges compatible updates on the fly in the queue itself. That is a different design point, and it is plausibly useful for async DRL. The FPGA prototype is legit: small hardware footprint, 50 Gbps queue logic, and the ns-3 multi-hop results show large, consistent AoM reductions versus FIFO. The formal AoM model and SMT fairness check are a nice complement, though they verify staleness fairness, not convergence.

Now the soft spots, in proportion. The convergence claim is not established. The evaluation uses a parameter server that accepts an update only if its episodic reward exceeds the stored global reward (ri > rg) and otherwise discards it. That is not the standard async PPO update rule used in the cited literature. Under this reward-gated PS, the switch's low-reward dropping is almost guaranteed to help the reported reward curves, because 'bad' updates are removed before they can perturb the PS. The Figure 8 comparison therefore cannot separate the benefit of staleness reduction from the benefit of a filter aligned with the PS's own gate. The paper calls the aggregation 'convergence-preserving' and says it is 'empirically observed,' but gives no proof, no reward threshold, and no sensitivity analysis. That is the load-bearing assumption, and it is currently unsupported.

Other issues are minor by comparison: the AoM metric is a direct adaptation of AoI and improves partly by construction, the evaluation uses a single RL environment with hand-picked parameters, there is no comparison against prior in-network aggregation systems, the title says Shesha while the text says Olaf, and no artifacts are released. The paper does acknowledge some limitations, notably the no-fragmentation constraint, which restricts practical model sizes.

For all that, the core systems idea deserves serious referee time. The right fix is a new evaluation with standard PPO as the PS rule, plus a sensitivity sweep over the aggregation/replacement threshold and a direct comparison with iSW or SwitchML-style aggregation under congestion. If the reward improvement survives those baselines, this is a solid subfield-level result. If not, the AoM reduction still stands as a queueing contribution, just not a convergence one.

Referee Report

4 major / 5 minor

Summary. The paper proposes Olaf, a programmable data-plane accelerator for asynchronous distributed reinforcement learning (DRL). In the proposed design, an in-network queue opportunistically aggregates, replaces, or drops model updates as they traverse a top-of-rack switch; a worker-side transmission-control algorithm modulates update sending based on reverse-path queue feedback; and an SMT-based verifier checks system-wide Age-of-Model (AoM) fairness objectives. The evaluation combines an FPGA prototype with ns-3 simulations and reports that, compared with a conventional FIFO queue, Olaf reduces update loss and average AoM, achieves speedup to a fixed reward target, and improves reward curves in a distributed PPO setup.

Significance. If the convergence claim is established, this is a valuable systems contribution: it demonstrates a working hardware prototype of in-network aggregation for asynchronous DRL, introduces AoM as a staleness metric, and provides a formal verification framework for multi-tenant AoM fairness. The paper is strongest on the networking side: the queue design, FPGA resource numbers, and ns-3 multi-hop experiments are concrete and reproducible in spirit. However, the headline claim that the architecture 'improves the convergence rate' is not yet supported by the evaluation as presented, because the reward-based update filtering is confounded with a nonstandard parameter-server update rule and is not evaluated for sensitivity. The central networking result—lower AoM and lower loss under congestion—is defensible; the convergence benefit needs a cleaner experimental isolation.

major comments (4)

[§3, §4 (Algorithm 1), §8.2] The convergence-preserving property of Olaf's reward-based filtering is asserted but not established. Section 3 states that updates with 'comparable' rewards are aggregated, higher-reward updates replace lower ones, and significantly lower-reward updates are dropped, with 'a predefined threshold' that is never specified. Algorithm 1 in §4 does not include any reward comparison at all, so the actual data-plane decision logic for aggregation/dropping is underspecified. No proof, threshold value, or sensitivity analysis is provided. Because this filtering changes the update stream seen by the parameter server, it is load-bearing for the convergence claim in Fig. 8 and must be specified and tested.
[§2.1, §8.2, Fig. 8] The reward improvement attributed to Olaf is confounded with the nonstandard PS update rule. In §2.1, the PS applies an update only when the worker's episodic reward exceeds the stored global reward (ri > rg) and otherwise discards the update. Under this rule, Olaf's switch-level dropping of low-reward updates is aligned with the PS's own acceptance criterion, so Fig. 8 may reflect the benefit of filtering rather than the benefit of staleness reduction through aggregation/replacement. The paper does not compare against a standard async PPO baseline (e.g., always applying the gradient, or with the reward gate disabled), which is necessary to separate the effects.
[§8, Fig. 7, Fig. 8] The convergence evaluation is too narrow to support the generality of the claim. Only one RL environment (LunarLander-v3) and one replayed trace are used, and there is no sensitivity analysis for the key parameters: the reward threshold, the transmission-control slope v, the feedback staleness threshold Δ̄T, the fairness tolerance ε, or the reward target N. In addition, the speedup metric in Fig. 7 defines N=200 as the number of updates needed to reach a reward under ideal async training (from Fig. 3), but under Olaf the PS receives aggregated updates, so the meaning of 'updates from each worker' is not directly comparable to FIFO. The paper should report the actual reward target used, clarify how aggregated updates are counted, and provide a sensitivity sweep.
[§3, §4 (Algorithm 1), §12.1] There is an internal inconsistency about the number of updates per cluster allowed in the queue. Section 3 and Algorithm 1 guarantee 'at most one model update per cluster is in the queue at any point of time,' but §12.1 states that the OlafQueue 'can hold and distinguish up to two active updates per cluster' to handle corner cases at the queue head. This distinction matters because the AoM model in §6 and the congestion analysis in §5 rely on the one-update-per-cluster invariant. The invariant should be stated precisely, and the AoM model should be reconciled with the two-update case.

minor comments (5)

[Title/metadata] The submission metadata in the review system lists the title as 'Shesha', while the manuscript text uses 'Olaf' throughout; please reconcile the title and system name.
[§6] The statement 'We fix ε = 0.1 without loss of generality' is inaccurate: ε is a fairness tolerance that should be swept to show robustness of the verification result.
[Algorithm 1, Fig. 4] The aggregation function f(M1, M2) is shown in Fig. 4 but never formally defined in the text or pseudocode; please provide the exact formula used to combine gradients and rewards.
[Table 1] In Table 1, the 'updates aggregated' column for Olaf exceeds the 'updates received@PS' count; clarify whether the aggregated count represents the number of constituent updates folded into the delivered aggregates.
[§2.1] The word 'stagglers' appears in the text; it should be 'stragglers'.

Circularity Check

1 steps flagged · score 2.0 of 10

AoM staleness reduction is partly definitional for replacement and aggregation, but the headline convergence result is measured independently against FIFO; overall no significant circularity.

self definitional [Section 6, Figure 5 caption; Section 4, Dequeue Process; Section 8.1]
"Aggregated or replaced models are less stale and, hence, minimize the AoM."

The AoM is defined as the time that has passed at the PS since the reception of the latest model update. The queue's replacement rule is defined so that a newer update from the same worker overwrites the older queued update and inherits its departure slot, making the delivered update newer by construction. Consequently, the peak-AoM formula Delta_p(k) = (D(k)-A(l)) * 1{D(k)<A(k+1)} records a later A(l) after replacement, so the reported 69-78% AoM reduction in Section 8.1 is partly a restatement of the queue definition rather than an independent empirical finding. However, the central convergence claim is evaluated separately via reward curves in Figure 8 against FIFO, so the main result does not reduce to this definition.

full rationale

The paper's central claim is that opportunistic in-network aggregation, replacement, and low-reward filtering of asynchronous DRL updates reduces staleness and congestion and improves convergence. The AoM reduction is partly definitional: replacing an older update with a newer one, and labeling the aggregated result as the delivered update, lowers the age of the update received at the PS by construction. This is the intended mechanism of the queue, not a hidden circularity in the main convergence result. The headline convergence comparison in Figure 8 measures average worker reward over iterations for Olaf versus FIFO under the same reward-gated parameter-server rule, and is therefore an independent, externally falsifiable comparison. The formal AoM verifier in Section 6 is a self-contained SMT constraint check over arrival and departure times; it does not assume the convergence improvement and is not fitted to the evaluation data. The self-citations, RPM [6] and CCAC [5], are used only as inspiration for reverse-path signaling and formal verification, respectively, and are not load-bearing. A substantive weakness is that the convergence-preserving claim for the low-reward drop (Section 3) is asserted as 'empirically observed' without specifying the reward threshold or providing a proof or sensitivity analysis, and the PS itself discards updates with ri <= rg, so Figure 8 cannot separate the benefit of the filter from the benefits of reduced congestion and fresher updates. That is a validity threat and a correctness risk, but it is a confound, not a circular derivation. No fitted parameter is later renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Overall, the only definitional component is the AoM freshness metric, which is a minor issue and does not undermine the independent convergence evaluation.

Assumptions & free parameters 5 free parameters · 7 assumptions · 1 invented entities

The central design rests on several unproven domain assumptions about RL gradient aggregation and on hand-chosen parameters for reward comparison, transmission backoff, and fairness tolerance. No code or data artifacts are provided. The only invented abstraction is the AoM metric.

free parameters (5)

reward aggregation/replacement threshold = not specified
Olaf aggregates updates when rewards are within a predefined threshold, replaces when the incoming reward exceeds it, and drops when rewards are significantly lower. The threshold value is not stated, and system behavior depends on it. See §3.
transmission probability slope v = not specified (suggested v=1/Δ̄T or v=Δ̄T)
In Ps = min(Qmax/N + f(Δ̂_i,n),1), the slope v controls how aggressively workers with stale feedback increase transmission probability. The paper gives two possible settings but no chosen value or sensitivity analysis. See §5.
feedback staleness threshold Δ̄T = 400 msec in the SMT example
Workers treat queue feedback as obsolete after Δ̄T. This threshold shapes transmission control and AoM fairness, and the SMT verification uses 400 msec without exploring alternatives. See §5, §6.
AoM fairness tolerance ε = 0.1
The SMT fairness query requires the difference in average peak AoM between any two clusters to be at most ε. The value is fixed 'without loss of generality,' but it directly defines what counts as fair. See §6.
reward target N for speedup = 200 updates
Speedup is defined as the ratio of time to receive N model updates from each worker under FIFO versus Olaf, with N chosen from the authors' own DRL experiment as the number of updates needed for an acceptable reward. See §8.2.

assumptions (7)

domain assumption Averaging compatible gradient updates in the network queue preserves or improves convergence in asynchronous PPO training.
The paper labels this 'convergence-preserving aggregation' in §3 and supports it only with empirical observation, not with a convergence proof.
domain assumption A newer update from the same worker subsumes the older update because it has collected more experience.
Used to justify replacement in §2.2 and §4. The subsumption property is assumed, not proven for general RL loss landscapes.
domain assumption Dropping updates with significantly lower reward is safe and stabilizes training.
The queue drops low-reward updates in §3, described as empirically observed to improve stability. No theoretical or statistical justification is given.
domain assumption AoM is a valid proxy for convergence time in asynchronous DRL.
The paper states in §2.2 that average AoM 'can be related' to convergence time, but no formal relationship is derived.
domain assumption Forward and reverse network paths are symmetric.
The ACK-based queue feedback in §5 assumes ACKs traverse the same engine as updates, noted as 'usual if deployed in a ToR switch.'
domain assumption The entire model update fits in one packet or jumbo frame.
§4 sets memory segment size to store one update, and §10 acknowledges fragmentation is unsupported, restricting model sizes.
domain assumption In the AoM model, the PS ACKs immediately and transmission delays are constant.
§6 simplifies the formal model by assuming immediate ACKs and constant delay R, which may not hold under queueing variation.

invented entities (1)

Age-of-Model (AoM)
purpose: Proxy metric for staleness of model updates at the parameter server and objective for fairness verification across clusters.
AoM is a constructed sawtooth age process analogous to Age-of-Information. It has no independent falsifiable handle outside the paper, so improvements in AoM are not by themselves evidence of convergence gains.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Shesha: Opportunistic In-network Acceleration of Asynchronous Distributed Reinforcement Learning." pith.science (2026). https://pith.science/paper/GSJJVMJV

@misc{pith2026250705876,
  author       = {Pith},
  title        = {Pith review of: Shesha: Opportunistic In-network Acceleration of Asynchronous Distributed Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSJJVMJV}},
  note         = {Machine review of arXiv:2507.05876}
}

read the original abstract

Large-scale training for distributed Machine Learning can cause congestion at bottleneck switch ports, leading to model staleness through update losses. This is particularly detrimental for asynchronous Distributed Reinforcement Learning (DRL) training, as stale updates are known to degrade convergence performance in asynchronous settings. This paper presents \textit{Shesha}, an in-network DRL accelerator engine, which \textit{opportunistically aggregates} asynchronously generated model updates \textit{on the fly} while they traverse the data plane queue. This aggregation operation motivates an alternative queue design, which we prototype and envision for future Top-of-Rack switches. We further present corresponding host-side transmission control in the face of possible congestion, taking advantage of in-network accelerator feedback. A quantification of model staleness, denoted Age-of-Model (AoM), together with a formal verifier allows us to reason on system-wide AoM objectives in multi DRL-cluster scenarios. Shesha shows significant reductions in model staleness and queue congestion, improving overall convergence behavior for asynchronous DRL workloads.

Figures

Figures reproduced from arXiv: 2507.05876 by the authors.

**Figure 2.** Average worker reward over training itera [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Olaf’s In-network engine. exceeds the queue memory size. Depending on the update send rate clusters can then influence their share of the bottleneck leading to disparities between clusters in update delivery rates at the PS and hence disparities between cluster AoM. Here, Olaf provides a formal model of the AoM that takes the parameters of the transmission control functions to verify whether AoM objectives over mul… view at source ↗

Figures from the paper (6 more)

**Figure 5.** Figure 5: Graphical example of the Age-of-Model: The AoM [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Time-to-Reward: Olaf’s speedup over FIFO for [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Average per-worker reward achieved by the worker’s [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

**Figure 9.** Figure 9: Multi-hop topology with 10 clusters Ci connected via switches SWi to the parameter server (PS). Queue config. update loss (%) AoM C1-C5 (msec) AoM C6-C10 (msec) AoM Fairness FIFO 88 1714 1710 0.88 Olaf 4.5 245 244 0.98 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: Average AoM for cluster groups S1 (C1 −C5) and S2 (C6 −C10) under FIFO and Olaf, for increasing topology heterogeneity. We vary the SW1 outgoing link capacity x1 while keeping the capacity x2 for SW2 fixed with α = x1/x2. Olaf reduces the discrepancy of the AoM remo…

**Figure 11.** Figure 11: Per-cluster update tracking: Each segment is made [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 50 canonical work pages

[1]

Backpropagation and stochastic gradi- ent descent method

Shun-ichi Amari. Backpropagation and stochastic gradi- ent descent method. Neurocomputing, 5(4-5):185–196, 1993

work page 1993
[2]

AMD Alveo U55C Data Center Accelera- tor Card

AMD. AMD Alveo U55C Data Center Accelera- tor Card. https://www.amd.com/en/products/ accelerators/alveo/u55c/a-u55c-p00g-pq-g. html. Accessed: 2025-04-04

work page 2025
[3]

VitisNetP4 IP for Adaptive SoCs and FPGAs

AMD. VitisNetP4 IP for Adaptive SoCs and FPGAs. https://www.amd.com/en/ products/adaptive-socs-and-fpgas/ intellectual-property/ef-di-vitisnetp4. html. Accessed: 2025-04-04

work page 2025
[4]

AMBA AXI4-Stream Protocol Specification

ARM Ltd. AMBA AXI4-Stream Protocol Specification. https://developer.arm.com/documentation/ ihi0051, 2010. Accessed: 2025-04-04

work page 2010
[5]

Toward formally verifying congestion control behavior

Venkat Arun, Mina Tahmasbi Arashloo, Ahmed Saeed, Mohammad Alizadeh, and Hari Balakrishnan. Toward formally verifying congestion control behavior. In Pro- ceedings of the ACM SIGCOMM Conference , pages 1–16, 2021

work page 2021
[6]

RPM: Reverse Path Congestion Marking on P4 Programmable Switches

Nehal Baganal-Krishna, Tuan-Dat Tran, Ralf Kundel, and Amr Rizk. RPM: Reverse Path Congestion Marking on P4 Programmable Switches. In Proceedings of the IEEE 48th Conference on Local Computer Networks (LCN), pages 1–4, 2023

work page 2023
[7]

P4: Programming protocol-independent packet processors

Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, et al. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review , 44(3):87–95, 2014

2014
[8]

Trading latency for compute in the net- work

Pietro Bressana, Noa Zilberman, Dejan Vucinic, and Robert Soulé. Trading latency for compute in the net- work. In Proceedings of the Workshop on Network Application Integration/CoDesign, pages 35–40, 2020

work page 2020

Show all 61 references

[9]

Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training

Zixuan Chen, Xuandong Liu, Minglin Li, Yinfan Hu, Hao Mei, Huifeng Xing, Hao Wang, Wanxin Shi, Sen Liu, and Yang Xu. Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training. In Proceedings of the IEEE 32nd Interna- tional Conference on Network ...

2024
[10]

Boosting distributed machine learning training through loss-tolerant transmission protocol

Zixuan Chen, Lei Shi, Xuandong Liu, Xin Ai, Sen Liu, and Yang Xu. Boosting distributed machine learning training through loss-tolerant transmission protocol. In Proceedings of the IEEE/ACM 31st International Sympo- sium on Quality of Service (IWQoS), pages 1–10, 2023

2023
[11]

Scaling multi- agent reinforcement learning with selective parameter sharing

Filippos Christianos, Georgios Papoudakis, Muham- mad A Rahman, and Stefano V Albrecht. Scaling multi- agent reinforcement learning with selective parameter sharing. In Proceedings of the International Conference on Machine Learning, pages 1989–1998, 2021

1989
[12]

Parameter shar- ing deep deterministic policy gradient for cooperative multi-agent reinforcement learning

Xiangxiang Chu and Hangjun Ye. Parameter shar- ing deep deterministic policy gradient for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:1710.00336, 2017

2017 arXiv
[13]

Cisco N9300 Series Smart Switches

Cisco Systems, Inc. Cisco N9300 Series Smart Switches. https://www.cisco.com/c/de_de/support/ switches/9300-series-smart-switches/series. html. Accessed: 2025-04-04

2025
[14]

Accelerating neural network training with distributed asynchronous and selective optimiza- tion (DASO)

Daniel Coquelin, Charlotte Debus, Markus Götz, Fab- rice von der Lehr, James Kahn, Martin Siggel, and Achim Streit. Accelerating neural network training with distributed asynchronous and selective optimiza- tion (DASO). Journal of Big Data, 9, 2022

2022
[15]

Toward understanding the impact of stal- eness in distributed machine learning

Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, and Eric P Xing. Toward understanding the impact of stal- eness in distributed machine learning. arXiv preprint arXiv:1810.03264, 2018

2018 arXiv
[16]

Distributed deep learning using synchronous stochastic gradient descent

Dipankar Das, Sasikanth Avancha, Dheevatsa Mudigere, Karthikeyan Vaidynathan, Srinivas Sridharan, Dhiraj Kalamkar, Bharat Kaul, and Pradeep Dubey. Distributed deep learning using synchronous stochastic gradient descent. arXiv preprint arXiv:1602.06709, 2016

2016 arXiv
[17]

Z3: An ef- ficient smt solver

Leonardo De Moura and Nikolaj Bjørner. Z3: An ef- ficient smt solver. In Proceedings of the International conference on Tools and Algorithms for the Construc- tion and Analysis of Systems, pages 337–340, 2008

2008
[18]

Large scale distributed deep networks

Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, An- drew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural informa- tion processing systems, 25, 2012. 13

2012
[19]

Distributed proxi- mal policy optimization for contention-based spectrum access

Akash Doshi and Jeffrey G Andrews. Distributed proxi- mal policy optimization for contention-based spectrum access. In Proceedings of the 55th Asilomar Conference on Signals, Systems, and Computers , pages 340–344, 2021

2021
[20]

P4-enabled network-assisted congestion feedback: A case for nacks

Anja Feldmann, Balakrishnan Chandrasekaran, Seifed- dine Fathalli, and Emilia N Weyulu. P4-enabled network-assisted congestion feedback: A case for nacks. In Proceedings of Workshop on Buffer Sizing, pages 1–7, 2019

2019
[21]

Scalable hierarchical aggregation protocol (SHArP): A hardware architecture for efficient data reduction

Richard L Graham, Devendar Bureddy, Pak Lui, Hal Rosenstock, Gilad Shainer, Gil Bloch, Dror Goldenerg, Mike Dubman, Sasha Kotchubievsky, Vladimir Koush- nir, et al. Scalable hierarchical aggregation protocol (SHArP): A hardware architecture for efficient data reduction. In Pro...

2016
[22]

Intel ® Tofino Intelligent Fabric Processors

Intel Corporation. Intel ® Tofino Intelligent Fabric Processors. https://www.intel.com/content/ www/us/en/products/details/network-io/ intelligent-fabric-processors/tofino.html. Accessed: 2025-04-04

2025
[23]

Accelerating deep learning us- ing multiple GPUs and FPGA-based 10GbE switch

Tomoya Itsubo, Michihiro Koibuchi, Hideharu Amano, and Hiroki Matsutani. Accelerating deep learning us- ing multiple GPUs and FPGA-based 10GbE switch. In Proceedings of the 28th Euromicro International Con- ference on Parallel, Distributed and Network-Based Pro- cessing (PDP),...

2020
[24]

The art of computer systems performance analysis

Raj Jain. The art of computer systems performance analysis. John Wiley & Sons, 1990

1990
[25]

Reinforcement learning: A survey

Leslie Pack Kaelbling, Michael L Littman, and An- drew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285, 1996

1996
[26]

Parameter sharing reinforcement learning architecture for multi agent driving

Meha Kaushik, Nirvan Singhania, Phaniteja S, and K Madhava Krishna. Parameter sharing reinforcement learning architecture for multi agent driving. InProceed- ings of the 4th International Conference on Advances in Robotics, pages 1–7, 2019

2019
[27]

Rein- forcement learning in robotics: A survey

Jens Kober, J Andrew Bagnell, and Jan Peters. Rein- forcement learning in robotics: A survey. The Interna- tional Journal of Robotics Research, 32(11):1238–1274, 2013

2013
[28]

ATP: In-network aggregation for multi-tenant learning

ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. ATP: In-network aggregation for multi-tenant learning. In Proceedings of the 18th USENIX Symposium on Net- worked Systems Design and Implementation, pages 741– 761, 2021

2021
[29]

Andersen, Jun Woo Park, Alexander J

Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eu- gene J. Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In Pro- ceedings of the 11th USENIX Conference on Operating Systems Desig...

2014
[30]

Communication efficient distributed machine learn- ing with the parameter server

Mu Li, David G Andersen, Alexander Smola, and Kai Yu. Communication efficient distributed machine learn- ing with the parameter server. Advances in neural infor- mation processing systems, 27, 2014

2014
[31]

Parameter server for distributed machine learning

Mu Li, Li Zhou, Zichao Yang, Aaron Li, Fei Xia, David G Andersen, and Alexander Smola. Parameter server for distributed machine learning. In Proceedings of the Big learning NIPS workshop, volume 6, 2013

2013
[32]

Accelerating dis- tributed reinforcement learning with in-switch comput- ing

Youjie Li, Iou-Jen Liu, Yifan Yuan, Deming Chen, Alexander Schwing, and Jian Huang. Accelerating dis- tributed reinforcement learning with in-switch comput- ing. In Proceedings of the 46th International Symposium on Computer Architecture, pages 279–291, 2019

2019
[33]

Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training

Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. Pipe-SGD: A decentralized pipelined SGD framework for distributed deep net training. Advances in Neural Information Pro- cessing Systems, 31, 2018

2018
[34]

Deep reinforcement learning: An overview

Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274, 2017

2017 arXiv
[35]

RLlib: Abstractions for distributed reinforcement learning

Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In Proceedings of the International conference on machine learning , pages 3053–3062, 2018

2018
[36]

Asynchronous local- sgd training for language modeling

Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Marc’Aurelio Ranzato. Asynchronous local- sgd training for language modeling. arXiv preprint arXiv:2401.09135, 2024

2024 arXiv
[37]

High-throughput synchronous deep RL

Iou-Jen Liu, Raymond Yeh, and Alexander Schwing. High-throughput synchronous deep RL. Advances in Neural Information Processing Systems, 33:17070– 17080, 2020

2020
[38]

FPGA-based AI smart NICs for scalable distributed AI training systems

Rui Ma, Evangelos Georganas, Alexander Heinecke, Sergey Gribok, Andrew Boutros, and Eriko Nurvitadhi. FPGA-based AI smart NICs for scalable distributed AI training systems. IEEE Computer Architecture Letters, 21(2):49–52, 2022. 14

2022
[39]

A Survey of DeepSeek Models

Fnu Neha and Deepshikha Bhati. A Survey of DeepSeek Models. Authorea Preprints, 2025

2025
[40]

Federated learning with buffered asynchronous aggregation

John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In Proceedings of the International con- ference on artificial intelligence and statistics , pages 3581–3607, 2022

2022
[41]

ns-3 Network Simulator

ns-3 Project. ns-3 Network Simulator. https://www. nsnam.org/. Accessed: 2025-04-04

2025
[42]

Di- rect preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is secretly a reward model. Advances in Neural Informa- tion Processing Systems, 36:53728–53741, 2023

2023
[43]

Smaller world models for reinforcement learning

Jan Robine, Tobias Uelwer, and Stefan Harmeling. Smaller world models for reinforcement learning. Neu- ral Processing Letters, 55(8):11397–11427, 2023

2023
[44]

An overview of gradient descent opti- mization algorithms

Sebastian Ruder. An overview of gradient descent opti- mization algorithms. arXiv preprint arXiv:1609.04747, 2016

2016 arXiv
[45]

On the convergence analysis of asyn- chronous SGD for solving consistent linear systems

Atal Narayan Sahu, Aritra Dutta, Aashutosh Tiwari, and Peter Richtárik. On the convergence analysis of asyn- chronous SGD for solving consistent linear systems. arXiv preprint arXiv:2004.02163, 2020

2004 arXiv
[46]

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019

1910 arXiv
[47]

Scaling distributed machine learning with In-Network aggregation

Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Kr- ishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. Scaling distributed machine learning with In-Network aggregation. In Proceedings of the 18th USENIX Symposium on Networked...

2021
[48]

Proximal policy optimiza- tion algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[49]

Addressing Stale Gradients in Scalable Federated Deep Reinforcement Learning

Justin Stanley and Ali Jannesari. Addressing Stale Gradients in Scalable Federated Deep Reinforcement Learning. In Proceedings of the Workshops of the Inter- national Conference on High Performance Computing, Network, Storage, and Analysis, pages 932–940, 2023

2023
[50]

Gymnasium: A standard interface for rein- forcement learning environments

Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for rein- forcement learning environments. arXiv preprint arXiv:2407.17032, 2024

2024 arXiv
[51]

A survey on distributed machine learning

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Reller- meyer. A survey on distributed machine learning. ACM Computing Surveys (csur), 53(2):1–33, 2020

2020
[52]

Rat-resilient allreduce tree for distributed machine learning

Xinchen Wan, Hong Zhang, Hao Wang, Shuihai Hu, Junxue Zhang, and Kai Chen. Rat-resilient allreduce tree for distributed machine learning. In Proceedings of the 4th Asia-Pacific Workshop on Networking, pages 52–57, 2020

2020
[53]

Domain-specific communi- cation optimization for distributed DNN training

Hao Wang, Jingrong Chen, Xinchen Wan, Han Tian, Ji- acheng Xia, Gaoxiong Zeng, Weiyan Wang, Kai Chen, Wei Bai, and Junchen Jiang. Domain-specific communi- cation optimization for distributed DNN training. arXiv preprint arXiv:2008.08445, 2020

2008 arXiv
[54]

Pufferfish: Communication-efficient Models At No Extra Cost

Hongyi Wang, Saurabh Agarwal, and Dimitris Papail- iopoulos. Pufferfish: Communication-efficient Models At No Extra Cost. In Proceedings of Machine Learning and Systems, volume 3, pages 365–386, 2021

2021
[55]

DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames

Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357, 2019

1911 arXiv
[56]

Reinforcement learning in autonomous driving

Dantong Xiang. Reinforcement learning in autonomous driving. Applied and Computational Engineering, 48:17– 23, 2024

2024
[57]

Asynchronous actor-critic for multi-agent reinforcement learning

Yuchen Xiao, Weihao Tan, and Christopher Amato. Asynchronous actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:4385–4400, 2022

2022
[58]

Open-NIC Project

Xilinx. Open-NIC Project. https://github.com/ Xilinx/open-nic. Accessed: 2025-04-04

2025
[59]

Using Trio: Juniper Networks’ programmable chipset-for emerging in-network applications

Mingran Yang, Alex Baban, Valery Kugel, Jeff Libby, Scott Mackie, Swamy Sadashivaiah Renu Kananda, Chang-Hong Wu, and Manya Ghobadi. Using Trio: Juniper Networks’ programmable chipset-for emerging in-network applications. In Proceedings of the ACM SIGCOMM Conference, pages 633...

2022
[60]

Age of information: An introduction and survey

Roy D Yates, Yin Sun, D Richard Brown, Sanjit K Kaul, Eytan Modiano, and Sennur Ulukus. Age of information: An introduction and survey. IEEE Journal on Selected Areas in Communications, 39(5):1183–1210, 2021

2021
[61]

Stellaris: Staleness-Aware Distributed 15 Reinforcement Learning with Serverless Computing

Hanfei Yu, Hao Wang, Devesh Tiwari, Jian Li, and Seung-Jong Park. Stellaris: Staleness-Aware Distributed 15 Reinforcement Learning with Serverless Computing. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Anal- ysis, page...

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Backpropagation and stochastic gradi- ent descent method

Shun-ichi Amari. Backpropagation and stochastic gradi- ent descent method. Neurocomputing, 5(4-5):185–196, 1993

work page 1993

[2] [2]

AMD Alveo U55C Data Center Accelera- tor Card

AMD. AMD Alveo U55C Data Center Accelera- tor Card. https://www.amd.com/en/products/ accelerators/alveo/u55c/a-u55c-p00g-pq-g. html. Accessed: 2025-04-04

work page 2025

[3] [3]

VitisNetP4 IP for Adaptive SoCs and FPGAs

AMD. VitisNetP4 IP for Adaptive SoCs and FPGAs. https://www.amd.com/en/ products/adaptive-socs-and-fpgas/ intellectual-property/ef-di-vitisnetp4. html. Accessed: 2025-04-04

work page 2025

[4] [4]

AMBA AXI4-Stream Protocol Specification

ARM Ltd. AMBA AXI4-Stream Protocol Specification. https://developer.arm.com/documentation/ ihi0051, 2010. Accessed: 2025-04-04

work page 2010

[5] [5]

Toward formally verifying congestion control behavior

Venkat Arun, Mina Tahmasbi Arashloo, Ahmed Saeed, Mohammad Alizadeh, and Hari Balakrishnan. Toward formally verifying congestion control behavior. In Pro- ceedings of the ACM SIGCOMM Conference , pages 1–16, 2021

work page 2021

[6] [6]

RPM: Reverse Path Congestion Marking on P4 Programmable Switches

Nehal Baganal-Krishna, Tuan-Dat Tran, Ralf Kundel, and Amr Rizk. RPM: Reverse Path Congestion Marking on P4 Programmable Switches. In Proceedings of the IEEE 48th Conference on Local Computer Networks (LCN), pages 1–4, 2023

work page 2023

[7] [7]

P4: Programming protocol-independent packet processors

Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, et al. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review , 44(3):87–95, 2014

2014

[8] [8]

Trading latency for compute in the net- work

Pietro Bressana, Noa Zilberman, Dejan Vucinic, and Robert Soulé. Trading latency for compute in the net- work. In Proceedings of the Workshop on Network Application Integration/CoDesign, pages 35–40, 2020

work page 2020