By linearity of expectation, Eπ h Gk t |h k i,t i =E π " 1 K KX ℓ=1 Gℓ t+1 |h k i,t # = 1 K KX ℓ=1 Eπ h Gℓ t+1 |h k i,t i , where ℓ denotes the sampling index at t+ 1

For agentiat timet, define gk i,t =ρ k i,t∇θi logπ θi(ak i,t |h i,t)G k t · 2001