Algorithmic collusion with endogenous exploration

I study a two-stage model in which two players simultaneously choose an exploration parameter for their Q-learning algorithms, which then repeatedly play a one-shot game chosen from a class of social dilemmas including the prisoner's dilemma, first and second-price auctions as well as Bertrand competition with horizontally differentiated products. The players collect the limit average payoffs obtained by their algorithms. I show that all equilibria are collusive: both players receive payoffs that are strictly higher than the payoffs received in the unique strict Nash equilibrium of the one-shot game. I then use extensive numerical simulations in a Bertrand duopoly and a parameterized prisoner's dilemma. Their results allow to gain insight on (i) the mechanism causing algorithmic collusion and (ii) the strategic role of exploration levels in the game. They reveal that in equilibrium, the players tend to choose algorithms that \textit{over-explore}, which comes at the detriment of joint payoff. These findings have important implications for algorithmic collusion.