{"id":"1c8a58d0-7347-4675-bcef-4ec64b400367","arxiv_id":"2507.18680","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Simulated RL market makers with dynamic inventory penalties and Pareto-front multi-objective training outperform baseline market makers, and a discounted Thompson sampling policy switcher handles non-stationary markets.","lead":"This PhD thesis tests reinforcement learning agents for market making in a simulated stock market, and it reports that they beat random and rule-based competitors on profit while controlling inventory. It also introduces a Thompson-sampling method for switching between pre-trained strategies when market conditions change.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported RL outperformance is conditional on a zero-fee, narrowest-spread-routing simulation against random and persistent baselines; relaxing those assumptions may erase the advantage.","rationale":"The reader identified the zero-fee and narrowest-spread-routing assumptions as the weakest point, and I agree that these are central. My stress-test sharpens the concern: the full comparison stack (random and persistent baselines, no informed flow, no fees, deterministic spread-based routing) is constructed so that a learned policy can win purely by undercutting the market spread. This means the experimental evidence does not establish the abstract's unqualified claim of outperforming 'traditional and baseline algorithmic strategies'; no traditional strategy is even implemented. The thesis's own limitation statements in Sections 3.5 and 4.1.3 acknowledge the sim-to-real gap, which tempers the claim to a simulated result. A concrete robustness test with probabilistic routing and positive fees would settle whether the reported advantage is specific to the artificial environment or reflects a more general property of the learned policies. The verdict remains CONDITIONAL: the paper should either run this robustness check or explicitly restrict the central claim to the zero-fee, narrowest-routing ABIDES configuration with random/persistent comparators.","tokens_in":47511,"tokens_out":6902,"duration_ms":81715,"concrete_test":"Re-run the Chapter 3 and Chapter 4 experiments with investor routing changed from deterministic narrowest-spread selection to a probabilistic logit choice over quoted spreads (e.g., with temperature tuned to match the observed spread differences) and with a positive per-share transaction fee charged to MM rewards, keeping all other settings fixed. If the DQL-MM/M3ORL average MtM advantage over Random-MM and Persistent-MM is not preserved within the reported margins, the central claim is an artifact of the zero-fee, greedy-routing design and must be narrowed to that setting.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the proposed RL MMs 'significantly outperform traditional and baseline algorithmic strategies' rests on experiments in which (Section 3.3.1) the 50 investor agents 'consistently opt to match their orders with the MM offering the narrowest spread' and 'transaction fees were not considered.' The same greedy-routing, zero-fee design is used in Section 4.1.2. Under this design, the MM problem is a pure zero-sum spread competition where the only way to earn reward is to quote a narrower spread than the competitors; the competitors are a Random-MM and a Persistent-MM that choose spreads randomly or fix them for the whole round. The DQL-MM learns negative buy/sell epsilons (roughly -0.2 to -0.4), i.e., it undercuts the market spread and captures all investor flow. Because there are no fees and no informed order flow, undercutting is unambiguously profitable. Thus the experiments demonstrate that a DQN can learn to undercut two random agents in a frictionless, adversarially simple setting, but they do not demonstrate market-making skill under realistic competition. No classical market-making algorithm (e.g., Avellaneda-Stoikov) is included as a baseline, so the word 'traditional' in the abstract is unsupported. The thesis itself acknowledges the sim-to-real gap in Sections 3.5 and 4.1.3, but the abstract and central claim are not similarly qualified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD thesis develops and evaluates reinforcement-learning (RL) market making agents in the ABIDES simulated market. Chapter 3 formulates market making as a deep Q-learning task with discrete buy, sell, and hedging actions, and studies single-agent, multi-agent, and direct-transfer scenarios. Chapter 4 addresses inventory management in two ways: a reward-engineered dynamic penalty called RIM with the AIIF and DITF coefficients, and a multi-objective agent called M3ORL that maintains separate Q-functions for mark-to-market and inventory objectives and uses Pareto-front tools. Chapter 5 introduces POW-dTS, a discounted-Thompson-sampling policy weighting method intended to adapt to non-stationary market regimes, and compares it with continual-learning baselines. The abstract's central claim is that the proposed RL-based approaches significantly outperform traditional and baseline algorithmic strategies across various performance metrics.","tokens_in":1957,"tokens_out":1777,"duration_ms":83258,"significance":"If the claims are taken in their stated breadth, the thesis would contribute useful techniques for RL market making: a dynamic inventory penalty that adapts to the agent's cash-to-inventory ratio, a Pareto-front multi-objective formulation with separate Q-networks, and a Thompson-sampling mechanism for combining pretrained policies. The experimental work is careful in several respects: most setups use 250 simulations, results are reported with rolling averages and standard deviations, multiple metrics are tracked (MtM, inventory, cash/inventory ratio, hypervolume, sparsity), and Chapter 5 includes an ablation study. The SHAP analysis and the reported evolution of buy/sell/hedge epsilons also provide useful interpretability. The main weakness is external validity: because the simulated investors always route to the narrowest spread and transaction fees are zero, the market making task reduces to a zero-sum spread competition, and the only baselines are a random and a persistent market maker. No classical market-making algorithm is implemented, so the abstract's phrase 'traditional and baseline algorithmic strategies' overstates what is actually tested.","major_comments":[{"comment":"The abstract claims that the proposed RL approaches 'significantly outperform traditional and baseline algorithmic strategies,' but the experiments only compare against a Random-MM and a Persistent-MM. No classical market-making strategy such as Avellaneda-Stoikov or a spread-based heuristic is implemented. Moreover, Section 3.3.1 states that the 50 investor agents always match with the MM offering the narrowest spread and that transaction fees are not considered; Section 4.1.2 repeats this design. Under these assumptions the market making problem is a pure zero-sum spread competition in which the dominant strategy is to undercut the spread, which is exactly what the DQL-MM learns (buy and sell epsilons converge near -0.2 to -0.4). The headline claim should be scoped to this simulated setting, or the experiments should include a classical baseline and variants with fees and non-greedy investor routing.","section":"Abstract; Section 3.3.1; Section 4.1.2"},{"comment":"The paper selects AIIF=0.2 as the best-performing reward configuration after testing ten AIIF values, but no multiple-comparison correction or pre-specified selection rule is reported. Given the reported standard deviations (for example, AIIF=0.2 gives MtM 12,833 ± 66 while AsymDamp gives 11,672 ± 162 over the tested sessions), the differences among the top configurations may not be statistically significant. The word 'significantly' in the abstract is not supported by significance tests. The authors should either provide paired hypothesis tests with an appropriate correction for the multiple AIIF values, or explicitly frame the comparison as exploratory.","section":"Section 4.1.2; Table 4.2; Table 4.3"},{"comment":"The text states that alpha in R2(s,a) is only a scaling factor for the weight w and that it does not affect learning or agent performance. This is not justified by the equations as written: R2 enters the TD target for Q2, so different alpha values produce differently scaled Q2 estimates, and the greedy action selection argmax(w*Q1 + (1-w)*Q2) then changes the effective trade-off between the two objectives. If the Pareto-front results are robust to alpha, the authors should demonstrate this by showing that w is adjusted inversely with alpha or that Q-values are normalized before combination; otherwise the reported fronts and comparisons may depend on an arbitrarily chosen constant.","section":"Section 4.2.1; Eq. (4.19); Algorithm 3"},{"comment":"The POW-dTS algorithm introduces hyperparameters alpha_inc, beta_inc, and gamma, and the experiments appear to use a single configuration (e.g., alpha_inc=1, beta_inc=1, gamma=0.4). No sensitivity analysis for these hyperparameters is reported, so the claimed advantage over continual-learning baselines may be specific to this hand-chosen setting. At minimum, the authors should report a small sensitivity study or justify the chosen values from the ablation.","section":"Section 5.4; Figure 5.9"}],"minor_comments":[{"comment":"The heading 'Markov decission process' contains a typo; it should be 'Markov decision process'.","section":"Section 2.1.1"},{"comment":"The subsection title 'Multi-objetive RL MM: M3ORL' contains a typo; it should be 'Multi-objective RL MM: M3ORL'.","section":"Section 4.2.1"},{"comment":"The coefficient is written as 'Alfa Inventory Impact Factor' in Definition 2 but as 'Alpha' elsewhere; the spelling should be consistent.","section":"Definition 2, Section 4.1.1"},{"comment":"The table header and footnote notation (e.g., 'Experiment Avg 1 σ2 Var3') is difficult to parse, and the commas in values such as '413,121,21' appear to be formatting artifacts. Please reformat the table for readability.","section":"Table 4.2"},{"comment":"The thesis is a compilation of published and submitted papers, but the chapters do not indicate which parts correspond to which source paper. A short note at the start of each chapter would help the reader understand the provenance and the amount of new material added.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the gap between the abstract's broad claim and the experimental scope: the baselines are random/persistent agents in a zero-fee, narrowest-spread-routing simulation, with no classical market-making baseline. This is fixable by substantially qualifying the claims and, ideally, adding a classical baseline and robustness experiments. The second concern is the alpha-scaling claim in M3ORL, which appears internally inconsistent as written. The editor may also wish to confirm that the dissertation's inclusion of previously published articles is acceptable under the journal's prior-publication policy, and whether the absence of a code repository is a barrier for a methods-oriented contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a real PhD thesis, not a paper-shaped object, and it has three genuinely new pieces — a dynamic inventory penalty (RIM with AIIF/DITF), a Pareto-front MORL market maker (M3ORL), and a Thompson-sampling policy switcher (POW-dTS). The work is carefully organized, the experiments are extensive (250 simulations per setup, rolling averages, standard deviations), and the authors are honest about the sim-to-real gap in Sections 3.5 and 4.1.3. Credit where due: the methods go beyond the cited prior work, which mostly uses static inventory penalties or single-objective rewards, and the thesis gives a useful tour of how to map market making onto DQN/MORL in ABIDES.\n\nThe soft spot is the central claim. The abstract says the RL agents 'significantly outperform traditional and baseline algorithmic strategies,' but the experiments compare only against Random-MM and Persistent-MM, with zero transaction fees and investor agents that always route to the narrowest spread. Under those assumptions, undercutting the spread is unambiguously profitable, so the DQN learning to quote negative epsilons is not evidence of market-making skill. No Avellaneda-Stoikov or other classical baseline appears, so the word 'traditional' is not supported. The reader's stress-test note lands. That said, the flaw is in the strength of the claim, not in the methods themselves; the RIM/M3ORL/POW-dTS components could still be useful if tested against stronger baselines and more realistic market microstructure.\n\nTwo secondary issues: the best-performing AIIF (0.2) is selected from ten tested values without multiple-comparison correction, and no statistical significance tests are reported. The POW-dTS chapter was truncated in the version I saw, so I could not assess its experimental section. No code or config files are released, which makes reproducibility harder.\n\nWho is this for? Researchers working on RL market making, especially those interested in inventory-aware rewards or multi-objective formulations. It deserves a serious referee — the thesis has real contributions and the experimental framework is reusable — but the authors should be asked to add a classical baseline, temper the abstract, and report significance or at least corrected selections. My own verdict would be 'major revisions,' not rejection.","headline":"A well-structured RL market-making thesis with three genuinely new components, but the headline outperformance claim rests on a simulator where zero fees and greedy spread-routing make undercutting trivially profitable against weak baselines.","tokens_in":48348,"tokens_out":1536,"would_cite":true,"duration_ms":18868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that deep reinforcement-learning market makers, trained with dynamic inventory penalties and a Pareto-front multi-objective objective, beat random and persistent baseline agents on average mark-to-market in a simulated…","keywords":["reinforcement learning","market making","inventory risk management","multi-objective reinforcement learning","Pareto front","Thompson sampling","non-stationary markets","deep Q-learning"],"falsifier":"Run the trained policies in a simulator variant where investor orders are split stochastically among market makers, or where a small transaction fee is charged, and measure average end-of-session mark-to-market; the central claim fails if the RL agents no longer beat the random and persistent baselines in that setting.","tokens_in":47198,"feed_emoji":"📈","tokens_out":6788,"duration_ms":67502,"temperature":0.7,"pith_summary":"This doctoral thesis sets out to show that reinforcement learning can do the full market-making job: post competitive bid and ask prices, control inventory risk, and keep adapting when the market changes. It claims that a deep Q-learning agent quoting buy, sell, and hedge actions from a discrete grid learns to profit in a simulated market where it competes with random and persistent market makers. To control inventory, the thesis introduces two mechanisms: a reward-engineering penalty with an alpha/timing coefficient pair, and a multi-objective RL agent that keeps profit and inventory as separate objectives and optimizes a Pareto front. To handle market drift, a discounted Thompson-sampling combiner (POW-dTS) weights pretrained policies online. If the claims hold, the practical consequence is a parameter-light recipe for building adaptive, inventory-aware market-making agents.","feed_headline":"RL market makers beat baseline agents with dynamic inventory control","feed_subtitle":"Learned agents post higher mark-to-market returns and keep inventory in check in a simulated stock market.","key_machinery":"The machinery is a deep Q-network with a discretized three-lever action space: at each step the agent chooses a buy-spread multiplier, a sell-spread multiplier, and a hedging fraction over a fixed grid. The thesis wraps this agent in three progressively stronger controllers: a reward term that penalizes inventory outside a dynamic threshold set by cash and recent mid-price; a multi-objective version with two independent DQN blocks whose weighted Q-values are used for action selection and whose undominated policies form a Pareto front; and a policy-weighting layer (POW-dTS) that draws a policy from a pretrained library according to discounted Thompson sampling, so the active policy mixture follows changing market conditions. The mechanism carries the argument by making the learning objective explicit and tunable: a higher alpha means stricter inventory control, the utility weight chooses a point on the Pareto front, and the Thompson-sampling discount favors policies that have earned recent reward.","core_discovery":"The central claim is that a DQN-based market maker operating in a simulated order-book market can learn to out-trade simple baselines: in single-agent and multi-agent competitions, the learned agent ends with positive average mark-to-market while random and persistent agents end negative. The thesis further claims that inventory can be managed without sacrificing profitability by making the penalty dynamic; two coefficients, the Alpha Inventory Impact Factor and the Dynamic Inventory Threshold Factor, set a time-varying inventory threshold based on cash and recent mid-prices. The M3ORL variant replaces the scalar reward with two Q-networks, one for mark-to-market and one for inventory, selected via a utility weight, producing a Pareto front of undominated policies and beating the reward-engineered version on multi-objective metrics. Finally, POW-dTS, which combines pretrained policies with discounted Thompson sampling, is claimed to outperform single-policy retraining and continual-learning baselines in non-stationary market scenarios.","pith_inferences":["The zero-fee, narrowest-spread routing rule makes the experiment a pure spread tournament; a natural next test is to add a per-trade fee or queue-position priority and see how much of the measured margin survives.","The dynamic penalty logic could be made volatility-adaptive by letting the risk coefficient track realized volatility; the thesis lists this as future work, so it is a testable extension rather than a claim.","POW-dTS is a general recipe for non-stationary RL: any domain that can supply a handful of pretrained policies could use the same discounted Thompson-sampling selector, for example robotics or recommendation systems.","The Pareto-front agent implies that inventory risk and profit are separable enough for an operator to choose a target point on the front; an empirical test would compare the front's best policy for a given utility against a scalarized agent tuned to that same utility."],"forward_implications":["A sweep over the inventory-penalty coefficient AIIF yields a family of market makers with different risk profiles, so an operator can select the desired spread aggressiveness without retraining from scratch.","Because the multi-objective agent stores each objective in its own Q-network, the learned value estimates remain usable when the operator later changes the trade-off weight, letting one training run serve many utility preferences.","Discounted Thompson sampling provides a continual-adaptation layer that can be applied on top of any library of pretrained policies, not only the Pareto-front agents trained here.","The learned policies converge to a recognizable pattern, with buy and sell spreads slightly tighter than market and about 20 percent of inventory hedged per step, suggesting the RL agent discovers a simple interpretable market-making rule in this environment.","If the claims transfer out of simulation, trading desks could replace hand-tuned spread and inventory rules with agents that tune themselves to each market regime."],"supporting_citations":[{"why":"Supplies the deep Q-learning algorithm with experience replay and target networks that all agents in the thesis use.","marker":"[17]"},{"why":"Provides the event-based simulated market environment in which all experiments are run.","marker":"[75]"},{"why":"Supplies the prior RL market-making formulation and the state/action design that this work adapts.","marker":"[88]"},{"why":"One of the benchmark reward functions used for comparison in the inventory-management experiments.","marker":"[71]"},{"why":"Another benchmark reward function with a full inventory penalty used for comparison.","marker":"[78]"},{"why":"Provides the multi-objective RL training framework and evaluation methodology that the M3ORL agent builds on.","marker":"[89]"},{"why":"Elastic weight consolidation is a continual-learning baseline in the non-stationary comparison.","marker":"[53]"},{"why":"Data rehearsal is a continual-learning baseline in the non-stationary comparison.","marker":"[50]"},{"why":"Freezing layers is a continual-learning baseline in the non-stationary comparison.","marker":"[49]"}],"fun_headline_variants":["RL agents beat baselines with adaptive inventory control","Reinforcement learning improves market making profit and risk","POW-dTS helps RL agents adapt to shifting market conditions","Multi-objective RL finds Pareto-optimal market-making policies","RL market maker learns to balance profit and inventory risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that investor orders always match the market maker with the narrowest spread and that trades carry zero fees; if real order routing is less price-focused or fees matter, the learned policies' edge in the simulated tournament may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["RL agents beat baselines with adaptive inventory control","Reinforcement learning improves market making profit and risk","POW-dTS helps RL agents adapt to shifting market conditions","Multi-objective RL finds Pareto-optimal market-making policies","RL market maker learns to balance profit and inventory risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2917,"prompt_tokens":980,"completion_tokens":1937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1859}},"tokens_in":596,"tokens_out":1937,"duration_ms":15459,"temperature":1.0,"reasoning_tokens":1859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:10:00.115789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained policies in a simulator variant where investor orders are split stochastically among market makers, or where a small transaction fee is charged, and measure average end-of-session mark-to-market; the central claim fails if the RL agents no longer beat the random and persistent baselines in that setting.","supporting_citations":[{"cited_title":"Market making with signals through deep re- inforcement learning,","cited_arxiv_id":null,"evidence_quote":"Another benchmark reward function with a full inventory penalty used for comparison."},{"cited_title":"Overcoming catastrophic forgetting in neural networks,","cited_arxiv_id":null,"evidence_quote":"Elastic weight consolidation is a continual-learning baseline in the non-stationary comparison."}],"review_version":2}