{"id":"dbd9138a-aee1-4aa1-9f6c-c2be0aa7cac1","arxiv_id":"2611.16992","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A stochastic control and reinforcement learning framework for media trading with censored auction prices, including a consistent price estimator from exploratory bids and numerical comparisons of plug-in policies.","lead":"This paper models real-time bidding for advertising as a continuous-time stochastic control problem where market-clearing prices are only observed when bids win. It proposes a price estimator using randomized exploratory subcampaigns and compares model-based (HJB/PINN) and model-free (actor-critic) computational approaches on synthetic data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Price estimator bound assumes compact support that the experimental model violates; the gap between plug-in policy and true optimal control is not quantified.","rationale":"The reader's weakest_assumption correctly identifies the compact-interval assumption as the most fragile point. The paper's own Remark 4.1 and the unbounded OU model in the experiments confirm this tension. My stress-test adds that even if the estimator were perfect, the plug-in policy's performance relative to the true partially observed optimum is unquantified. The concrete test I propose would separate the effect of the estimator's truncation error from the inherent suboptimality of the plug-in approach. The verdict remains CONDITIONAL because the authors could address the compact-support issue by modifying the estimator or by providing a bound for unbounded processes, and they could add a decomposition of the performance gap. The reader and I agree on the main concern; I do not see a reason to change the verdict category.","tokens_in":18408,"tokens_out":632,"duration_ms":6665,"concrete_test":"Re-run the synthetic experiment of Section 7 with a price process that is bounded almost surely (e.g., a reflected OU or a process with compact state space) so that the compact-support assumption of Lemma 4.1 holds. Compare the performance gap between the observed-price and estimated-price policies. If the gap narrows substantially relative to Figure 10, then the truncation error from the unbounded OU is a major driver of the performance loss, and the estimator's theoretical guarantee is not the limiting factor. If the gap persists, then the plug-in approximation itself (not the estimator) is the dominant source of suboptimality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of a consistent price estimator with MSE O(N^{-1}) (Lemma 4.1) relies on the assumption that the latent price St lies almost surely in a known compact interval K (Remark 4.1). The authors acknowledge this assumption is violated by the Ornstein–Uhlenbeck model used in their own experiments (Section 7.1, sentence [b242]). The estimator (4.4) projects onto K, introducing a truncation error that is not bounded by the lemma. More critically, the plug-in policy that replaces St with the estimated price ŜN_t in the HJB feedback does not come with any performance guarantee relative to the true partially observed optimum. Proposition 4.2 bounds the payoff difference for a fixed policy under an anti-concentration condition, but the authors explicitly state (sentence [b145]–[b147]) that this does not establish near-optimality of an optimizer of the proxy problem. The numerical comparison (Figure 10) shows a substantial gap between observed-price and estimated-price policies, but the paper does not decompose this gap into estimation error vs. the fundamental cost of using a plug-in policy in a controlled filtering problem. Without such a decomposition, the practical relevance of the estimator for control remains unclear.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper formulates a continuous-time stochastic control model for media trading where the market-clearing price is latent and observed only when a bid wins. It proposes a consistent price estimator from randomized exploratory subcampaigns with a mean-square error bound O(N^{-1}). Two computational approaches are described: a model-based HJB/PINN method and an entropy-regularized actor-critic method. Synthetic experiments illustrate the estimator and compare several policies. The paper acknowledges that the plug-in policy does not solve the partially observed control problem and that the estimator's MSE bound assumes compact support violated by the experimental model.","tokens_in":18838,"tokens_out":2064,"duration_ms":13121,"significance":"If the results hold, the paper provides a clear formulation of the information-constrained bidding problem and a tractable price estimator with finite-sample guarantees. The distinction between full-information benchmarks, estimated-price plug-in policies, and the genuinely partially observed problem is useful. However, the central claim of a consistent estimator with MSE O(N^{-1}) relies on an assumption (compact support) that is violated in the experiments, and the gap between plug-in policy and optimal control is not quantified. The paper is honest about limitations but does not resolve them.","major_comments":[{"comment":"The mean-square error bound of Lemma 4.1 assumes that the latent price St lies almost surely in a known compact interval K. The authors acknowledge in Remark 4.1 that this assumption is violated by the Ornstein–Uhlenbeck model used in the experiments (Section 7.1). The estimator (4.4) projects onto K, introducing a truncation error that is not bounded by the lemma. Since the experiments are the primary demonstration of the estimator's performance, the practical validity of the O(N^{-1}) bound is unclear. The paper should either provide a bound that holds under the experimental model (e.g., using a truncation argument with a tail bound) or explain why the violation is negligible in the experiments.","section":"§4.2, Lemma 4.1 and Remark 4.1","authors_acknowledge":"Remark 4.1.The mean-square error bound in Lemma 4.1 assumes that the latent price lies almost surely in a fixed compact interval.","authors_acknowledge_p":0.78},{"comment":"Proposition 4.2 bounds the payoff difference for a fixed policy under an anti-concentration condition, but the authors explicitly state (sentences [b145]–[b147]) that this does not establish near-optimality of an optimizer of the proxy problem. The numerical comparison (Figure 10) shows a substantial gap between observed-price and estimated-price policies. The paper does not decompose this gap into estimation error vs. the fundamental cost of using a plug-in policy in a controlled filtering problem. Without such a decomposition, the practical relevance of the estimator for control remains unclear. The paper should either provide a bound on the suboptimality of the plug-in policy under appropriate assumptions or clearly state that the estimator is only useful for monitoring, not for control.","section":"§4.2, Proposition 4.2 and surrounding discussion","authors_acknowledge":"Therefore convergence of the price estimator alone does not establish near-optimality of an optimizer of the proxy problem.","authors_acknowledge_p":0.93},{"comment":"The HJB equation (5.2) is derived under full information. The paper then uses it as a plug-in approximation by replacing St with the estimated price. The authors note (sentence [b39]) that this is not the solution of the partially observed control problem. However, the paper does not discuss the separated problem (the controlled filtering equation) or provide any justification for the plug-in approach beyond computational convenience. This is a significant gap: the paper claims to address a partially observed control problem, but the main computational method does not actually solve it. The paper should either develop a rigorous approximation theory for the plug-in policy or reframe the contribution as purely heuristic.","section":"§5.2, HJB equation","authors_acknowledge":"In particular, the estimated-price feedback is aplug-in strategy, not the solution of the underlying partially observed control problem.","authors_acknowledge_p":0.76},{"comment":"The comparison in Figure 10 includes a PID budget-pacing rule that is not a solution to the optimization problem (3.4). The paper states (sentence [b264]) that the PID rule ignores the market price, campaign performance, and the agency margin. Including it as a benchmark is reasonable, but the paper does not provide a clear interpretation of the comparison. The gap between observed-price and estimated-price policies is attributed to both exploration cost and threshold effect, but no quantitative decomposition is given. The paper should either provide a more detailed analysis of the gap or temper the claims about the estimator's usefulness for control.","section":"§7.5, Figure 10 and comparison","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The related work section mentions [9] but does not clearly distinguish the contributions of the present paper from that work beyond stating that the decision problem differs. A more explicit comparison of the modeling assumptions and results would be helpful.","section":"§1, Introduction","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Hamiltonian (2.1) includes a term κ(1+β)^2 a^2, but the derivation of the optimal bid awin uses a0(p) = (β - (1+β)p_y) / (2κ(1+β)^2). This is correct, but the notation could be clarified: the denominator is 2κ(1+β)^2, not 2κ(1+β)^2? It is correct as written, but the parentheses are dense.","section":"§2.2, Linear–quadratic benchmark","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The capped exponential distribution (4.1) is defined with a normalization factor 1 - e^{-λ Amax}. This is correct for a truncated exponential, but the denominator should be 1 - e^{-λ Amax} (not 1 - e^{-λ Amax}? Actually it is written as 1 - e^{-λ Amax}, which is correct. The expression is fine.","section":"§4.1, Equation (4.1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The proof states that projection onto an interval does not increase distance to a point in that interval. This is true if the point is in the interval, but if St is outside K (which is assumed not to happen), the projection would introduce error. The assumption that St ∈ K almost surely is critical.","section":"§4.2, Lemma 4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The calibration loss L(η) includes a data term L_data(η) that is not defined. The paper should specify what this term measures (e.g., likelihood, mean-square error between observed and predicted moments).","section":"§4.3, Forward-equation calibration","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The policy iteration algorithm is described at a high level. The paper does not specify how the policy is represented or how the pointwise maximization is performed (e.g., grid search, optimization). More implementation details would improve reproducibility.","section":"§5.3, Algorithm 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The critic loss L_critic(θ) is defined as the sum of squared conditional expectations E[δ_k | O_k]. In practice, this is often approximated by the mean squared temporal-difference error. The paper should clarify how the conditional expectation is estimated (e.g., using a function approximator).","section":"§6.3, Actor–critic approximation","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The parameter d = 0.1 is the share of budget allocated to price estimation. The paper should explain how this share is implemented: is it a separate budget for exploratory subcampaigns, or is it deducted from the main budget? The text says 'using an exploratory allocation corresponding to 10% of the initial budget' (Section 7.2), but the budget dynamics in Section 3 do not account for this deduction. The paper should clarify the budget accounting for exploration.","section":"§7.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The figure caption and text describe the comparison but do not report error bars or confidence intervals. Given the stochastic nature of the problem, reporting standard errors or quantiles would strengthen the comparison.","section":"§7.5, Figure 10","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The limitations section is brief. The paper should explicitly state that the plug-in policy is not guaranteed to be near-optimal and that the estimator's MSE bound does not apply to the experimental model. These are already mentioned in the text, but a consolidated limitations paragraph would be helpful.","section":"§8, Discussion","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The derivation of Z^π as a diffusion process assumes that π_t(a) is C^1 in a and has finite variation in time. This is a strong assumption for a learned policy. The paper should note that this is a theoretical expression and may not hold for neural-network policies.","section":"Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses the term 'media trading' but the model is more general. The title and abstract should reflect that the focus is on real-time bidding with censored prices.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nThis paper formulates media trading as a continuous-time stochastic control problem where the agency only sees clearing prices when it wins an auction. That is a real operational constraint, and the paper does a good job of making the trade-offs explicit: campaign performance, margin, budget depletion, and the fact that bids determine what you learn. The main technical contribution is a price estimator from randomized exploratory subcampaigns, with a mean-square error bound of order N^{-1}. The paper then shows two computational routes—HJB/PINN and actor-critic—and compares them on synthetic data, including a PID budget-pacing heuristic as a practical baseline.\n\nWhat the paper does well: it is transparent about its limitations. The authors state repeatedly that the plug-in policy is not the solution of the partially observed control problem, that the estimator bound assumes a compact interval violated by their own OU model, and that the numerical comparisons are illustrative. That honesty is rare and should be credited. The framework is clearly laid out and could serve as a reference for anyone working on bidding under censored prices.\n\nThe soft spots are real but acknowledged. The compact-interval assumption for the MSE bound (Lemma 4.1) is violated by the OU model in the experiments (Remark 4.1). The authors flag this, but it means the central theoretical guarantee does not apply to the setting they simulate. The gap between the plug-in policy and the true optimal control is not quantified beyond Proposition 4.2, which the authors themselves say does not establish near-optimality. The numerical experiments lack error bars and sensitivity analysis, so the comparisons are suggestive rather than conclusive.\n\nWho this is for: researchers working on stochastic control with censored observations, or on the intersection of control and RL for advertising. The paper is a modeling and computational framework, not a breakthrough in theory or a validated empirical method. It deserves a serious referee because the problem is well-motivated, the formulation is clean, and the limitations are honestly stated. A referee can push for tighter numerical evidence and a discussion of how to handle the compact-support gap, but the core of the paper is sound.\n\nI would accept this for peer review. It is not a desk reject.","headline":"A clean formulation of a practical problem, with honest limitations; the central estimator bound has a known gap with the experiments, but the paper is transparent about it and the framework is useful.","tokens_in":19254,"tokens_out":549,"duration_ms":6751},"pith_extraction":{"msc":["90B60","93E20","93E35"],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"This paper formulates media trading as a continuous-time stochastic control problem where the market price is latent and only observed when a bid wins, and proposes a price estimator from randomized exploratory subcampaigns with a…","keywords":["real-time bidding","stochastic control","censored data","price estimation","reinforcement learning","physics-informed neural networks","budget allocation"],"falsifier":"Run the price estimator on synthetic data where the latent price follows a process that frequently exits the assumed compact interval K (e.g., a Brownian motion with large volatility). If the mean-square error does not decrease as 1/N or becomes unbounded, the estimator's theoretical guarantee fails in that setting, confirming the limitation the authors acknowledge.","tokens_in":19883,"feed_emoji":"📈","tokens_out":1763,"duration_ms":11597,"temperature":0.7,"pith_summary":"This paper addresses the problem faced by a marketing agency bidding in real-time auctions: the market-clearing price is only observed when the agency wins an auction, so the information available for future decisions depends on past bids. The authors formulate this as a continuous-time stochastic control model with a latent price process. To construct a usable price signal, they propose running parallel exploratory subcampaigns with bids drawn from a capped exponential distribution, independent of the price. Under a stylized cross-sectional sampling model, the resulting estimator of the latent price is consistent and has a mean-square error of order 1/N, where N is the number of exploratory observations. The paper then describes two computational approaches for turning this price signal into bidding policies: a model-based method combining forward-equation calibration with a Hamilton-Jacobi-Bellman equation solved via physics-informed neural networks (PINNs), and an entropy-regularized actor-critic reinforcement learning method. Synthetic experiments illustrate the methods and compare them to a budget-pacing heuristic. A key finding is that the plug-in policy that substitutes the estimated price into the full-information optimal policy suffers a noticeable performance gap due to both the cost of exploration and a threshold effect: a bid near the estimated price wins with high probability in the proxy model but only about half the time in reality. The paper clarifies that the estimated-price feedback is a plug-in strategy, not a solution to the genuinely partially observed control problem.","feed_headline":"Latent price estimated from censored auction data with O(1/N) error","feed_subtitle":"Continuous-time control framework for media trading with randomized exploratory subcampaigns","key_machinery":"The central object is the inverse estimator \\(\\hat{S}^N_t = (h_t|_K)^{-1}(\\operatorname{proj}_{[h_t(s_+), h_t(s_-)]} z^N_t)\\), where \\(z^N_t\\) is the empirical winning fraction from N exploratory subcampaigns and \\(h_t(s) = \\$int_0^{{A_{\\max}}$} q_\\$\\lambda$(a) \\Phi((a-s)/v_t) da\\) is the conditional winning probability given the latent price s. The key identity is that \\(h_t\\) is continuous and strictly decreasing (since its derivative is negative), so it can be inverted on a compact interval K containing the latent price. This inversion, combined with the binomial variance bound \\(\\operatorname{Var}(z^N_t | S_t) \\le 1/(4N)\\) and the Lipschitz property of the inverse on K, yields the mean-square error bound of order \\(1/N\\). The machinery does the work of turning censored auction outcomes into a usable price signal with a known error rate.","core_discovery":"The paper's central claim is that a consistent estimator of the latent market-clearing price can be constructed from censored auction outcomes by running independent exploratory subcampaigns with bids drawn from a capped exponential distribution. The estimator inverts the empirical winning fraction using a known, strictly decreasing function that relates the winning probability to the latent price. The key theoretical result is a finite-sample mean-square error bound of order 1/N, where N is the number of exploratory observations. The paper then shows how this price signal can be used in two computational frameworks for bidding: a model-based HJB/PINN approach and a model-free actor-critic approach. The paper's main contribution is the formulation of the media trading problem with control-dependent information and the proposal of a practical, theoretically grounded method for estimating the hidden price, along with a clear delineation of the limitations of the plug-in approach.","pith_inferences":["The paper's framework could be extended to jointly optimize the exploration budget and the main campaign policy, rather than treating exploration as a fixed-cost add-on. This would address a key limitation acknowledged by the authors.","The assumption that the latent price lies almost surely in a known compact interval K is violated by the Ornstein-Uhlenbeck model used in the experiments, as the authors note. A practical implementation would need to handle unbounded price processes, perhaps by using adaptive or robust estimation techniques.","The threshold effect identified in the paper (a bid near the estimated price wins with probability ~1/2 in reality) suggests that a policy that occasionally 'overbids' relative to the estimate might improve performance by increasing the win rate, at the cost of higher expenditure. This could be explored as a simple heuristic.","The actor-critic approach is described as model-free, but it still requires a sufficiently rich observation state. The paper does not establish convergence of the discrete-time actor-critic to a continuous-time optimal policy, so practitioners should be cautious about its theoretical guarantees."],"forward_implications":["If the price estimator works as claimed, agencies can obtain a real-time price signal from a small fraction of their budget allocated to randomized exploration, without needing to change the main campaign's optimization.","The mean-square error bound of O(1/N) provides a theoretical guarantee that the price estimate improves with more exploration, allowing agencies to trade off exploration cost against estimation accuracy.","The paper's clear demonstration that the plug-in policy (using the estimated price in a full-information optimal policy) performs worse than the full-information benchmark highlights the need for more sophisticated methods that account for the uncertainty in the price estimate, such as robust or Bayesian approaches.","The two computational frameworks (HJB/PINN and actor-critic) provide a starting point for implementing bidding strategies that use the estimated price, with the actor-critic method being model-free and thus potentially more robust to model misspecification.","The comparison with the budget-pacing heuristic shows that a simple rule that ignores price and performance can achieve high performance but low criterion value, underscoring the importance of the full objective function that includes margin and budget penalties."],"supporting_citations":[{"why":"Provides the standard reference for continuous-time stochastic control and the well-posedness of the controlled SDEs under the Lipschitz and growth conditions assumed in the paper.","marker":"[12]"},{"why":"Supplies the physics-informed neural network (PINN) methodology used in the model-based HJB approach.","marker":"[13]"},{"why":"Provides the entropy-regularized stochastic control and continuous-time reinforcement learning framework that the actor-critic method is inspired by.","marker":"[15]"},{"why":"Supplies the continuous-time policy evaluation and temporal-difference learning framework that the actor-critic method builds upon.","marker":"[5]"},{"why":"Supplies the continuous-time Q-learning framework that the actor-critic method builds upon.","marker":"[6]"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The mean-square error bound of the price estimator requires that the latent price lies almost surely in a known compact interval K and that the derivative of the winning probability function is bounded away from zero on that interval; the paper's own experiments use an Ornstein-Uhlenbeck price process that does not satisfy the compactness assumption.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7.2e-05,"raw_usage":{"cost":7.23632e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.23632e-05,"upstream_inference_prompt_cost":7.19992e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":998,"prompt_tokens":996,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":996,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":2456,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T07:36:12.017302+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the price estimator on synthetic data where the latent price follows a process that frequently exits the assumed compact interval K (e.g., a Brownian motion with large volatility). If the mean-square error does not decrease as 1/N or becomes unbounded, the estimator's theoretical guarantee fails in that setting, confirming the limitation the authors acknowledge.","supporting_citations":[{"cited_title":"Springer-Verlag Berlin Heidelberg, 2009","cited_arxiv_id":null,"evidence_quote":"Provides the standard reference for continuous-time stochastic control and the well-posedness of the controlled SDEs under the Lipschitz and growth conditions assumed in the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the physics-informed neural network (PINN) methodology used in the model-based HJB approach."},{"cited_title":"Reinforcement learning in continuous time and space: A stochastic control approach.Journal of Machine Learning Research, 21(198):1–34, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the entropy-regularized stochastic control and continuous-time reinforcement learning framework that the actor-critic method is inspired by."},{"cited_title":"Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach.Journal of Machine Learning Research, 23(154):1–55, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the continuous-time policy evaluation and temporal-difference learning framework that the actor-critic method builds upon."},{"cited_title":"q-Learning in Continuous Time.Journal of Machine Learning Research, 24(161):1–61, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the continuous-time Q-learning framework that the actor-critic method builds upon."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}