{"id":"2baf698c-3f44-4e6c-9b73-2d04518ffbe7","arxiv_id":"2608.12719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A reverse auction mechanism for LLM routing that stays incentive compatible and individually rational even when providers' success predictions and the platform's evaluations are noisy.","lead":"This paper proposes EA-RAM, a reverse auction where LLM providers bid their own predicted success probability and cost, instead of a central router predicting performance. It proves incentive-compatibility and a welfare-loss bound under noisy predictions and noisy evaluations, and reports improved cost-quality trade-offs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BIC proof assumes either that sellers observe the common evaluation noise epsilon_post or that H is independent of it; under the stated information structure, conditioning on the win event correlates H with the evaluator signal, so E[V tilde_mu_i | H <= s_hat] = V g_i fails and Theorem 3.2 is…","rationale":"The reader's weakest assumption is the same one I would put first: the BIC proof silently treats the seller's belief about payment as V g_i and treats H as uninformative. I agree with that diagnosis. My stress-test pass goes further and says the issue is not merely a missing assumption; under the model as written, the equality E[V tilde_mu_i | H <= s_hat] = V g_i is false whenever epsilon_post is common and the seller does not condition on it or observe it. This invalidates the first-order condition in Theorem 3.2 and the IR calculation in Theorem 3.3, so the central theoretical contribution is not established. The welfare-loss bound can probably be repaired by redefining g_i as the seller's posterior mean, and the BIC proof would then go through if epsilon_post_i is seller-specific and independent, so I do not call for rejection; the paper needs a substantive revision of the information structure and proof. The empirical issue with oracle labels in Table 2 (pi = 0.1 and pi = 0.2 use ground-truth labels) is real and should be fixed, but it is secondary to the theory. Since the reader already issued a conditional verdict and my concern confirms that condition, I leave the verdict unchanged. No machine-checked proofs or released code provide independent support for the BIC claim.","tokens_in":17807,"tokens_out":12108,"duration_ms":147255,"concrete_test":"Numerically instantiate the model with N = 2, logistic sigma, common epsilon_post ~ N(0,1), independent epsilon_ante_i, and seller information I_1 = {phi_1, c_1, epsilon_ante_1} (epsilon_post not observed). Let seller 2 report truthfully and compute seller 1's true expected utility U_1(s_hat) = E[(V tilde_mu_1 - H - c_1) 1_{H <= s_hat}] using E[V sigma(phi_1 + epsilon_post) 1_{H <= s_hat}] rather than V g_1 F_H(s_hat). Check whether s_hat = V E[sigma(phi_1 + epsilon_post) | I_1] - c_1, or the paper's V g_1 - c_1, maximizes U_1. If the argmax differs, Theorem 3.2 fails as stated; repeating with seller-specific independent epsilon_post_i should restore the proof and confirm the needed modeling fix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim (Theorem 3.2) rests on Eq. (5) in Appendix C.2: U_i(s_hat) = F_H(s_hat)(T_bar_i - E[H | H <= s_hat]), which substitutes E[V tilde_mu_i | H <= s_hat] = V g_i. This substitution is valid only if the event H <= s_hat is uninformative about tilde_mu_i. In Section 2.2, h_i = sigma(phi_i + epsilon_post) and g_i = sigma(phi_i + epsilon_post + epsilon_ante_i) share the same epsilon_post, and epsilon_post is common across sellers. Rivals' truthful reports V g_k - c_k therefore also depend on epsilon_post, so H is correlated with epsilon_post and with the winner's evaluator signal. Conditioning on winning (H <= s_hat) changes the conditional expectation of payment. If the seller does not observe epsilon_post, its correct belief is E[sigma(phi_i + epsilon_post) | I_i], not sigma(phi_i + epsilon_post + epsilon_ante_i); if it does observe epsilon_post, the epsilon_ante_i term should not enter the conditional mean. Under either reading, Eq. (5) is not the seller's interim utility, and the first-order condition U_i'(s_hat) = f_H(s_hat)(T_bar_i - s_hat) does not follow. The same gap propagates to the IR guarantee (Theorem 3.3), which is evaluated at the unproven equilibrium report. Appendix C.3 also mixes a random g_i with the expectation g_i^(1) = E[sigma(phi_i + epsilon_post)], so the welfare-loss bound is stated for an object that is not cleanly defined. This is the most load-bearing soft spot because the paper's headline contribution is BIC/IR under Dual Error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EA-RAM, a reverse-auction mechanism for routing LLM queries. Providers bid with self-predicted success probabilities and execution costs; the buyer allocates to the highest reported surplus V p_hat - c_hat and pays the winner V times a noisy evaluator signal minus the runner-up score. The authors model two sources of noise: provider-side prediction error and buyer-side evaluation error, which they call Dual Error. They claim that under this model EA-RAM is Bayesian incentive compatible and individually rational, satisfies sufficient conditions for center rationality, and has an expected welfare loss bounded by 2 V L_sigma (M_post + M_ante) relative to the error-free benchmark. They also report simulation and RouterBench experiments suggesting that EA-RAM is robust to noise and outperforms centralized routing baselines.","tokens_in":18202,"tokens_out":13072,"duration_ms":161858,"significance":"If the theoretical claims were rigorously established, the paper would make a useful contribution: it shifts ex-ante quality prediction from a central router to LLM providers, explicitly models noisy self-prediction and noisy evaluation in a mechanism-design setting, and provides an error-aware welfare bound. The paper is also empirically grounded, with experiments on RouterBench and an LLM-as-a-judge noise study, and it explicitly labels its oracle and realistic information variants. However, the central BIC/IR proof rests on an unexamined conditional-expectation step, and the welfare-loss proof applies a pointwise Lipschitz bound to a random quantity. These issues currently prevent the main theoretical results from being accepted as stated.","major_comments":[{"comment":"The BIC proof replaces E[V tilde_mu_i | H <= s_hat] with V g_i, but under the stated model epsilon_post is a common shock entering every h_i and every g_i, so H = max(0, max_{k neq i}(V g_k - c_k)) is correlated with epsilon_post and therefore with the winner's evaluator signal. Conditioning on the win event H <= s_hat is informative, and E[V tilde_mu_i | H <= s_hat] is not equal to V g_i in general. Consequently Eq. (5) is not the seller's interim utility, the first-order condition U_i'(s_hat) = f_H(s_hat)(Tbar_i - s_hat) does not follow, and Theorems 3.2 and 3.3 are unsupported as written. The authors should either make epsilon_post seller-specific and independent across sellers, or specify the seller's information set precisely and derive the correct conditional expectation; the current text conflates two readings of the information structure.","section":"Section 3.2 and Appendix C.2, Eq. (5)"},{"comment":"The welfare-loss proof bounds |g_i - p_i| <= L_sigma (M_post + M_ante) as if this held pointwise, but g_i = sigma(phi_i + eta_i) is random and the Lipschitz lemma (C.4) bounds the expectation E|sigma(x + epsilon) - sigma(x)|, not the realized absolute deviation |sigma(x + epsilon) - sigma(x)|. For unbounded errors the pointwise inequality is false. The proof needs an explicit expectation step: it should bound E|g_i - E[sigma(phi_i + epsilon_post)]| using the ante channel and |E[sigma(phi_i + epsilon_post)] - p_i| using the post channel, then sum the two bounds. As written, Theorem 3.7 does not follow from the argument given.","section":"Appendix C.3, Eq. (39) and Theorem 3.7"},{"comment":"The CR proof conditions on the selected winner's phi_(1) and applies Lemma C.4 to the unconditional distribution of epsilon_post. But the identity of the winner is selected using reported scores that depend on epsilon_post, so the conditional distribution of epsilon_post given the event that seller (1) wins is not the unconditional distribution used in the lemma. The bound |E[h_(1)] - E[p_(1)]| <= Delta_gate needs an argument that accounts for this selection effect; as it stands, sufficient condition (A) of Theorem 3.4 is not established.","section":"Appendix C.5, proof of Theorem 3.4"}],"minor_comments":[{"comment":"The sentence 'We therefore propose the Error-Aware Reverse Auction Mechanism (EA-RAM) Crucially, EA-RAM explicitly models...' is missing a period or comma after '(EA-RAM)'.","section":"Section 1, paragraph on EA-RAM"},{"comment":"The notation uses epsilon_post without a seller subscript for h_i and inside the definition of g_i, while epsilon_ante,i carries a subscript. Please state explicitly whether epsilon_post is a single common shock across all sellers or a seller-specific shock; this distinction is essential for the BIC conditioning argument.","section":"Section 2.2"},{"comment":"The rows 'with pi=0.1/0.2' and 'with omega=0.1/0.2' should be labeled as 'oracle local information' and 'realistic local information' in the table itself, since the oracle rows use ground-truth labels and are upper bounds rather than deployable configurations.","section":"Table 2"},{"comment":"The statement that the error-aware allocation attains the error-free welfare if and only if the error-aware winner is welfare-optimal is tautological; consider stating it as a definition or replacing it with a discussion of when the argmax is preserved.","section":"Proposition 3.6"},{"comment":"The two-layer MLP description would benefit from specifying hidden dimensions and the exact train/test split procedure for the predictor and evaluator, since the Pareto-frontier comparison depends on these implementation details.","section":"Section 4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The main idea is interesting and the empirical work is substantial, but the theoretical core currently has a load-bearing gap in the BIC proof (common epsilon_post creates correlation between the win event and the evaluator signal) and a technical error in the welfare-loss proof. These are fixable in principle by clarifying the information structure or restricting the noise model, but they need substantive reworking rather than cosmetic changes. I would ask the authors to address the three major comments before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core idea is good, but the BIC theorem has a conditioning gap that appears load-bearing. The seller's interim utility in Eq. (5) treats E[V μ̃_i | H≤ŝ] as V g_i, but H depends on other sellers' reports, which depend on the common evaluator noise ε_post, so winning is informative about ε_post and thus about μ̃_i. This is a winner's-curse problem. The proof doesn't handle it. Theorems 3.2 and 3.3 don't follow as stated.\n\nWhat's actually new: the reverse-auction framing for LLM routing, with the Dual Error model and the structural insights (error cancellation, saturation, noise flattening), is a fresh and sensible pitch. The experiments are decent: in the base setting (no oracle), EA-RAM already beats the centralized baselines on RouterBench; the oracle rows are labeled as such, which is honest. Some of the robustness propositions (3.8–3.10) are plausible and independent of the BIC issue.\n\nSoft spots, in order. First, the information structure is not well-defined: does the seller observe ε_post? If yes, g_i should be h_i; if no, g_i is not a number the seller knows. Either way, the belief used in the proof is not the posterior conditional on winning. This is fixable—say, by making ε_post idiosyncratic to the seller and independent across sellers, in which case H is independent of μ̃_i and the stylized second-price logic goes through. As written, though, the central guarantee is unproven. Second, the welfare-loss bound proof in Appendix C.3 writes a non-random quantity as if bounded by a random one and only takes expectations later; that's a minor stylistic issue, not fatal. Third, the oracle local-information rows give the paper an optimistic sheen, but the base results already carry the qualitative claim.\n\nWho this is for: researchers working on market-based LLM routing and on mechanism design with noisy evaluation. They'll find the problem formulation and experiments worth engaging with, even while the theory is a work in progress. It deserves a serious referee—send it to review, but the referee should be pointed at the conditioning issue and the authors should be expected to fix it before acceptance.","headline":"The reverse-auction-for-LLM-routing idea is worth a look, but the BIC proof conditions on a winning event that is correlated with the evaluator noise, so the paper's central theorem doesn't hold as stated.","tokens_in":18694,"tokens_out":6920,"would_cite":false,"duration_ms":83240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EA-RAM shows that LLM routing can be implemented as an error-aware reverse auction in which providers' honest bids on success probability and cost remain a Bayesian equilibrium even when both self-prediction and the platform's evaluation…","keywords":["LLM routing","reverse auction","mechanism design","Bayesian incentive compatibility","Dual Error","welfare loss bound","cost-performance Pareto","provider-side prediction"],"falsifier":"Give sellers a signal that contains only their own prediction noise, withhold the evaluator's $\\varepsilon_{\\mathrm{post}}$ from them, and run the auction with nonzero evaluation variance; if truthful reporting of $Vg_i-c_i$ is not the expected-utility maximizer, for instance if a seller can profit by over-reporting when the evaluator noise is positive, the BIC theorem as stated is false. Concretely, compute $E[V\\tilde{\\mu}_i|\\text{seller signal}]$ and compare it with $Vg_i$; the proof's key equality fails whenever the seller's signal is not informative about $\\varepsilon_{\\mathrm{post}}$.","tokens_in":17599,"feed_emoji":"⚖️","tokens_out":7896,"duration_ms":85646,"temperature":0.7,"pith_summary":"The paper proposes replacing centralized LLM routers, which predict every model's performance at the task center, with a reverse auction in which LLM providers bid their self-predicted success probability and execution cost. It introduces EA-RAM, whose allocation ranks sellers by reported surplus $V\\hat{p}_i-\\hat{c}_i$ and whose payment to the winner is $V\\tilde{\\mu}_j-H$, where $\\tilde{\\mu}_j$ is the buyer's noisy acceptance signal and $H$ is the runner-up surplus; the noise is modeled explicitly as Dual Error, the provider's ex-ante prediction error plus the center's ex-post evaluation error. The paper proves that under this Dual Error the mechanism is Bayesian incentive compatible and individually rational, gives sufficient conditions for the buyer's expected utility to be non-negative, and bounds the expected welfare loss relative to the error-free benchmark by $2VL_{\\sigma}(M_{\\mathrm{post}}+M_{\\mathrm{ante}})$. If correct, this lets routing scale to large model pools without per-model profiling and makes truthful bidding the equilibrium even when predictions and evaluations are imperfect.","feed_headline":"LLM routing via reverse auction stays incentive-compatible under noise","feed_subtitle":"Trades per-model profiling for a provably bounded-loss auction with honest provider bids.","key_machinery":"The load-bearing object is the pair consisting of a ranking score and a payment rule: the buyer ranks sellers by reported surplus $\\hat{s}_i=V\\hat{p}_i-\\hat{c}_i$, and the winner receives $V\\tilde{\\mu}_j-H$, where $\\tilde{\\mu}_j$ is the binary acceptance signal from the noisy ex-post evaluator and $H$ is the maximum runner-up reported surplus (or zero for the null allocation). Around this pair the paper defines the effective surplus $\\overline{T}_i=Vg_i-c_i$ with the belief map $g_i=\\sigma(\\phi_i+\\varepsilon_{\\mathrm{post}}+\\varepsilon_{\\mathrm{ante},i})$, and the central identity is $U_i^{\\mathrm{seller}\\;\\prime}(\\hat{s}_i)=f_H(\\hat{s}_i)(\\overline{T}_i-\\hat{s}_i)$, which shows that reporting the effective surplus is the unique maximizer of interim expected utility whenever that surplus is positive. The globally Lipschitz link function $\\sigma$ then converts the second-moment radii of the two error channels, $M_{\\mathrm{post}}=\\sqrt{b_{\\mathrm{post}}+a_{\\mathrm{post}}^2}$ and $M_{\\mathrm{ante}}=\\max_i\\sqrt{b_{\\mathrm{ante},i}+a_{\\mathrm{ante},i}^2}$, into the explicit welfare-loss bound.","core_discovery":"The central discovery is that a Vickrey-style allocation rule can survive both noisy provider prediction and noisy platform evaluation if the auction ranks sellers by the reported surplus $\\hat{s}_i=V\\hat{p}_i-\\hat{c}_i$ and pays the winner $V\\tilde{\\mu}_j-H$, where $\\tilde{\\mu}_j$ is the evaluator's acceptance signal and $H$ is the runner-up score. In the error-free setting these rules satisfy dominant-strategy incentive compatibility, individual rationality, center rationality, and economic efficiency. Under Dual Error, with seller belief $g_i=\\sigma(\\phi_i+\\varepsilon_{\\mathrm{post}}+\\varepsilon_{\\mathrm{ante},i})$ and evaluator acceptance probability $h_i=\\sigma(\\phi_i+\\varepsilon_{\\mathrm{post}})$, the paper proves that truthful reporting of the effective surplus $\\overline{T}_i=Vg_i-c_i$ is a Bayesian best response (Theorem 3.2), that all sellers have non-negative expected utility at equilibrium (Theorem 3.3), that the buyer's expected utility is non-negative when either the runner-up margin is large enough or the evaluation is conservative (Theorem 3.4), and that the expected welfare loss relative to the error-free benchmark is at most $2VL_\\sigma(M_{\\mathrm{post}}+M_{\\mathrm{ante}})$ (Theorem 3.7). The argument turns on the derivative identity $U_i^{\\mathrm{seller}\\;\\prime}(\\hat{s}_i)=f_H(\\hat{s}_i)(\\overline{T}_i-\\hat{s}_i)$, which makes the seller's interim expected utility peak exactly at its effective surplus.","pith_inferences":["The incentive proof assumes the seller's belief $g_i$ already incorporates the evaluator's noise $\\varepsilon_{\\mathrm{post}}$; in a real deployment a seller may observe only its own prediction noise, in which case the equality $E[V\\tilde{\\mu}_i|\\cdot]=Vg_i$ used for Bayesian incentive compatibility would not hold and truthful bidding could cease to be an equilibrium. This information-structure as","The paper defines the mechanism with respect to an announced task value $V$ and states as a future direction that misspecification of $V$ may change participation incentives; a natural extension is to analyze how a strategic or inaccurate announced value shifts the equilibrium operating point.","The same reverse-auction template could apply to any setting where providers hold private success probabilities and a noisy verifier exists, such as code execution, math checking, or translation quality, not only LLM quality routing.","The tighter welfare-loss bound under opposite-signed errors suggests that deliberately pairing error regimes, for instance a conservative evaluator with optimistic sellers, could be engineered to reduce allocation distortion in practice."],"forward_implications":["If EA-RAM is deployed, the task center needs only a model-agnostic evaluator and the auction logic; it no longer trains or profiles per-model predictors, so adding a new LLM does not require retraining a router.","The welfare-loss bound shrinks to zero as the second-moment radii of prediction and evaluation errors shrink, so improving evaluator and predictor calibration directly brings routing efficiency close to the error-free benchmark.","Clear-cut queries, where $|\\phi_i|$ is large, are nearly immune to noise when the link function saturates (e.g., logistic), so the mechanism is most sensitive in close competitions between models.","Provider-side local information can shift the empirical cost-performance Pareto frontier upward, as shown on RouterBench, meaning sellers' private signals can be harnessed rather than treated as a threat to the mechanism.","Center-side latency stays nearly constant as the model pool grows, while communication overhead grows linearly with the number of providers and becomes a bottleneck only at low per-channel bandwidths, as the paper's efficiency experiments indicate."],"supporting_citations":[{"why":"Supplies the error-free benchmark mechanism FTMD and the four desiderata (DSIC, IR, CR, EE) that EA-RAM extends to the Dual Error setting.","marker":"[13]"},{"why":"Provides the capability-modeling assumption that fulfillment probability is $\\sigma(\\phi)$ from model ability and task difficulty.","marker":"[6]"},{"why":"Also supplies the ability-difficulty matching model for $p_i=\\sigma(\\phi(m_i,d))$ used in the paper's belief setup.","marker":"[16]"},{"why":"RouterBench is the real-world benchmark that provides the per-query accuracy and cost data for the routing Pareto comparison.","marker":"[17]"},{"why":"EmbedLLM is a centralized baseline that EA-RAM must beat in the cost-performance Pareto frontier analysis.","marker":"[4]"},{"why":"RouteLLM is a centralized baseline used for the empirical comparison against EA-RAM.","marker":"[5]"},{"why":"FrugalGPT is a cascading baseline that EA-RAM is compared with on real-world routing benchmarks.","marker":"[24]"},{"why":"Cascade Routing is another baseline in the empirical evaluation that EA-RAM is designed to outperform.","marker":"[25]"}],"fun_headline_variants":["Reverse auction routes LLM queries with provable noise bounds","Market-based LLM routing survives noisy model predictions","Error-aware auction keeps LLM routing honest under uncertainty","Auction-based LLM routing: incentives hold despite evaluation noise","Bidding fixes LLM routing: noise-proven auction mechanism"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire incentive guarantee presumes that the seller's reported probability matches its expectation of the buyer's noisy evaluation, specifically that the seller's belief $g_i$ incorporates the evaluator's noise $\\varepsilon_{\\mathrm{post}}$, so if sellers observe only their own prediction error and not the evaluator's noise, truthful bidding may no longer be a Bayesian equilibrium.","fun_headline_variants_meta":{"raw":{"variants":["Reverse auction routes LLM queries with provable noise bounds","Market-based LLM routing survives noisy model predictions","Error-aware auction keeps LLM routing honest under uncertainty","Auction-based LLM routing: incentives hold despite evaluation noise","Bidding fixes LLM routing: noise-proven auction mechanism"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1631,"prompt_tokens":1109,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":725,"tokens_out":522,"duration_ms":5849,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:47:08.174147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give sellers a signal that contains only their own prediction noise, withhold the evaluator's $\\varepsilon_{\\mathrm{post}}$ from them, and run the auction with nonzero evaluation variance; if truthful reporting of $Vg_i-c_i$ is not the expected-utility maximizer, for instance if a seller can profit by over-reporting when the evaluator noise is positive, the BIC theorem as stated is false. Concretely, compute $E[V\\tilde{\\mu}_i|\\text{seller signal}]$ and compare it with $Vg_i$; the proof's key equality fails whenever the seller's signal is not informative about $\\varepsilon_{\\mathrm{post}}$.","supporting_citations":[{"cited_title":"Fault tolerant mechanism design.Artificial Intelligence, 172(15):1783–1799, 2008","cited_arxiv_id":null,"evidence_quote":"Supplies the error-free benchmark mechanism FTMD and the four desiderata (DSIC, IR, CR, EE) that EA-RAM extends to the Dual Error setting."},{"cited_title":"IRT-router: Effective and interpretable multi-LLM routing via item response theory","cited_arxiv_id":null,"evidence_quote":"Also supplies the ability-difficulty matching model for $p_i=\\sigma(\\phi(m_i,d))$ used in the paper's belief setup."},{"cited_title":"EmbedLLM: Learning compact representations of large language models","cited_arxiv_id":null,"evidence_quote":"EmbedLLM is a centralized baseline that EA-RAM must beat in the cost-performance Pareto frontier analysis."},{"cited_title":"A unified approach to routing and cascading for LLMs","cited_arxiv_id":null,"evidence_quote":"Cascade Routing is another baseline in the empirical evaluation that EA-RAM is designed to outperform."}],"review_version":1}