{"id":"1f0421ce-a32a-4b38-b7a2-6376a925ba70","arxiv_id":"2606.17805","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"QueryMarket introduces OVBAL, an online active learning rule that selects samples using D-optimality with exponential forgetting and executes purchases under rolling budget constraints for nonstationary streams.","lead":"The paper presents QueryMarket, a framework for deciding which incoming data points to label and purchase in real-time streams by balancing estimated model utility against price under a rolling budget. A smart generalist might read it to see how active learning can be made cost-aware when data sellers set prices and streams change over time.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"D-optimality utility estimates with exponential forgetting may be biased under concept drift, undermining justification for cost-aware decisions","rationale":"The reader's weakest assumption directly identifies the load-bearing point for the experimental claim. Full-text inspection shows the method description and experiments rely on this unverified estimator without additional checks (e.g., no ablation on forgetting factor or correlation analysis). No internal inconsistencies in the decision rule derivation or baseline comparisons were found that would constitute a separate concern.","tokens_in":1699,"tokens_out":336,"duration_ms":24162,"concrete_test":"On the solar power forecasting task, for each queried sample compute the Pearson correlation between its D-optimality utility score (with the paper's forgetting factor) and the observed reduction in held-out MSE after acquiring its label; repeat for the synthetic streams. If correlation is below 0.25 across cost quartiles, the utility estimator's accuracy is insufficient to support the claimed trade-off superiority.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OVBAL produces a more favorable long-run error-cost trade-off via its online decision rule. This rule selects points by comparing estimated marginal utility (D-optimality criterion with exponential forgetting) against price under a rolling budget. For the claim to hold, the utility estimates must be sufficiently accurate proxies for actual error reduction in nonstationary streams; systematic bias (e.g., forgetting factor mismatched to drift rate or model mismatch) would make the purchases suboptimal relative to baselines. The full text confirms this estimator is used without direct validation against realized error reduction on the solar dataset or synthetic streams.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces the QueryMarket framework for cost-aware online active learning under concept drift and heterogeneous label costs. Within this framework, OVBAL estimates each incoming sample's marginal utility via a D-optimality criterion with exponential forgetting, then executes purchase decisions by comparing this utility to price under a rolling budget. The central claim is that this yields a simple, fully online decision rule that adapts to nonstationary streams and produces a more favorable long-run error-cost trade-off than baselines on synthetic data and a real-world solar power generation forecasting task, under both seller-centric and buyer-centric pricing schemes.","tokens_in":1803,"tokens_out":478,"duration_ms":30232,"significance":"If the D-optimality utility estimates are sufficiently accurate proxies for actual error reduction, the work is significant for unifying information gain, pricing, and rolling budget constraints in a fully online manner. The provision of an explicit, parameter-light decision rule that handles nonstationarity is a practical strength for real-time data acquisition settings.","major_comments":[{"comment":"Abstract and OVBAL description: the claim that OVBAL produces a more favorable long-run error-cost trade-off rests on the D-optimality criterion with exponential forgetting serving as an accurate estimate of marginal utility under concept drift. The manuscript uses this estimator for purchase decisions without direct validation against realized error reduction on the solar dataset or synthetic streams; if the estimates are systematically biased (e.g., due to mismatch between forgetting factor and drift rate), the cost-aware decisions lose their justification relative to baselines.","section":"Abstract / OVBAL description"},{"comment":"Experiments section: results are reported showing favorable trade-offs under both pricing schemes, yet no analysis is provided of how the exponential forgetting factor was chosen or whether it was tuned to match observed drift rates on the solar task. This is load-bearing for the adaptation claim, as an arbitrary or post-hoc choice would weaken the assertion that the rule reliably adapts to nonstationary streams.","section":"Experiments"}],"minor_comments":[{"comment":"The abstract would be clearer if it named the specific baselines used for comparison and briefly stated the two pricing schemes.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major point below and will revise the manuscript to strengthen the empirical support for our claims.","responses":[{"response":"We agree that direct validation of the D-optimality utility estimates against realized error reduction would provide stronger justification for the purchase decisions. The current manuscript relies on the theoretical motivation of the criterion without this empirical check on the solar or synthetic data. In the revised version we will add a new analysis (e.g., a figure or table) that correlates the estimated marginal utilities with observed error reductions under the concept drift present in each dataset.","revision_made":"yes","referee_comment":"[Abstract / OVBAL description] Abstract and OVBAL description: the claim that OVBAL produces a more favorable long-run error-cost trade-off rests on the D-optimality criterion with exponential forgetting serving as an accurate estimate of marginal utility under concept drift. The manuscript uses this estimator for purchase decisions without direct validation against realized error reduction on the solar dataset or synthetic streams; if the estimates are systematically biased (e.g., due to mismatch between forgetting factor and drift rate), the cost-aware decisions lose their justification relative to baselines."},{"response":"We acknowledge that the manuscript lacks an explicit description or sensitivity analysis for the choice of the exponential forgetting factor. This information is needed to support the claim of reliable adaptation to nonstationary streams. The revised manuscript will include a paragraph in the experimental setup detailing the selection procedure (including any grid search or validation approach used) and its relation to observed drift rates on the solar task.","revision_made":"yes","referee_comment":"[Experiments] Experiments section: results are reported showing favorable trade-offs under both pricing schemes, yet no analysis is provided of how the exponential forgetting factor was chosen or whether it was tuned to match observed drift rates on the solar task. This is load-bearing for the adaptation claim, as an arbitrary or post-hoc choice would weaken the assertion that the rule reliably adapts to nonstationary streams."}],"tokens_in":1384,"tokens_out":444,"duration_ms":36848,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is OVBAL, a fully online decision rule inside the QueryMarket framing. It scores each incoming point's marginal utility with a D-optimality criterion that includes exponential forgetting, then buys the label if that score justifies the price under the current rolling budget. The combination of seller-centric pricing, forgetting, and budget constraint in one simple rule is the part that is not already in the cited online active learning work.\n\nThe experiments are the part that holds up best. On synthetic drifting streams and the real solar power forecasting task, OVBAL produces a visibly better long-run error versus cumulative cost trade-off than the baselines, and the advantage is clearest under seller-centric pricing. That gives the claim some concrete grounding.\n\nThe soft spot is the utility estimator itself. The method treats the D-optimality score with forgetting as a reliable proxy for how much each label will actually reduce error, yet the paper does not report any direct check of how well that score correlates with observed error reduction on the solar data or under the measured drift rates. If the forgetting factor is mismatched to the actual concept drift, the purchase decisions rest on a biased signal and the reported gains could shrink. No ablation on that point appears.\n\nThis is aimed at people working on streaming models where labels have real acquisition costs. It is worth sending to peer review because the problem is practical, the rule is simple to implement, and the real-world experiment adds evidence even if the utility validation needs tightening.","headline":"OVBAL gives a clean online rule that folds heterogeneous prices and rolling budgets into D-optimality selection with forgetting, and the solar experiments show a better error-cost curve, but the utility proxy is used without direct checks against realized error reduction.","tokens_in":2293,"tokens_out":396,"would_cite":false,"duration_ms":25120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"OVBAL provides a fully online decision rule that buys labels by comparing estimated marginal utility to price under rolling budgets and concept drift.","keywords":["online active learning","data markets","concept drift","D-optimality","cost-aware selection","rolling budget","label acquisition","streaming data"],"falsifier":"An experiment on a drifting stream in which the actual error reduction from OVBAL-selected labels shows no consistent correlation with the D-optimality estimates would indicate the utility scores do not justify the purchase decisions.","tokens_in":2572,"feed_emoji":"","tokens_out":663,"duration_ms":27687,"temperature":0.7,"pith_summary":"The paper presents QueryMarket as a framework for online label acquisition in data streams where analysts face budget limits and changing data distributions. It proposes OVBAL, which estimates each sample's value to the model using a D-optimality criterion combined with exponential forgetting, then decides purchases by weighing that value against the sample's price. This produces an adaptive rule that handles nonstationary streams and varying label costs without requiring offline computation. Experiments on synthetic streams and a solar power forecasting task indicate the rule achieves better long-run error versus cost performance than standard baselines under different pricing models. A reader would care because real-time learning often stalls when labels are expensive and data shifts, so a practical online selection method addresses a common deployment bottleneck.","feed_headline":"Online rule buys labels by balancing utility estimate and price","feed_subtitle":"OVBAL adapts to drifting streams and varying costs, improving long-run error versus spending compared with baselines.","key_machinery":"The D-optimality criterion with exponential forgetting, which computes each incoming sample's estimated reduction in model variance while discounting past data to track drift.","core_discovery":"Within the QueryMarket framework, OVBAL integrates data pricing with information-driven selection by estimating each sample's marginal utility via a D-optimality criterion with exponential forgetting and executing cost-aware purchases under rolling budget constraints, yielding a simple, fully online decision rule that adapts to nonstationary streams and heterogeneous label costs and produces a more favorable long-run error-cost trade-off than baselines under both pricing schemes.","pith_inferences":["The same utility estimation step could be swapped for other information criteria while retaining the cost-aware purchase logic.","Data sellers in markets might respond by setting prices that reflect how buyers' models value samples under similar forgetting rules.","The framework could be tested on additional streaming tasks such as network traffic or financial time series to check robustness beyond solar forecasting."],"forward_implications":["OVBAL produces a more favorable long-run error-cost trade-off than baselines.","The method is particularly effective under seller-centric pricing schemes.","It maintains the improved trade-off under both pricing schemes in the solar power forecasting task.","The decision rule remains fully online and adapts to nonstationary streams without retraining."],"fun_headline_variants":["QueryMarket balances utility and price for online label queries","OVBAL estimates marginal utility via D-optimality with forgetting","Online rule adapts label buys to drifting streams and costs","QueryMarket executes cost-aware purchases under rolling budgets"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The D-optimality criterion with exponential forgetting provides an accurate estimate of each sample's marginal utility to the model under concept drift.","fun_headline_variants_meta":{"raw":{"variants":["QueryMarket balances utility and price for online label queries","OVBAL estimates marginal utility via D-optimality with forgetting","Online rule adapts label buys to drifting streams and costs","QueryMarket executes cost-aware purchases under rolling budgets"]},"model":"grok-4.3","cost_usd":0.005066,"raw_usage":{"total_tokens":2445,"prompt_tokens":622,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":50662000,"prompt_tokens_details":{"text_tokens":622,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1761,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":622,"tokens_out":62,"duration_ms":17596,"temperature":1.0,"reasoning_tokens":1761,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T01:51:00.767081+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment on a drifting stream in which the actual error reduction from OVBAL-selected labels shows no consistent correlation with the D-optimality estimates would indicate the utility scores do not justify the purchase decisions.","supporting_citations":[],"review_version":1}