Pith. sign in

REVIEW 2 major objections 1 minor 36 references

QueryMarket: Cost-Aware Online Active Learning in Data Markets

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OVBAL provides a fully online decision rule that buys labels by comparing estimated marginal utility to price under rolling budgets and concept drift.

desk verdict OVBAL gives a clean online rule that folds heterogeneous prices and rolling budgets into D-optimality selection with forgetting, and the solar experiments show a better error-cost curve, but the utility proxy is used without direct checks against realized error reduction. read the letter →

arxiv 2606.17805 v1 pith:U5CJGHEX submitted 2026-06-16 cs.LG

Xiwen Huang , Pierre Pinson This is my paper · ORCID
classification cs.LG
keywords onlineactivelearningdatamarketsconceptdriftD-optimalitycost-awareselectionrollingbudgetlabelacquisitionstreaming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents QueryMarket as a framework for online label acquisition in data streams where analysts face budget limits and changing data distributions. It proposes OVBAL, which estimates each sample's value to the model using a D-optimality criterion combined with exponential forgetting, then decides purchases by weighing that value against the sample's price. This produces an adaptive rule that handles nonstationary streams and varying label costs without requiring offline computation. Experiments on synthetic streams and a solar power forecasting task indicate the rule achieves better long-run error versus cost performance than standard baselines under different pricing models. A reader would care because real-time learning often stalls when labels are expensive and data shifts, so a practical online selection method addresses a common deployment bottleneck.

What carries the argument

The D-optimality criterion with exponential forgetting, which computes each incoming sample's estimated reduction in model variance while discounting past data to track drift.

What would settle it

An experiment on a drifting stream in which the actual error reduction from OVBAL-selected labels shows no consistent correlation with the D-optimality estimates would indicate the utility scores do not justify the purchase decisions.

Watch

Extended reading notes

Core claim

Within the QueryMarket framework, OVBAL integrates data pricing with information-driven selection by estimating each sample's marginal utility via a D-optimality criterion with exponential forgetting and executing cost-aware purchases under rolling budget constraints, yielding a simple, fully online decision rule that adapts to nonstationary streams and heterogeneous label costs and produces a more favorable long-run error-cost trade-off than baselines under both pricing schemes.

Load-bearing premise

The D-optimality criterion with exponential forgetting provides an accurate estimate of each sample's marginal utility to the model under concept drift.

Editorial extensions

If this is right

  • OVBAL produces a more favorable long-run error-cost trade-off than baselines.
  • The method is particularly effective under seller-centric pricing schemes.
  • It maintains the improved trade-off under both pricing schemes in the solar power forecasting task.
  • The decision rule remains fully online and adapts to nonstationary streams without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same utility estimation step could be swapped for other information criteria while retaining the cost-aware purchase logic.
  • Data sellers in markets might respond by setting prices that reflect how buyers' models value samples under similar forgetting rules.
  • The framework could be tested on additional streaming tasks such as network traffic or financial time series to check robustness beyond solar forecasting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces the QueryMarket framework for cost-aware online active learning under concept drift and heterogeneous label costs. Within this framework, OVBAL estimates each incoming sample's marginal utility via a D-optimality criterion with exponential forgetting, then executes purchase decisions by comparing this utility to price under a rolling budget. The central claim is that this yields a simple, fully online decision rule that adapts to nonstationary streams and produces a more favorable long-run error-cost trade-off than baselines on synthetic data and a real-world solar power generation forecasting task, under both seller-centric and buyer-centric pricing schemes.

Significance. If the D-optimality utility estimates are sufficiently accurate proxies for actual error reduction, the work is significant for unifying information gain, pricing, and rolling budget constraints in a fully online manner. The provision of an explicit, parameter-light decision rule that handles nonstationarity is a practical strength for real-time data acquisition settings.

major comments (2)
  1. [Abstract / OVBAL description] Abstract and OVBAL description: the claim that OVBAL produces a more favorable long-run error-cost trade-off rests on the D-optimality criterion with exponential forgetting serving as an accurate estimate of marginal utility under concept drift. The manuscript uses this estimator for purchase decisions without direct validation against realized error reduction on the solar dataset or synthetic streams; if the estimates are systematically biased (e.g., due to mismatch between forgetting factor and drift rate), the cost-aware decisions lose their justification relative to baselines.
  2. [Experiments] Experiments section: results are reported showing favorable trade-offs under both pricing schemes, yet no analysis is provided of how the exponential forgetting factor was chosen or whether it was tuned to match observed drift rates on the solar task. This is load-bearing for the adaptation claim, as an arbitrary or post-hoc choice would weaken the assertion that the rule reliably adapts to nonstationary streams.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it named the specific baselines used for comparison and briefly stated the two pricing schemes.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and will revise the manuscript to strengthen the empirical support for our claims.

read point-by-point responses

  1. Referee: [Abstract / OVBAL description] Abstract and OVBAL description: the claim that OVBAL produces a more favorable long-run error-cost trade-off rests on the D-optimality criterion with exponential forgetting serving as an accurate estimate of marginal utility under concept drift. The manuscript uses this estimator for purchase decisions without direct validation against realized error reduction on the solar dataset or synthetic streams; if the estimates are systematically biased (e.g., due to mismatch between forgetting factor and drift rate), the cost-aware decisions lose their justification relative to baselines.

    Authors: We agree that direct validation of the D-optimality utility estimates against realized error reduction would provide stronger justification for the purchase decisions. The current manuscript relies on the theoretical motivation of the criterion without this empirical check on the solar or synthetic data. In the revised version we will add a new analysis (e.g., a figure or table) that correlates the estimated marginal utilities with observed error reductions under the concept drift present in each dataset. revision: yes

  2. Referee: [Experiments] Experiments section: results are reported showing favorable trade-offs under both pricing schemes, yet no analysis is provided of how the exponential forgetting factor was chosen or whether it was tuned to match observed drift rates on the solar task. This is load-bearing for the adaptation claim, as an arbitrary or post-hoc choice would weaken the assertion that the rule reliably adapts to nonstationary streams.

    Authors: We acknowledge that the manuscript lacks an explicit description or sensitivity analysis for the choice of the exponential forgetting factor. This information is needed to support the claim of reliable adaptation to nonstationary streams. The revised manuscript will include a paragraph in the experimental setup detailing the selection procedure (including any grid search or validation approach used) and its relation to observed drift rates on the solar task. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; derivation uses independent D-optimality estimator validated on external data

full rationale

The paper introduces QueryMarket and OVBAL, which estimates marginal utility via D-optimality with exponential forgetting and applies it to cost-aware selection under rolling budgets. The central claim of improved error-cost trade-off is supported by experiments on synthetic streams and a real-world solar forecasting dataset, comparing against baselines under two pricing schemes. No equations, self-citations, or steps in the abstract or described framework reduce the utility estimator or decision rule to a fitted parameter defined by the evaluation data itself, nor to any self-referential construction. The method is self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Abstract-only review; free parameters such as the forgetting factor, budget reset interval, and D-optimality scaling are not enumerated. No invented physical entities. Axioms are standard online-learning assumptions (concept drift exists, utility can be estimated from variance reduction) plus the domain assumption that label prices are exogenous.

assumptions (1)
  • domain assumption D-optimality criterion with exponential forgetting accurately ranks sample utility under concept drift
    Invoked to justify the marginal-utility estimate that drives purchase decisions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QueryMarket: Cost-Aware Online Active Learning in Data Markets." pith.science (2026). https://pith.science/paper/U5CJGHEX

@misc{pith2026260617805,
  author       = {Pith},
  title        = {Pith review of: QueryMarket: Cost-Aware Online Active Learning in Data Markets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5CJGHEX}},
  note         = {Machine review of arXiv:2606.17805}
}
read the original abstract

Data acquisition is a major bottleneck for learning in real-time streams: analysts must decide on the fly which labels to purchase while respecting a rolling budget. However, existing online active learning rarely unifies pricing, information gain, and rolling budget constraints under concept drift. We introduce QueryMarket, a market-inspired framework that queries each incoming data point based on its estimated utility to the model and its price. Within this framework, we propose OVBAL (online variance-based active learning), which integrates data pricing with information-driven selection by estimating each sample's marginal utility via a D-optimality criterion with exponential forgetting and executing cost-aware purchases under rolling budget constraints. OVBAL yields a simple, fully online decision rule that adapts to nonstationary streams and heterogeneous label costs. Experiments on synthetic data and a real-world solar power generation forecasting task show that OVBAL is particularly effective under seller-centric pricing and yields a more favorable long-run error-cost trade-off in the real-world task under both pricing schemes.

Figures

Figures reproduced from arXiv: 2606.17805 by the authors.

Figure 1
Figure 1. QueryMarket architecture. At time t, the analyst applies the OVBAL rule to decide whether to query a label from a seller. Queried labels are purchased at price pt (red dashed arrow) and used to update the forecasting model for downstream tasks. Blue blocks correspond to the buyer (analyst) and the grey block to the seller; black arrows indicate the data flow. purchased under a rolling budget. At each discrete time s… view at source ↗
Figure 2
Figure 2. Synthetic data: running prequential MSE versus time step under buyer (BC) and seller (SC) pricing. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Synthetic data: MSE versus cumulative spend (equal-budget comparison) under buyer (BC) and seller (SC) pricing. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Solar forecasting: running MSE over time under buyer (BC) and seller (SC) pricing. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Solar forecasting: equal-budget MSE versus cumulative spend under buyer (BC) and seller (SC) pricing. Numbers in parentheses denote the final [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 3 canonical work pages

  1. [1]

    Online learning and online convex optimization,

    S. Shalev-Shwartz, “Online learning and online convex optimization,” Foundations and Trends® in Machine Learning, vol. 4, no. 2, pp. 107– 194, 2012

    2012

  2. [2]

    Online learning: A compre- hensive survey,

    S. C. Hoi, D. Sahoo, J. Lu, and P. Zhao, “Online learning: A compre- hensive survey,”Neurocomputing, vol. 459, pp. 249–289, 2021

    2021

  3. [3]

    Active learning for data streams: a survey,

    D. Cacciarelli and M. Kulahci, “Active learning for data streams: a survey,”Machine Learning, vol. 113, no. 1, pp. 185–239, 2024

    2024

  4. [4]

    Active learning with statistical models,

    D. Cohn, Z. Ghahramani, and M. Jordan, “Active learning with statistical models,”Advances in neural information processing systems, vol. 7, 1994

    1994

  5. [5]

    Active learning literature survey,

    B. Settles, “Active learning literature survey,” University of Wisconsin– Madison, Tech. Rep., 2009

    2009

  6. [6]

    How to purchase labels? a cost-effective approach using active learning markets,

    X. Huang and P. Pinson, “How to purchase labels? a cost-effective approach using active learning markets,”INFORMS Journal on Data Science, 2026

    2026

  7. [7]

    Regression markets and applica- tion to energy forecasting,

    P. Pinson, L. Han, and J. Kazempour, “Regression markets and applica- tion to energy forecasting,”TOP, vol. 30, no. 3, pp. 533–573, 2022

    2022

  8. [8]

    Online Learning: A Modern Introduction Using Convex Optimization

    F. Orabona, “A modern introduction to online learning,” 2025. [Online]. Available: https://arxiv.org/abs/1912.13213

    work page Pith review arXiv 2025

Show all 36 references
  1. [9]

    Budgeted stream-based active learning via adaptive submodular maximization,

    K. Fujii and H. Kashima, “Budgeted stream-based active learning via adaptive submodular maximization,”Advances in Neural Information Processing Systems, vol. 29, 2016

    2016

  2. [10]

    Online active learning for soft sensor development using semi-supervised autoencoders,

    D. Cacciarelli, M. Kulahci, and J. Tyssedal, “Online active learning for soft sensor development using semi-supervised autoencoders,”arXiv preprint arXiv:2212.13067, 2022

    2022

  3. [11]

    The true sample complexity of active learning,

    M.-F. Balcan, S. Hanneke, and J. W. Vaughan, “The true sample complexity of active learning,”Machine learning, vol. 80, no. 2, pp. 111–139, 2010

    2010

  4. [12]

    A general agnostic active learning algorithm,

    S. Dasgupta, D. J. Hsu, and C. Monteleoni, “A general agnostic active learning algorithm,”Advances in neural information processing systems, vol. 20, 2007

    2007

  5. [13]

    Composite active learning: Towards multi-domain active learning with theoretical guaran- tees,

    G.-Y . Hao, H. Huang, H. Wang, J. Gao, and H. Wang, “Composite active learning: Towards multi-domain active learning with theoretical guaran- tees,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 11, 2024, pp. 12 286–12 294

    2024

  6. [14]

    Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment,

    C. L ¨uth, T. Bungert, L. Klein, and P. Jaeger, “Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment,”Advances in Neural Information Processing Systems, vol. 36, pp. 9789–9836, 2023

    2023

  7. [15]

    Active classification with few queries under misspecification,

    V . Kontonis, M. Ma, and C. Tzamos, “Active classification with few queries under misspecification,”Advances in Neural Information Pro- cessing Systems, vol. 37, pp. 51 684–51 707, 2024

    2024

  8. [16]

    Active regression by stratification,

    S. Sabato and R. Munos, “Active regression by stratification,”Advances in Neural Information Processing Systems, vol. 27, 2014

    2014

  9. [17]

    Active learning in approximately linear regression based on conditional expectation of generalization error

    M. Sugiyama and G. Ridgeway, “Active learning in approximately linear regression based on conditional expectation of generalization error.” Journal of Machine Learning Research, vol. 7, no. 1, 2006

    2006

  10. [18]

    Active learning literature survey,

    B. Settles, “Active learning literature survey,” University of Wisconsin– Madison, Computer Sciences Technical Report 1648, 2009

    2009

  11. [19]

    Improving generalization with active learning,

    D. Cohn, L. Atlas, and R. Ladner, “Improving generalization with active learning,”Machine learning, vol. 15, no. 2, pp. 201–221, 1994

    1994

  12. [20]

    A survey on deep active learning: Recent advances and new frontiers,

    D. Li, Z. Wang, Y . Chen, R. Jiang, W. Ding, and M. Okumura, “A survey on deep active learning: Recent advances and new frontiers,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 4, pp. 5879–5899, 2024

    2024

  13. [21]

    Budgeted online active learning with expert advice and episodic priors,

    K. Goebel, W. Solow, P. Pesantez-Cabrera, M. Keller, and A. Fern, “Budgeted online active learning with expert advice and episodic priors,” arXiv preprint arXiv:2506.03307, 2025

    2025

  14. [22]

    Streaming active learning with deep neural networks,

    A. Saran, S. Yousefi, A. Krishnamurthy, J. Langford, and J. T. Ash, “Streaming active learning with deep neural networks,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 30 005–30 021

    2023

  15. [23]

    Online active linear regression via thresholding,

    C. Riquelme, R. Johari, and B. Zhang, “Online active linear regression via thresholding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017

    2017

  16. [24]

    Stream-based active learning with linear models,

    D. Cacciarelli, M. Kulahci, and J. S. Tyssedal, “Stream-based active learning with linear models,”Knowledge-Based Systems, vol. 254, p. 109664, 2022

    2022

  17. [25]

    Agarwal, M

    A. Agarwal, M. Dahleh, and T. Sarkar, “A marketplace for data: An 10 0 10 20 30 40 Cumulative spend (£) 0 1 2 3 4 Running MSE (kWh2) OVBAL (£45.3) Greedy (£46.3) RS (£46.2) (a) Buyer pricing 0 10 20 30 40 Cumulative spend (£) 0.0 0.5 1.0 1.5 2.0 2.5 Running MSE (kWh2) OVBAL (£...

    2019

  18. [26]

    Selling privacy at auction,

    A. Ghosh and A. Roth, “Selling privacy at auction,” inProceedings of the 12th ACM conference on Electronic commerce, 2011, pp. 199–208

    2011

  19. [27]

    Strategic coalition for data pricing in iot data markets,

    S. R. Pandey, P. Pinson, and P. Popovski, “Strategic coalition for data pricing in iot data markets,”IEEE Internet of Things Journal, vol. 11, no. 4, pp. 6454–6468, 2023

    2023

  20. [28]

    Fundamentals of task-agnostic data valuation,

    M. M. Amiri, F. Berdoz, and R. Raskar, “Fundamentals of task-agnostic data valuation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, 2023, pp. 9226–9234

    2023

  21. [29]

    Information value theory,

    R. A. Howard, “Information value theory,”IEEE Transactions on systems science and cybernetics, vol. 2, no. 1, pp. 22–26, 2007

    2007

  22. [30]

    Data shapley: Equitable valuation of data for machine learning,

    A. Ghorbani and J. Zou, “Data shapley: Equitable valuation of data for machine learning,” inInternational conference on machine learning. PMLR, 2019, pp. 2242–2251

    2019

  23. [31]

    Data valuation using reinforcement learning,

    J. Yoon, S. Arik, and T. Pfister, “Data valuation using reinforcement learning,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 10 842–10 851

    2020

  24. [32]

    Second order techniques for learning time-series with structural breaks,

    T. Osogami, “Second order techniques for learning time-series with structural breaks,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 10, 2021, pp. 9259–9267

    2021

  25. [33]

    Online convex programming and generalized infinitesi- mal gradient ascent,

    M. Zinkevich, “Online convex programming and generalized infinitesi- mal gradient ascent,” inInternational Conference on Machine Learning (ICML), 2003, pp. 928–936

    2003

  26. [34]

    Logarithmic regret algorithms for online convex optimization,

    E. Hazan, A. Agarwal, and S. Kale, “Logarithmic regret algorithms for online convex optimization,”Machine Learning, vol. 69, no. 2, pp. 169– 192, 2007

    2007

  27. [35]

    Efficient second order online learning by sketching,

    H. Luo, A. Agarwal, N. Cesa-Bianchi, and J. Langford, “Efficient second order online learning by sketching,”Advances in Neural Information Processing Systems, vol. 29, 2016

    2016

  28. [36]

    Unisolar: An open dataset of photovoltaic solar energy generation in a large multi-campus university setting,

    S. Wimalaratne, D. Haputhanthri, S. Kahawala, G. Gamage, D. Ala- hakoon, and A. Jennings, “Unisolar: An open dataset of photovoltaic solar energy generation in a large multi-campus university setting,” in 2022 15th International Conference on Human System Interaction (HSI). IE...

    2022

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.