pith. sign in

arxiv: 2604.27944 · v1 · submitted 2026-04-30 · 💻 cs.LG · cs.CY· cs.GT· physics.ao-ph

Calibrating Attribution Proxies for Reward Allocation in Participatory Weather Sensing

Mark C. Ballandies , Michael T. C. Chiu , Claudio J. Tessone This is my paper

Pith reviewed 2026-05-07 06:17 UTC · model grok-4.3

classification 💻 cs.LG cs.CYcs.GTphysics.ao-ph
keywords gradient attributiondata valuationparticipatory sensingweather forecastingincentive mechanismsAI weather modelsreward allocationIoT sensors
0
0 comments X

The pith

Gradient-based attribution on differentiable AI weather models provides a validated proxy for valuing individual sensor data contributions to support reward allocation in participatory IoT networks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to establish that gradient-based attribution extracted from differentiable AI weather models can act as a practical signal for the marginal value of individual data points in large-scale weather sensing networks. This matters because existing valuation methods either overlook contribution quality or demand the full costly data assimilation infrastructure of operational meteorology, leaving participatory IoT networks without fair incentive mechanisms. By evaluating the approach on gridded GFS analysis inputs across more than 400 configurations, the work shows that attribution aligns with near-optimal sensor placement utility and supports monotonically faithful payments. At the same time, it demonstrates vulnerability to inflation by adversarial inputs, which can be detected only when external baseline data is available. These results position gradient attribution as a lower-cost alternative for model-informed reward systems.

Core claim

By applying gradient-based attribution to gridded GFS analysis inputs within differentiable AI weather models, the authors derive a candidate value signal for data contributions that achieves near-optimal utility for sensor placement decisions and supports monotonically faithful reward payments, as demonstrated through fidelity, calibration, cost, and gaming analyses across more than 400 configurations. The signal remains computationally tractable without full operational data assimilation yet can be inflated by adversarial inputs, with reliable detection requiring external baseline comparisons.

What carries the argument

Gradient-based attribution computed on differentiable AI weather models applied to gridded GFS analysis inputs, functioning as a proxy for the marginal contribution of each sensor datum to forecast quality.

If this is right

  • Attribution-guided placement of sensors reaches near-optimal network utility for weather forecasting.
  • Payments scaled directly from attribution scores remain monotonically faithful to each contribution.
  • The method avoids the infrastructure overhead of traditional adjoint-based valuation in meteorology.
  • Adversarial data inputs can systematically inflate attribution values and therefore payments.
  • External baseline data is required to detect and mitigate such inflation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The proxy could enable incentive programs for citizen-science weather networks at scales where full assimilation remains impractical.
  • Similar gradient techniques may extend to data valuation in other domains that already run differentiable simulation models, such as climate or air-quality forecasting.
  • Live deployment in actual participatory networks would expose calibration drift and participation dynamics not visible in offline configuration sweeps.

Load-bearing premise

That gradient attributions produced by AI weather models accurately reflect the true marginal impact of individual data contributions on forecast quality even without the complete data assimilation systems used in operational meteorology.

What would settle it

A side-by-side test in which specific sensor observations are added or removed from both an AI weather model and a full operational forecast system, followed by direct comparison of attribution-derived values against measured changes in forecast error metrics.

Figures

Figures reproduced from arXiv: 2604.27944 by Claudio J. Tessone, Mark C. Ballandies, Michael T. C. Chiu.

Figure 1
Figure 1. Figure 1: Spatial IG importance maps for Zurich (star marker) for 2 m temperature: FourCastNet (a) and SFNO (b), with distance (c) and uniform (d) baselines. view at source ↗
Figure 2
Figure 2. Figure 2: Global attribution fidelity: Spearman ρ between IG attribution and ablation utility across 30 configurations (2 models × 5 cities × 3 target variables). Bold values with ∗ are significant (p < 0.05, n = 24 variables); faded values are not. FCN fails systematically for t2m (all five cities non￾significant) while SFNO maintains ρ > 0.5 in 14 of 15 configurations. Per￾configuration bootstrap 95% CIs are in th… view at source ↗
Figure 3
Figure 3. Figure 3: Cardinal calibration and payment accuracy. view at source ↗
read the original abstract

Large-scale IoT weather sensing networks require incentive mechanisms to sustain participation, yet determining how much value individual data contributions bring to the network remains an open problem. Existing approaches address data quality but not data valuation; in operational meteorology, adjoint-based methods derive value from the forecast model itself but require full data assimilation infrastructure. We propose to utilise differentiable AI weather models to fill this gap and characterise gradient-based attribution on gridded GFS analysis inputs as a candidate value signal, evaluating fidelity, calibration, cost, and gaming vulnerability across more than 400 configurations. Attribution captures near-optimal sensor placement utility with monotonically faithful payments, but can be inflated by adversarial inputs, with detection requiring external baseline data. These findings establish gradient attribution as a computationally validated signal for model-informed reward allocation in participatory weather sensing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes gradient-based attribution on differentiable AI weather models as a proxy for valuing individual sensor contributions in large-scale participatory weather sensing networks. Using gridded GFS analysis data as inputs, the authors evaluate this approach across more than 400 configurations for fidelity to sensor placement utility, calibration for reward allocation, computational cost, and vulnerability to adversarial gaming. They conclude that the attribution method achieves near-optimal placement utility with monotonically increasing payments, while highlighting that adversarial inputs can inflate attributions, detectable only with external baseline data. This is presented as a scalable alternative to adjoint methods requiring full data assimilation infrastructure.

Significance. If the gradient attribution faithfully captures marginal forecast value, the work could significantly advance incentive design for IoT-based weather observation networks by providing a model-informed, computationally tractable valuation signal. It bridges machine learning attribution techniques with meteorological applications, potentially reducing reliance on complex operational systems. The identification of adversarial vulnerabilities and the need for external baselines adds practical insight for robust implementation. However, the significance is tempered by the absence of direct validation against traditional adjoint sensitivities in operational settings.

major comments (2)
  1. [Evaluation (across >400 configurations)] The evaluations of fidelity, calibration, and near-optimal placement utility are conducted entirely within the differentiable AI weather model framework (as described in the abstract and evaluation sections) without direct benchmarking against operational adjoint sensitivities on the same GFS inputs. This is load-bearing for the transferability claim, since operational meteorology derives data value via full data-assimilation pipelines incorporating background-error covariances, observation operators, and iterative minimization rather than standalone gradient attribution.
  2. [Abstract and Results] The abstract reports claims of fidelity and monotonicity over more than 400 configurations yet provides no visible error bars, confidence intervals, detailed exclusion criteria, or access to raw data. Without these, it is impossible to verify the statistical robustness of the 'near-optimal sensor placement utility' and 'monotonically faithful payments' assertions.
minor comments (2)
  1. [Abstract] The abstract could more explicitly name the differentiable AI weather model architecture and the precise preprocessing steps applied to the gridded GFS analysis inputs to improve reproducibility.
  2. [Methods] Notation for the attribution scores and payment functions should be introduced with a clear table or equation early in the methods to avoid ambiguity when discussing monotonicity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments, which help clarify the scope and presentation of our work. We address each major comment below and have revised the manuscript accordingly where feasible.

read point-by-point responses

  1. Referee: [Evaluation (across >400 configurations)] The evaluations of fidelity, calibration, and near-optimal placement utility are conducted entirely within the differentiable AI weather model framework (as described in the abstract and evaluation sections) without direct benchmarking against operational adjoint sensitivities on the same GFS inputs. This is load-bearing for the transferability claim, since operational meteorology derives data value via full data-assimilation pipelines incorporating background-error covariances, observation operators, and iterative minimization rather than standalone gradient attribution.

    Authors: We agree that the absence of direct benchmarking against operational adjoint sensitivities limits the strength of transferability claims to full data-assimilation systems. Such benchmarking is not possible in the current study because operational adjoint tools and full DA pipelines are not publicly accessible for controlled experiments on identical GFS inputs. Our evaluations instead demonstrate that gradient attribution serves as a near-optimal and calibrated proxy within the differentiable AI model, which itself approximates forecast sensitivities. We have added a dedicated limitations subsection in the discussion that explicitly states this scope, qualifies the transferability claim, and outlines the requirements for future operational validation. The abstract and introduction have also been revised to describe the method as a computationally tractable alternative rather than a direct substitute. revision: partial

  2. Referee: [Abstract and Results] The abstract reports claims of fidelity and monotonicity over more than 400 configurations yet provides no visible error bars, confidence intervals, detailed exclusion criteria, or access to raw data. Without these, it is impossible to verify the statistical robustness of the 'near-optimal sensor placement utility' and 'monotonically faithful payments' assertions.

    Authors: We accept that the original presentation lacked sufficient statistical detail for independent verification. The revised manuscript now includes error bars (representing standard deviation across configurations) and 95% confidence intervals in the key results figures and tables. We have added explicit exclusion criteria in the evaluation section (configurations with sensors outside the valid domain or with numerical instabilities in gradient computation were removed, accounting for approximately 8% of runs). A public repository link has been added to the abstract and data-availability statement, providing the full set of raw attribution values, placement utilities, and analysis scripts for the 400+ configurations. revision: yes

Circularity Check

0 steps flagged

No significant circularity; evaluation uses external GFS data and independent utility benchmarks

full rationale

The paper proposes gradient-based attribution on differentiable AI weather models as a candidate value signal for sensor data contributions, then evaluates its fidelity, calibration, cost, and vulnerability across >400 configurations against near-optimal sensor placement utility and monotonic payment properties. These evaluations rely on external GFS analysis inputs and direct comparisons to placement utility rather than any self-referential fitting or redefinition. No load-bearing step reduces a claimed result to a fitted parameter or prior self-citation by construction; the central proxy claim is tested against independent external baselines instead of being asserted tautologically. This is the expected non-finding for an empirical calibration study that does not derive its performance metrics from its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Based solely on the abstract; no explicit free parameters, axioms, or invented entities are identifiable. The approach implicitly assumes that differentiable AI models can substitute for full assimilation systems, but this is not formalized as an axiom in the provided text.

pith-pipeline@v0.9.0 · 5448 in / 1246 out tokens · 90625 ms · 2026-05-07T06:17:28.077182+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages

  1. [1]

    From do-it-yourself (diy) to do-it-together (dit): Reflections on designing a citizen-driven air quality monitoring framework in taiwan,

    S. Mahajan, C.-H. Luo, D.-Y . Wu, and L.-J. Chen, “From do-it-yourself (diy) to do-it-together (dit): Reflections on designing a citizen-driven air quality monitoring framework in taiwan,”Sustainable Cities and Society, vol. 66, p. 102628, 2021

    work page 2021

  2. [2]

    Can the crowdsourcing data paradigm take atmospheric science to a new level? A case study of the urban heat island of London quantified using Netatmo weather stations,

    L. Chapman, C. Bell, and S. Bell, “Can the crowdsourcing data paradigm take atmospheric science to a new level? A case study of the urban heat island of London quantified using Netatmo weather stations,” International Journal of Climatology, vol. 37, no. 9, pp. 3597–3605, 2017

    work page 2017

  3. [3]

    Crowd- sourcing air temperature from citizen weather stations for urban climate research,

    F. Meier, D. Fenner, T. Grassmann, M. Otto, and D. Scherer, “Crowd- sourcing air temperature from citizen weather stations for urban climate research,”Urban Climate, vol. 19, pp. 170–191, 2017

    work page 2017

  4. [4]

    How useful are crowdsourced air temperature observations? An assessment of Netatmo stations and quality control schemes over the United Kingdom,

    J. Coney, B. Pickering, D. Dufton, M. Lukach, B. Brooks, and R. R. Neely, III, “How useful are crowdsourced air temperature observations? An assessment of Netatmo stations and quality control schemes over the United Kingdom,”Meteorological Applications, vol. 29, no. 3, p. e2075, 2022

    work page 2022

  5. [5]

    Crowdsensing-based trans- portation services—an analysis from business model and sustainabil- ity viewpoints,

    M. Heiskala, J.-P. Jokinen, and M. Tinnil ¨a, “Crowdsensing-based trans- portation services—an analysis from business model and sustainabil- ity viewpoints,”Research in Transportation Business & Management, vol. 18, pp. 38–48, 2016

    work page 2016

  6. [6]

    Big data management in partici- patory sensing: Issues, trends and future directions,

    A. Karim, A. Siddiqa, Z. Safdar, M. Razzaq, S. A. Gillani, H. Tahir, S. Kiran, E. Ahmed, and M. Imran, “Big data management in partici- patory sensing: Issues, trends and future directions,”Future Generation Computer Systems, vol. 107, pp. 942–955, 2020

    work page 2020

  7. [7]

    Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing,

    D. Yang, G. Xue, X. Fang, and J. Tang, “Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing,” inProceedings of the 18th Annual International Conference on Mobile Computing and Networking (MobiCom). Istanbul, Turkey: ACM, 2012, pp. 173–184

    work page 2012

  8. [8]

    Incentives for mobile crowd sensing: A survey,

    X. Zhang, Z. Yang, W. Sun, Y . Liu, S. Tang, K. Xing, and X. Mao, “Incentives for mobile crowd sensing: A survey,”IEEE Communications Surveys & Tutorials, vol. 18, no. 1, pp. 54–67, 2016

    work page 2016

  9. [9]

    Incentive mechanisms for par- ticipatory sensing: Survey and research challenges,

    F. Restuccia, S. K. Das, and J. Payton, “Incentive mechanisms for par- ticipatory sensing: Survey and research challenges,”ACM Transactions on Sensor Networks, vol. 12, no. 2, pp. 13:1–13:40, 2016

    work page 2016

  10. [10]

    DePIN: A framework for token-incentivized participatory sensing,

    M. T. C. Chiu, S. Mahajan, M. C. Ballandies, and U. V . Kalabi ´c, “DePIN: A framework for token-incentivized participatory sensing,” in 2025 IEEE International Conference on Blockchain and Cryptocurrency (ICBC). Pisa, Italy: IEEE, 2025, pp. 1–7

    work page 2025

  11. [11]

    To incentivize or not: Impact of blockchain-based cryptoeconomic tokens on human information sharing behavior,

    M. C. Ballandies, “To incentivize or not: Impact of blockchain-based cryptoeconomic tokens on human information sharing behavior,”IEEE Access, vol. 10, pp. 74 111–74 130, 2022

    work page 2022

  12. [12]

    Quality control for crowdsourced personal weather stations to enable operational rainfall monitoring,

    L. W. de V os, H. Leijnse, A. Overeem, and R. Uijlenhoet, “Quality control for crowdsourced personal weather stations to enable operational rainfall monitoring,”Geophysical Research Letters, vol. 46, no. 15, pp. 8820–8829, 2019

    work page 2019

  13. [13]

    CrowdQC+—a quality-control for crowdsourced air-temperature ob- servations enabling world-wide urban climate applications,

    D. Fenner, B. Bechtel, M. Demuzere, J. Kittner, and F. Meier, “CrowdQC+—a quality-control for crowdsourced air-temperature ob- servations enabling world-wide urban climate applications,”Frontiers in Environmental Science, vol. 9, p. 720747, 2021

    work page 2021

  14. [14]

    Strategic coalition for data pricing in IoT data markets,

    S. R. Pandey, P. Pinson, and P. Popovski, “Strategic coalition for data pricing in IoT data markets,”IEEE Internet of Things Journal, vol. 11, no. 4, pp. 6454–6468, 2024

    work page 2024

  15. [15]

    Fair allocation of sensor measurements using Shapley value,

    S.-S. Byun, H. Moussavinik, and I. Balasingham, “Fair allocation of sensor measurements using Shapley value,” in2009 IEEE 34th Conference on Local Computer Networks, 2009, pp. 459–466

    work page 2009

  16. [16]

    Data Shapley: Equitable valuation of data for machine learning,

    A. Ghorbani and J. Zou, “Data Shapley: Equitable valuation of data for machine learning,” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, CA: PMLR, 09–15 Jun 2019, pp. 2242–2251

    work page 2019

  17. [17]

    Monitoring the observation impact on the short-range forecast,

    C. Cardinali, “Monitoring the observation impact on the short-range forecast,”Quarterly Journal of the Royal Meteorological Society, vol. 135, no. 638, pp. 239–250, 2009

    work page 2009

  18. [18]

    Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system,

    R. H. Langland and N. L. Baker, “Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system,” Tellus A, vol. 56, no. 3, pp. 189–201, 2004

    work page 2004

  19. [19]

    Reward mechanism,

    WeatherXM, “Reward mechanism,” https://docs.weatherxm.com/rewards/reward- mechanism, 2024, accessed: 2025-03-14

    work page 2024

  20. [20]

    Data quality,

    ——, “Data quality,” https://docs.weatherxm.com/rewards/data-quality, 2024, accessed: 2025-03-14

    work page 2024

  21. [21]

    Towards efficient data valuation based on the Shapley value,

    R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. G ¨urel, B. Li, C. Zhang, D. Song, and C. J. Spanos, “Towards efficient data valuation based on the Shapley value,” inProceedings of the Twenty- Second International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, K. Chaudhuri and M. Sugiyama, Eds., ...

    work page 2019

  22. [22]

    Beta Shapley: a unified and noise-reduced data valuation framework for machine learning,

    Y . Kwon and J. Zou, “Beta Shapley: a unified and noise-reduced data valuation framework for machine learning,” inProceedings of The 25th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, G. Camps-Valls, F. J. R. Ruiz, and I. Valera, Eds., vol. 151. PMLR, 28–30 Mar 2022, pp. 8780–8802

    work page 2022

  23. [23]

    Axiomatic attribution for deep networks,

    M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” inProceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, D. Precup and Y . W. Teh, Eds., vol. 70. Sydney, Australia: PMLR, 06–11 Aug 2017, pp. 3319–3328

    work page 2017

  24. [24]

    The THOR- PEX observation impact intercomparison experiment,

    R. Gelaro, R. H. Langland, S. Pellerin, and R. Todling, “The THOR- PEX observation impact intercomparison experiment,”Monthly Weather Review, vol. 138, no. 11, pp. 4009–4025, 2010

    work page 2010

  25. [25]

    The value of surface-based meteorological observation data,

    D. Kull, L. P. Riishojgaard, J. Eyre, and R. A. Varley, “The value of surface-based meteorological observation data,” World Bank Group, Washington, D.C., Tech. Rep., 2021. [Online]. Available: https://openknowledge.worldbank.org/handle/10986/35178

    work page 2021

  26. [26]

    FourCastNet: Acceler- IEEE INTERNET OF THINGS JOURNAL 13 ating global high-resolution weather forecasting using adaptive Fourier neural operators,

    T. Kurth, S. Subramanian, P. Harrington, J. Pathak, M. Mardani, D. Hall, A. Miele, K. Kashinath, and A. Anandkumar, “FourCastNet: Acceler- IEEE INTERNET OF THINGS JOURNAL 13 ating global high-resolution weather forecasting using adaptive Fourier neural operators,” inProceedings of the Platform for Advanced Scientific Computing Conference (PASC ’23), 2023

    work page 2023

  27. [27]

    Spherical Fourier neural operators: Learning stable dynamics on the sphere,

    B. Bonev, T. Kurth, C. Hundt, J. Pathak, M. Baust, K. Kashinath, and A. Anandkumar, “Spherical Fourier neural operators: Learning stable dynamics on the sphere,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, ...

    work page 2023

  28. [28]

    Learning skillful medium-range global weather forecasting,

    R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. For- tunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Huet al., “Learning skillful medium-range global weather forecasting,”Science, vol. 382, no. 6677, pp. 1416–1421, 2023

    work page 2023

  29. [29]

    Accurate medium-range global weather forecasting with 3D neural networks,

    K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, “Accurate medium-range global weather forecasting with 3D neural networks,” Nature, vol. 619, no. 7970, pp. 533–538, 2023

    work page 2023

  30. [30]

    A foundation model for the Earth system,

    C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Allen, J. Brand- stetteret al., “A foundation model for the Earth system,”Nature, vol. 641, no. 8065, pp. 1180–1187, 2025

    work page 2025

  31. [31]

    Making the black box more transparent: Understanding the physical implications of machine learn- ing,

    A. McGovern, R. Lagerquist, D. J. Gagne II, G. E. Jergensen, K. L. Elmore, C. R. Homeyer, and T. Smith, “Making the black box more transparent: Understanding the physical implications of machine learn- ing,”Bulletin of the American Meteorological Society, vol. 100, no. 11, pp. 2175–2199, 2019

    work page 2019

  32. [32]

    Physically interpretable neural networks for the geosciences: Applications to earth system variability,

    B. A. Toms, E. A. Barnes, and I. Ebert-Uphoff, “Physically interpretable neural networks for the geosciences: Applications to earth system variability,”Journal of Advances in Modeling Earth Systems, vol. 12, no. 9, p. e2019MS002002, 2020

    work page 2020

  33. [33]

    Are AI weather models learning atmo- spheric physics? A sensitivity analysis of cyclone Xynthia,

    J. Ba ˜no-Medina, A. Sengupta, J. D. Doyle, C. A. Reynolds, D. Watson- Parris, and L. Delle Monache, “Are AI weather models learning atmo- spheric physics? A sensitivity analysis of cyclone Xynthia,”npj Climate and Atmospheric Science, vol. 8, p. 92, 2025

    work page 2025

  34. [34]

    The well-calibrated Bayesian,

    A. P. Dawid, “The well-calibrated Bayesian,”Journal of the American Statistical Association, vol. 77, no. 379, pp. 605–610, 1982

    work page 1982

  35. [35]

    Deep inside convolutional networks: Visualising image classification models and saliency maps,

    K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” inInternational Conference on Learning Representations (ICLR) Work- shop, Banff, Canada, 2014

    work page 2014

  36. [36]

    Agent-based modelling of ethereum consensus,

    B. Kraner, N. Vallarano, C. Schwarz-Schilling, and C. J. Tessone, “Agent-based modelling of ethereum consensus,” in2023 IEEE Interna- tional Conference on Blockchain and Cryptocurrency (ICBC). IEEE, 2023, pp. 1–8

    work page 2023

  37. [37]

    Burn-and-mint tokenomics: Deflation and strategic incentives,

    U. V . Kalabi ´c, M. C. Ballandies, K. Paruch, H. H. Nax, and T. Nigg, “Burn-and-mint tokenomics: Deflation and strategic incentives,” in2023 IEEE 9th World Forum on Internet of Things (WF-IoT). IEEE, 2023, pp. 1–6

    work page 2023