Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Data Reliability Scoring

T0 review · 2 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper tries to establish that a single number—the Gram determinant—can rank datasets by their reliability relative to hidden ground truth, using only auxiliary observations from an unknown experiment, and that this ranking is independen

desk verdict Worth a look for the preservation theorems and the no-ground-truth framework, but the 'unique up to scaling' claim in the abstract overreaches—Prop 4.4 needs an extra homogeneity assumption that isn't advertised. read the letter →

arxiv 2510.17085 v2 pith:KQZH3ISQ submitted 2025-10-20 cs.LG cs.GTstat.ML

Yiling Chen , Shi Feng , Paul Kattuman , Fang-Yi Yu This is my paper
classification cs.LGcs.GTstat.ML
keywords datareliabilityscoringGramdeterminantscoreexperimentagnosticismgroundtruth-freeevaluationBlackwellorderingHammingkernelmethodsquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Without ground truth, how can one tell which of two reported datasets is more reliable? The paper's answer: if you also observe outcomes of an unknown experiment that depends on the truth (e.g., blood tests for disease status, embeddings for image labels), you can rank datasets by the volume spanned by their class-conditional observation distributions. The proposed Gram determinant score computes this volume, and the paper shows it preserves three natural ground-truth-based orderings—exact match, Blackwell dominance, and approximate Hamming/distance—whenever the experiment is linearly independent and misreports are not too severe. It further proves the score is, up to scaling, the unique reliability score whose ranking does not depend on the experiment. If correct, this gives a task-agnostic, ground-truth-free way to audit data quality in settings where reports are noisy or strategically distorted.

What carries the argument

Gram determinant score Γ = det((P Q)^T (P Q)), where P Q is the joint distribution matrix of observations and reported labels; geometrically it is the squared volume of the parallelepiped spanned by the observation-distribution columns grouped by reported label. Its load-bearing property is the multiplicative identity Γ = det(P^T P) det(Q)^2, which separates the unknown experiment P from the misreport matrix Q and makes the ranking experiment-agnostic; the determinant is also what gives finite-sample estimators via plug-in and stratified matching.

What would settle it

Compute, for a linearly independent P and two diagonally dominant misreport matrices Q and Q' with L-balanced truth and Hamming error below 1/(64 L^2 d^2), the Gram determinants det(Q^T Q) and det((Q')^T Q'); if the dataset with smaller Hamming distance to the truth has the smaller determinant, Theorem 4.3(3) is false. Alternatively, exhibit P with linearly dependent columns and two datasets whose reliability ranking flips across two experiments—that would violate the claimed necessity of P_indep in Proposition 3.1.

Watch

Extended reading notes

Core claim

The central discovery is that the determinant factorizes: for a misreport matrix Q and an experiment P, the Gram matrix of reports is Q^T P^T P Q, so det(Q^T P^T P Q) = det(P^T P) det(Q)^2. Since det(P^T P) is positive exactly when P has linearly independent columns, the score's ordering over datasets depends on Q alone—the misreport structure—and not on the experiment. The paper claims this multiplicative decoupling is what lets the Gram determinant preserve exact-match, Blackwell-dominant, and approximate Hamming (or general distance) reliability orderings under stated conditions, and that it underlies the uniqueness result: any continuous, experiment-agnostic reliability score that is hom

Load-bearing premise

The guarantees require that the unknown experiment's columns be linearly independent (no two true labels induce the same observation distribution) and, for the approximate Hamming/dist result, that the truth is L-balanced and corruption is small (Hamming error below 1/(64 L^2 d^2)); the paper's impossibility results show these conditions are essentially necessary, and in real deployments they cannot be checked—the employment-data analysis concedes its conditional-independence

Editorial extensions

If this is right

  • Datasets can be reliability-ranked without ground truth whenever auxiliary observations come from a linearly independent experiment; this includes exact-match and Blackwell-dominant comparisons exactly, and Hamming/dist comparisons up to a constant factor under balance and low-corruption conditions.
  • The ranking is experiment-agnostic: any two datasets are ordered the same way regardless of which (invertible) observation experiment generated the auxiliary data, and the Gram determinant is essentially the only score with this property.
  • A plug-in estimator built from empirical matching frequencies asymptotically preserves all the same orderings, so the score is computable in the detail-free setting where neither P nor Q is known.
  • Kernel extensions (e.g., radial-basis or feature-map kernels) extend the guarantees to continuous observation spaces such as image embeddings, with the same ordering preservation.
  • The impossibility results imply these conditions are nearly tight: if two labels induce identical observation distributions, no score can preserve the orderings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The uniqueness result suggests that any experiment-agnostic reliability score in this setting must be a function of the determinant, so the Gram determinant is not one option among many but the canonical one; other singular-value-based scores (top-k volume, maximal correlation) are not experiment-agnostic and may rank datasets inconsistently across experiments.
  • Because linear independence of the experiment is unverifiable in practice, a practical diagnostic would be the condition number or smallest eigenvalue of the empirical Gram matrix; near-zero values signal that the score's guarantees do not apply.
  • The paper's own caveat that the conditional-independence assumption in the employment-data study cannot be validated suggests that in real deployments the score should be used as an indicative signal, not a certified measure—unless the experiment is known or genuinely independent of reporting.
  • A testable extension: on benchmark datasets with known ground-truth corruption levels, compare the Gram-determinant ranking against Hamming distance at fixed finite N; the theory predicts exact agreement only when the corruption level is below 1/(64 L^2 d^2), so deviations above that threshold would delimit the score's practical envelope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces a ground-truth-free reliability scoring framework for reported datasets, given auxiliary observations y generated by an unknown statistical experiment P. It defines exact-match, Blackwell-dominant, and dist/Hamming reliability orderings relative to unobserved truth x, and proposes the Gram determinant score Γ(PQ)=det((PQ)^T PQ), which factorizes as det(P^T P)det(Q)^2. The main theoretical claims are: (i) preservation of the reliability orderings under linearly independent experiments and suitable misreport classes (Theorem 4.3); (ii) an experiment-agnostic invariance property, with a claimed uniqueness characterization up to scaling (Proposition 4.4); (iii) plug-in and stratified-matching estimators, plus kernelized extensions. Experiments on synthetic data, CIFAR-10 embeddings, and CES employment-data vintages illustrate the score's empirical behavior.

Significance. The geometric volume interpretation and the determinant factorization are elegant, and the main preservation proofs are largely self-contained. If the order-preservation theorems and a suitably qualified uniqueness theorem hold, the Gram determinant provides a practical, task-agnostic way to rank datasets by reliability without ground truth, with impossibility results delineating when no score can work. The kernel extensions and finite-sample estimators are useful contributions. However, two issues affect the central claims as stated: the exact-match preservation theorem is vacuous under the current definition of Q_nonperm, and the advertised 'unique up to scaling' result requires an extra homogeneity assumption that is not part of the definition of experiment agnosticism. Both are fixable but must be addressed before the results can be accepted as stated.

major comments (2)
  1. [Section 2.3 / Section 3 / Theorem 4.3(1)] Q_nonperm is defined as the set of misreport matrices Q whose associated Q_hatx|x is neither a permutation matrix nor the identity matrix. Hence a truthful report \hat x=x, for which Q_hatx|x=I, is excluded from Q_nonperm. Since the exact-match ordering only ranks the truthful report above non-truthful reports, no pair (\hat x,\hat x') with \hat x≻_EXACT \hat x' can have both associated misreport matrices in Q_nonperm. Thus Theorem 4.3(1) is vacuously true as stated. The proof of Lemma D.1 also uses Q=Q_x for the truthful report, which is not in Q_nonperm. The intended set appears to be the one excluding only non-identity permutations (equivalently, including the identity); this would also make the inclusion chain Q_L,δ⊂...⊂Q_reg⊂Q_nonperm in Section 3 correct. Please correct the definition and restate the theorem accordingly.
  2. [Section 4.1 / Proposition 4.4 / Appendix D.2] The abstract and introduction claim that the Gram determinant is 'up to scaling' the unique experiment-agnostic reliability score. The proof of Proposition 4.4 additionally assumes S(tQ)=c(t)S(Q) for all t>0 with continuous c; this homogeneity condition is not part of Definition 2.2 or of Eq. (6). Without it the uniqueness statement is false: S(Q)=log(1+det(Q^T Q)) is continuous and positive on GL_d, satisfies Eq. (6) for every P∈GL_d, but is not of the form α det(Q^T Q)^β and is not homogeneous. Moreover, even under the homogeneity assumption the theorem yields arbitrary β≠0, so the characterization is a one-parameter family of equivalent orderings rather than a unique scaling. Proposition 4.4 should be restated as a characterization of homogeneous experiment-agnostic scores, and the abstract/introduction should be softened accordingly.
minor comments (4)
  1. [Section 5.2] The text says 'the score increases monotonically with p'; given that p is the corruption probability and Fig. 4 plots score against p, this should be 'decreases monotonically with p', consistent with Experiment 1 and with the stated association between higher score and lower error.
  2. [Example 4.2] The displayed misreport matrix Q appears to have row sums 1/4, which is inconsistent with d=2 and uniform true labels (whose marginal frequencies should sum to 1/2). Please check the normalization or clarify the intended Q_x.
  3. [Section H.2] The authors candidly note that the conditional-independence assumption in the employment-data experiment cannot be validated and that the reported scores should be treated only as indicative. This limitation is important enough to state in the main experimental section rather than only in the appendix.
  4. [General] There are small typos: 'deter' should be 'defer' in Section 4.1, and the abstract contains 'experimen' for 'experiment'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Gram determinant score has no fitted parameters, the reliability orderings are defined externally, and the preservation proofs are direct. The noted uniqueness overclaim is a correctness concern, not circularity.

full rationale

The paper's central derivation is self-contained rather than circular. The Gram determinant score Γ(PQ)=det((PQ)^T PQ) is a fixed, parameter-free function of the joint distribution; no constant or function is fitted to data and then renamed as a prediction. The ground-truth orderings (exact match, Blackwell dominant, Hamming/dist) are defined in Section 2.3 from the misreport matrix Q alone, independently of the score. The preservation results in Theorem 4.3 follow from the multiplicative identity Γ(PQ)=det(P^T P) det(Q)^2, so the score inherits orderings that are monotone in det(Q)^2; the proofs (Lemmas D.1–D.3) are direct linear-algebra arguments, with the only cited external facts being standard Perron–Frobenius-type bounds and determinant perturbation inequalities. The experiment-agnosticism property Eq. (6) is also a direct consequence of the same multiplicativity. The uniqueness claim in Proposition 4.4 does contain an unadvertised extra homogeneity assumption S(tQ)=c(t)S(Q), and the advertised 'unique up to scaling' therefore overreaches (e.g., S(Q)=log(1+det(Q^T Q)) is a continuous experiment-agnostic score not of the stated form); however, this is a mathematical overclaim / correctness risk, not a circular reduction to the paper's own inputs. The self-citation [47] is used only to justify a standard symmetrization step in Section C and a Perron–Frobenius corollary in Lemma D.1; it is not load-bearing, and the paper explicitly credits Kong [28] for the determinant-mutual-information antecedent. The empirical sections compare the score to external corruption levels and Hamming/ℓ2 errors without fitting the score, so they are validation rather than circular prediction. No step in the derivation chain reduces to a fitted parameter or to a definition of the target result.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted in the score; experimental corruption-policy constants are data-generation choices, not part of the method. All substantive assumptions are explicit in the model and theorems, though the P_indep and Q_{L,δ} conditions are not verifiable in real applications.

assumptions (5)
  • domain assumption Observations y_n are generated independently as y_n ∼ P_{x_n} from an unknown column-stochastic experiment P.
    Section 2.1; if observations are dependent or the experiment varies across n, the joint-distribution representation P Q and the Gram-determinant calculations fail.
  • domain assumption Experiment P has linearly independent columns (P ∈ P_indep).
    Theorems 4.3 and 4.10 and Proposition 4.4 require it so that P^T P is positive definite; Proposition 3.1 shows with dependent columns no score can preserve the orderings.
  • domain assumption True data are L-balanced and the Hamming corruption is bounded by δ ≤ 1/(64L²d²) with Q diagonally dominant (Q ∈ Q_{L,δ}).
    Needed for Lemma D.3's determinant bounds; far more restrictive than the Q_dom impossibility class, so the paper's 'nearly tight' claim is not established.
  • domain assumption For Blackwell dominance, misreport matrices Q and Q' are invertible and row diagonally maximal (Q_reg).
    Definition 2.3; required for the Blackwell relation to be a strict partial order and for the proof of Lemma D.2.
  • ad hoc to paper For the uniqueness result, S is continuous, positive, and satisfies S(tQ)=c(t)S(Q) with continuous c.
    Proposition 4.4 imposes these regularity and homogeneity conditions to characterize S; they are not derived from the reliability problem itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Reliability Scoring." pith.science (2026). https://pith.science/paper/KQZH3ISQ

@misc{pith2026251017085,
  author       = {Pith},
  title        = {Pith review of: Data Reliability Scoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQZH3ISQ}},
  note         = {Machine review of arXiv:2510.17085}
}
read the original abstract

How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth-based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth-based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experimen -- a property we term experiment agnosticism. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.

Figures

Figures reproduced from arXiv: 2510.17085 by the authors.

Figure 1
Figure 1. Gram determinant scores and parallelepipeds. The Gram determinant score of true data, [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Gram determinant reliability score on categorical synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Matched rankings on categorical synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Gram determinant reliability for image–label experiments under six manipulation policies [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the Top-k volume, Max correlation, KL divergence, and χ 2 -mutual infor￾mation scores under different corruption levels and metrics. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Statistical alignment with human ratings does not imply strategic alignment: LLM-as-a-Judge is highly correlated but easily manipulated, while a new statement-level mutual-information metric is robust to manipulation.

Reference graph

Works this paper leans on

67 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    A general class of coefficients of divergence of one distribution from another.Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131–142, 1966

    Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another.Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131–142, 1966

    1966

  2. [2]

    Fundamentals of task- agnostic data valuation, 2022

    Mohammad Mohammadi Amiri, Frederic Berdoz, and Ramesh Raskar. Fundamentals of task- agnostic data valuation, 2022. URLhttps://arxiv.org/abs/2208.12354

    arXiv 2022

  3. [3]

    Theory of reproducing kernels.Transactions of the American mathemat- ical society, 68(3):337–404, 1950

    Nachman Aronszajn. Theory of reproducing kernels.Transactions of the American mathemat- ical society, 68(3):337–404, 1950. 16

    1950

  4. [4]

    Springer Science & Business Media, 2011

    Alain Berlinet and Christine Thomas-Agnan.Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011

    2011

  5. [5]

    Equivalent comparisons of experiments.The annals of mathematical statis- tics, pages 265–272, 1953

    David Blackwell. Equivalent comparisons of experiments.The annals of mathematical statis- tics, pages 265–272, 1953

    1953

  6. [6]

    Anomaly detection: A survey.ACM computing surveys (CSUR), 41(3):1–58, 2009

    Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey.ACM computing surveys (CSUR), 41(3):1–58, 2009

    2009

  7. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pages 1597–1607. PmLR, 2020

    2020

  8. [8]

    Informational substitutes

    Yiling Chen and Bo Waggoner. Informational substitutes. In2016 IEEE 57th Annual Sympo- sium on Foundations of Computer Science (FOCS), pages 239–247, 2016. doi: 10.1109/FOCS. 2016.33

    doi:10.1109/focs 2016

Show all 67 references
  1. [9]

    Dennis Cook and Sanford Weisberg

    R. Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression.Technometrics, 22(4):495–508, 1980. ISSN 00401706. URLhttp://www.jstor.org/stable/1268187

    1980

  2. [10]

    Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten.Magyer Tud

    Imre Csisz´ ar. Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten.Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85–108, 1964

    1964

  3. [11]

    A class of measures of informativity of observation channels.Periodica Mathe- matica Hungarica, 2(1-4):191–213, 1972

    Imre Csisz´ ar. A class of measures of informativity of observation channels.Periodica Mathe- matica Hungarica, 2(1-4):191–213, 1972

    1972

  4. [12]

    Crowdsourced judgement elicitation with endogenous proficiency

    Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In Daniel Schwabe, Virg ´ ılio A. F. Almeida, Hartmut Glaser, Ricardo Baeza- Yates, and Sue B. Moon, editors,22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro...

    2013

  5. [13]

    Vector-valued property elicitation

    Rafael Frongillo and Ian A Kash. Vector-valued property elicitation. InConference on Learning Theory, pages 710–727. PMLR, 2015

    2015

  6. [14]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. InInternational conference on machine learning, pages 2242–2251. PMLR, 2019

    2019

  7. [15]

    Academic Press, 2014

    Boris Vladimirovich Gnedenko, Yu K Belyayev, and Aleksandr Dmitrieviˇ c Solovyev.Mathe- matical methods of reliability theory. Academic Press, 2014

    2014

  8. [16]

    Strictly proper scoring rules, prediction, and estima- tion.Journal of the American statistical Association, 102(477):359–378, 2007

    Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estima- tion.Journal of the American statistical Association, 102(477):359–378, 2007

    2007

  9. [17]

    Error detecting and error correcting codes.The Bell system technical journal, 29(2):147–160, 1950

    Richard W Hamming. Error detecting and error correcting codes.The Bell system technical journal, 29(2):147–160, 1950

    1950

  10. [18]

    Cambridge university press, 2012

    Roger A Horn and Charles R Johnson.Matrix analysis. Cambridge university press, 2012. 17

    2012

  11. [19]

    Information value theory.IEEE Transactions on systems science and cybernetics, 2(1):22–26, 2007

    Ronald A Howard. Information value theory.IEEE Transactions on systems science and cybernetics, 2(1):22–26, 2007

    2007

  12. [20]

    When are the inverses of stochastic matrices also stochastic matrices? Mathe- matics Stack Exchange

    Angina Seng (https://math.stackexchange.com/users/436618/angina seng). When are the inverses of stochastic matrices also stochastic matrices? Mathe- matics Stack Exchange. URLhttps://math.stackexchange.com/q/2392982. URL:https://math.stackexchange.com/q/2392982 (version: 2017-08-14)

    2017

  13. [21]

    John Wiley & Sons, 2004

    Peter J Huber.Robust statistics, volume 523. John Wiley & Sons, 2004

    2004

  14. [22]

    Ilse C. F. Ipsen and Dean J. Lee. Determinant approximations, 2011. URLhttps://arxiv. org/abs/1105.0437

    2011 arXiv

  15. [23]

    Ilse C. F. Ipsen and Rizwana Rehman. Perturbation bounds for determinants and characteristic polynomials.SIAM Journal on Matrix Analysis and Applications, 30(2):762–776, 2008. doi: 10.1137/070704770. URLhttps://doi.org/10.1137/070704770

    2008 doi

  16. [24]

    The selection of prior distributions by formal rules

    Robert E Kass and Larry Wasserman. The selection of prior distributions by formal rules. Journal of the American statistical Association, 91(435):1343–1370, 1996

    1996

  17. [25]

    Kim and Eugene H

    Gene H. Kim and Eugene H. Spafford. The design and implementation of tripwire: a file system integrity checker. InProceedings of the 2nd ACM Conference on Computer and Communications Security, CCS ’94, page 18–29, New York, NY, USA, 1994. Associ- ation for Computing Machinery....

    1994

  18. [26]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 1885–

  19. [27]

    Dominantly truthful multi-task peer prediction with a constant number of tasks

    Yuqing Kong. Dominantly truthful multi-task peer prediction with a constant number of tasks. InProceedings of the fourteenth annual acm-siam symposium on discrete algorithms, pages 2398–2411. SIAM, 2020

    2020

  20. [28]

    Dominantly truthful peer prediction mechanisms with a finite number of tasks

    Yuqing Kong. Dominantly truthful peer prediction mechanisms with a finite number of tasks. J. ACM, 71(2), April 2024. ISSN 0004-5411. doi: 10.1145/3638239. URLhttps://doi.org/ 10.1145/3638239

    2024 doi

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  22. [30]

    On information and sufficiency.The annals of mathematical statistics, 22(1):79–86, 1951

    Solomon Kullback and Richard A Leibler. On information and sufficiency.The annals of mathematical statistics, 22(1):79–86, 1951

    1951

  23. [31]

    Eliciting properties of probability distributions

    Nicolas S Lambert, David M Pennock, and Yoav Shoham. Eliciting properties of probability distributions. InProceedings of the 9th ACM Conference on Electronic Commerce, pages 129– 138, 2008

    2008

  24. [32]

    Surrogate scoring rules and a dominant truth serum for information elicitation.CoRR, abs/1802.09158, 2018

    Yang Liu and Yiling Chen. Surrogate scoring rules and a dominant truth serum for information elicitation.CoRR, abs/1802.09158, 2018. URLhttp://arxiv.org/abs/1802.09158. 18

    2018 arXiv

  25. [33]

    Peer loss functions: Learning from noisy labels without knowing noise rates

    Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. InInternational conference on machine learning, pages 6226–6236. PMLR, 2020

    2020

  26. [34]

    John Wiley & Sons, 2021

    William Q Meeker, Luis A Escobar, and Francis G Pascual.Statistical methods for reliability data. John Wiley & Sons, 2021

    2021

  27. [35]

    Miller, P

    N. Miller, P. Resnick, and R. Zeckhauser. Eliciting informative feedback: The peer-prediction method.Management Science, pages 1359–1373, 2005

    2005

  28. [36]

    Markov processes and the h-theorem.Journal of the Physical Society of Japan, 18(3):328–331, 1963

    Tetsuzo Morimoto. Markov processes and the h-theorem.Journal of the Physical Society of Japan, 18(3):328–331, 1963. doi: 10.1143/JPSJ.18.328. URLhttps://doi.org/10.1143/ JPSJ.18.328

    1963 doi

  29. [37]

    Learning with noisy labels.Advances in neural information processing systems, 26, 2013

    Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels.Advances in neural information processing systems, 26, 2013

    2013

  30. [38]

    University of California, Berkeley, 1985

    Kent Harold Osband.Providing Incentives for Better Cost Forecasting (Prediction, Uncertainty Elicitation). University of California, Berkeley, 1985

    1985

  31. [39]

    Optimum bounds for the distributions of martingales in banach spaces.The Annals of Probability, pages 1679–1706, 1994

    Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces.The Annals of Probability, pages 1679–1706, 1994

    1994

  32. [40]

    A survey of data quality requirements that matter in ml development pipelines.ACM Journal of Data and Information Quality, 15 (2):1–39, 2023

    Maria Priestley, Fionnt´ an O’donnell, and Elena Simperl. A survey of data quality requirements that matter in ml development pipelines.ACM Journal of Data and Information Quality, 15 (2):1–39, 2023

    2023

  33. [41]

    Bureau of Labor Statistics

    U.S. Bureau of Labor Statistics. CES Vintage Data Information, March 2025. URLhttps: //www.bls.gov/web/empsit/cesvininfo.htm. Current Employment Statistics (CES). Last modified March 7, 2025

    2025

  34. [42]

    Federal Tax Deposits (Table IV)

    U.S. Department of the Treasury, Bureau of the Fiscal Service. Daily Treasury Statement (DTS): Federal Tax Deposits, August 2025. URLhttps://fiscaldata.treasury.gov/ datasets/daily-treasury-statement/federal-tax-deposits. Dataset page on U.S. Trea- sury Fiscal Data. Last updat...

    2025

  35. [43]

    Is the determinant the only group homomorphism fromGL n(R) to R×? Mathematics Stack Exchange

    user856. Is the determinant the only group homomorphism fromGL n(R) to R×? Mathematics Stack Exchange. URLhttps://math.stackexchange.com/q/727050. URL:https://math.stackexchange.com/q/727050 (version: 2017-04-13)

    2017

  36. [44]

    Beyond accuracy: What data quality means to data consumers.Journal of management information systems, 12(4):5–33, 1996

    Richard Y Wang and Diane M Strong. Beyond accuracy: What data quality means to data consumers.Journal of management information systems, 12(4):5–33, 1996

    1996

  37. [45]

    L dmi: A novel information-theoretic loss function for training deep nets robust to label noise.Advances in neural information processing systems, 32, 2019

    Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: A novel information-theoretic loss function for training deep nets robust to label noise.Advances in neural information processing systems, 32, 2019

    2019

  38. [46]

    Barlow twins: Self- supervised learning via redundancy reduction, 2021

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ ephane Deny. Barlow twins: Self- supervised learning via redundancy reduction, 2021. URLhttps://arxiv.org/abs/2103. 03230. 19

    2021

  39. [47]

    The limits of multi-task peer prediction.CoRR, abs/2106.03176, 2021

    Shuran Zheng, Fang-Yi Yu, and Yiling Chen. The limits of multi-task peer prediction.CoRR, abs/2106.03176, 2021. URLhttps://arxiv.org/abs/2106.03176

    2021 arXiv

  40. [48]

    Proper dataset valuation by pointwise mutual information, 2025

    Shuran Zheng, Xuan Qi, Rui Ray Chen, Yongchan Kwon, and James Zou. Proper dataset valuation by pointwise mutual information, 2025. URLhttps://arxiv.org/abs/2405.18253

    2025 arXiv

  41. [49]

    Characteristic kernels on hilbert spaces, banach spaces, and on sets of measures, 2022

    Johanna Ziegel, David Ginsbourger, and Lutz D¨ umbgen. Characteristic kernels on hilbert spaces, banach spaces, and on sets of measures, 2022. URLhttps://arxiv.org/abs/2206. 07588

    2022

  42. [50]

    Zou and Ryan P

    James Y. Zou and Ryan P. Adams. Priors for diversity in generative latent variable models. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’12, page 2996–3004, Red Hook, NY, USA, 2012. Curran Associates Inc. 20 A Pre...

    2012

  43. [52]

    anti-reflexive: no element is larger than itself

  44. [53]

    asymmetry: ifa≻bthen notb≻a

  45. [54]

    Next, we show that the reliability orderings defined in Section 2 form a strict partial order over reports, given a fixed true data

    Transitivity: ifa≻bandb≻c, thena≻c. Next, we show that the reliability orderings defined in Section 2 form a strict partial order over reports, given a fixed true data. Proposition B.1.For anyx∈ X N , the exact match ordering≻ x EXACT is a strict partial order on all ˆxand ˆx′...

  46. [55]

    If there is a reliability score that preserves the Hamming ordering onP indep,Q dom, E[S(P1Q1)]>E[S(P 1Q′ 1)] andE[S(P 2Q2)]<E[S(P 2Q′ 2)] (11) which reaches a contradiction asP 1Q1 =P 2Q2 andP 1Q′ 1 =P 2Q′

  47. [56]

    To this end, we construct P1 =   0.74 0 0.26 0.26 0.74 0 0 0.26 0.74   ,Q 1 = 1 3   0.8 0 0.2 0.2 0.8 0 0 0.2 0.8   ,Q ′ 1 = 1 3   0.7 0.3 0 0 0.7 0.3 0.3 0 0.7   . For the second setting, we defineP 2 =I, and Q2 =P 1Q1 = 1 3   0.592 0.052 0.356 0.356 0.592...

  48. [57]

    By direct computation, we have Tr(Q 1) = 24 30 > Tr(Q′

  49. [58]

    = 21 30 and Tr(Q2) = 1776 3000 <Tr(Q ′

  50. [59]

    Finally, note that we can easily generalize this construction beyond three dimensions by padding the other dimension with identity

    = 1788 3000 . Finally, note that we can easily generalize this construction beyond three dimensions by padding the other dimension with identity. Interestingly, the same construction works for general dist-ordering, due to the symmetry in Q1,Q ′ 1,Q 2 andQ ′

  51. [60]

    Hence, with Eq

    First note that PN n=1 dist(ˆxn, xn) =N P i,j∈[d] Q(i, j) dist(i, j) =N⟨Q,dist⟩F where⟨·,·⟩ F is the Frobenius inner product defined in Section A. Hence, with Eq. (11), it is sufficient to show the above construction satisfies ⟨Q1,dist⟩ F >⟨Q ′ 1,dist⟩ F and⟨Q 2,dist⟩ F <⟨Q ′ ...

  52. [61]

    Moreover, delta kernel is injective, so the statement also holds

    Because P∈ Pindep andϕis injective, the quadratic form equals zero if and only ifa(x) = 0 for allx. Moreover, delta kernel is injective, so the statement also holds. Finally, for any pseudo-posterior observations, Eq. (28) can be written as ⟨ X x,y a(x)P(y, x)y, X x′,y′ a(x′)P...

  53. [62]

    Total variation has Φ(a) as 1 2 |a−1|

  54. [63]

    KL-divergence hasaloga 3.χ 2-divergence hasa 2 −1

  55. [64]

    10P/Qis the Radon-Nikodym derivative between measuresPandQ, and it is equal to the ratio of density function

    Squared Hellinger distance has (1− √a)2 8We set 0/0 = 0 ifw(y) = 0 9Pis absolutely continuous with respect toQ: for any measurable setA∈ F,Q(A) = 0⇒P(A) = 0. 10P/Qis the Radon-Nikodym derivative between measuresPandQ, and it is equal to the ratio of density function. 35 In the...

  56. [65]

    Top-kvolume hasψ ∧k(s) = Qk i=1 ¯si

  57. [66]

    The maximum correlation can be also written as max (f,g)∈S E[f(x)g(y)] whereSis the collection of real-valued random variables so thatEf(x) =Eg(y) = 0 and Ef(x) 2 =Eg(y) 2 = 1

    Maximal correlationψ max = ¯s1. The maximum correlation can be also written as max (f,g)∈S E[f(x)g(y)] whereSis the collection of real-valued random variables so thatEf(x) =Eg(y) = 0 and Ef(x) 2 =Eg(y) 2 = 1

  58. [67]

    experiment-agnostic

    Ky-Fank-sum Pk i=1 ¯si 4.χ 2-mutual informationI χ2 (x,y) = P x,y µ ˆx(x)µy(y)( J(y,x) µ ˆx(x)µy(y) −1) 2 =∥ ¯J∥ F = P i ¯s2 i Similarly, the impossibility results in Section 3 still apply and they are generally not experiment-agnostic. G.2 Experiments on Score Comparison We f...

  59. [1894]

    URLhttps://proceedings.mlr.press/v70/koh17a.html

    PMLR, 06–11 Aug 2017. URLhttps://proceedings.mlr.press/v70/koh17a.html

    2017

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.