REVIEW 3 major objections 5 minor 45 references
Beyond Exposure: Optimizing Ranking Fairness with Non-linear Time-Income Functions
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Ranking fairness should be measured by provider income—what providers actually earn from exposure—and a derivative-aware scoring rule can optimize it efficiently.
desk verdict The income-fairness idea is worth building on, but Eq. (23) drops the f_d(t_n) factor in the first-order term, so the printed DIDRF score does not provably optimize the stated objective; fixable, but needs a correction and re-run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the per-item score function s(d,n) that drives DIDRF. It is obtained by taking the first two terms of the Taylor expansion of the income-fairness objective with respect to marginal income ΔI(d,n). The first-order term g(d,n) measures the current marginal contribution of giving item d exposure to moving cumulative income toward proportionality with relevance. The second-order term h(d,n), derived from the diagonal of the Hessian (the self-impact approximation), captures how an item's marginal allocation interacts with the rest of the set; the bound ΔE² ≤ ΔE turns this quadratic into a linear surrogate. In online settings, an uncertainty term û(d,n) is added. The rearrang
What would settle it
In a production recommender or search system where actual per-provider income is observable, run DIDRF (with f_d estimated from historical conversion data) against a strong exposure-fairness baseline such as MCFair over many queries with time-varying conversion. If DIDRF does not reduce the income-unfairness metric (Eq. 11) at equal or better cNDCG, the central claim fails. A cheaper check: on the same datasets used here, replace the hand-crafted f_d with a realistically noisy estimator; if the fairness gains vanish under mild noise, the method's practical advantage dissipates.
Extended reading notes
Core claim
The central claim is that income fairness can be optimized efficiently through a marginal, Taylor-expansion-based reduction. The objective Obj(q,n) = eff + γ·fair is maximized when items are sorted by the score s(d,n) = R(d) + γ(g(d,n) + 1/2 h(d,n)), where g(d,n) is the first-order effect of granting item d marginal income on the fairness metric, and h(d,n) is the diagonal second-order (self-impact) term. Because the fairness metric is a quadratic polynomial in income, the second-order expansion is exact; the diagonal approximation plus the bound ΔE² ≤ ΔE yields a linear surrogate that preserves a pure sorting form. In the online setting, an additional term −η·û(d,n), derived from a second-o
Load-bearing premise
The paper assumes that the exposure-to-income function f_d(t) is reliably estimated, and its experiments supply that function by hand; if real-world conversion rates cannot be predicted accurately, the income-fairness metric and DIDRF's scores are undefined, and the simulated gains may not transfer to production systems.
Editorial extensions
If this is right
- Exposure-fair ranking algorithms (FairCo, MCFair, PLFair, MMF) do not optimize income fairness when conversion from exposure to income is time-dependent; income, not exposure, should be the measured and controlled resource.
- DIDRF reduces fair ranking to a single per-query sort, giving the same asymptotic cost as plain relevance ranking, so income fairness becomes practical at large scale.
- The second-order term h(d,n) consistently improves both fairness and effectiveness across periodic, aperiodic, and constant conversion settings, including when the problem reduces to plain exposure fairness.
- In online settings, the second-order uncertainty term (1/ʳ) implements automatic exploration-exploitation scheduling: it pushes exposure toward items with high estimation uncertainty and fades as estimates stabilize.
- Under the rearrangement inequality, the top-k list produced by the score is exactly optimal for the marginal objective at each timestep, not merely a heuristic.
Reading between the lines
- If conversion rates are predictable (by hour, season, or news cycle), the same score-based scheme extends naturally beyond time to any contextual dimension—region, device, user segment—where an income function can be estimated.
- The practical bottleneck is likely to be the income-function estimator rather than the ranking optimizer; pairing DIDRF with a well-calibrated revenue-prediction model (e.g., purchase-rate or CTR forecasting) is the natural deployment test.
- Because the fairness metric is a sum over item pairs, the same Taylor-derivative machinery could be adapted to group-level income fairness by aggregating income and relevance over groups.
- A direct empirical check of the framework: with real advertising or marketplace logs where per-item revenue is recorded, DIDRF should reduce income unfairness relative to exposure-fairness baselines—if it does not, the mismatch lies in the estimated income function, not the optimization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that exposure fairness is an incomplete proxy for provider utility and proposes 'income fairness', defined as proportionality between cumulative context-dependent income and relevance. It introduces an income-unfairness metric (Eq. 11), and a Dynamic-Income-Derivative-aware Ranking Fairness (DIDRF) algorithm that greedily maximizes a marginal objective via a Taylor expansion of the fairness term, yielding a per-item score and a sorting-based ranking rule (Eqs. 23–25). The paper reports offline and online simulated experiments on MQ2008 and Istella-s with periodic and aperiodic income functions, claiming that DIDRF consistently outperforms existing fair-ranking baselines in income fairness while preserving effectiveness.
Significance. If the central derivation were sound, the paper would make a useful contribution: it formalizes a natural generalization of exposure fairness to context-dependent conversion, and the proposed O(|D| log |D|) scoring rule is computationally attractive. The conceptual point that exposure-based fairness can diverge from utility-based fairness is well taken, and the experiments, though synthetic, bridge that gap in a controlled way. However, the main derivation contains a concrete algebraic error, and the surrogate used in the optimization is not exact as claimed. These issues are fixable, but they are load-bearing for the paper's central claim that DIDRF optimizes the stated income-fairness objective.
major comments (3)
- [§5.1, Eq. (23)] Eq. (23) drops the income-function factor f_d(t_n) from the first-order fairness term. From Eq. (16), ΔI(d,n)=f_d(t_n)ΔE(d,n), so the first-order term in Eq. (18) is Σ_d [∂fair/∂I(d,n)] f_d(t_n) ΔE(d,n). Eq. (19) defines g(d,n)=∂fair/∂I(d,n), so the correct score is R(d)+γ(f_d(t_n)g(d,n)+1/2 h(d,n)), not R(d)+γ(g(d,n)+1/2 h(d,n)). Since h in Eq. (42) already contains f_d^2(t_n), the printed score misweights the marginal fairness signal whenever f_d(t_n)≠1. In the very setting the paper targets, this is not a harmless rescaling: it changes the relative contribution of the first-order term and can alter the ranking. The rearrangement-inequality argument in Eq. (24) therefore proves optimality for a different objective than Obj(q,n) in Eq. (12).
- [§5.1, Eq. (22) and Appendix A] Even after restoring the missing f_d factor, the derivation is not exact. Eq. (22) and Appendix A replace the Hessian by a diagonal self-impact approximation and then use ΔE^2≤ΔE to obtain the linear surrogate h(d,n)ΔE(d,n). Because h(d,n)≤0, this is a lower bound on the quadratic term, not an equality. Consequently, the text's claim that the second-order Taylor expansion is 'exact' is only true before these surrogates are applied; the final score optimizes a lower-bounding surrogate, not Δfair. The paper should explicitly state this and, ideally, provide a bound on the approximation gap. As written, the conclusion that DIDRF 'jointly optimizes effectiveness and income fairness' overstates what is established.
- [§5.1 and §6.1.3] The experimental evidence is entirely conditioned on the assumption that the income function f_d(t) is known or reliably estimated. The paper acknowledges this in §5.1 ('we assume that the income function f_d has been reliably estimated') and instantiates it with two hand-crafted synthetic functions in Table 3 under t_n=n. This is acceptable for a controlled simulation study, but it does not support the strong practical claim in §7 that DIDRF 'provides a robust and efficient approach for practical ranking optimization.' I would ask for either a sensitivity analysis with noisy or mis-estimated income functions, or a clear softening of the practical-application claim.
minor comments (5)
- [Eq. (6)] The double sum uses the same dummy symbol d in both summands; the intent is clearly d_x and d_y. Please fix the notation.
- [Eq. (13)] The notation 'max Obj(q,n)=max ΔObj(q,n)' is imprecise; for a fixed history, the argmax of Obj(q,n) equals the argmax of ΔObj(q,n), but the two maxima have different values. Rephrase to avoid confusing the optimization argument with the objective value.
- [Table 1] The 'best fairness' point is not defined: is it the minimum unfairness over the γ sweep, or the point maximizing a combined criterion? Also, the unfairness magnitudes differ by orders of magnitude between MQ2008 and Istella-s; reporting normalized or relative unfairness would aid comparability. The statement 'p<0.001' is reported only for DIDRF vs. MCFair; please specify the test and whether multiple-comparison corrections were applied.
- [§6.1.3 and Appendix D] The claim that one periodic and one aperiodic function 'cover most real-world income-change scenarios' is an overstatement. Suggest rephrasing to 'represent two broad classes' and note the lack of validation on real income data.
- [§6.1.2] The baselines are said to be 're-derived to match the income-fairness formulation,' but no details are given. Please provide the modified objectives or scoring rules for at least FairCo, MCFair, and FARA so the comparison is reproducible.
Circularity Check
No significant circularity; the derivation is self-contained. (An algebraic inconsistency in Eq. 23 is a correctness issue, not circularity.)
full rationale
The paper's central derivation is self-contained: it defines an income-fairness metric (Eq. 11), forms an objective max Obj = eff + gamma*fair (Eq. 12), and derives a per-item score (Eq. 23) via Taylor expansion and the rearrangement inequality. Evaluating an algorithm on the same metric it is designed to optimize is standard for an optimizer, not circular. No parameter is fitted to the target result: gamma and eta are trade-off sweeps, and the income functions f_d are hand-constructed in Table 3, not inferred from the fairness outcome. The self-citations to the authors' own prior work (MMF/MCFair/FARA, refs [35],[37],[38]) are background/technique citations; the load-bearing derivation in Eqs. (13)-(25) is performed in the paper and does not depend on an unverified uniqueness theorem or an imported ansatz. I also flag two non-circular concerns per the reviewing rule: (1) Eq. (23) substitutes sum_d g(d,n) ΔE(d,n) for the first-order fairness term sum_d g(d,n) ΔI(d,n) = sum_d g(d,n) f_d(t_n) ΔE(d,n), omitting the multiplicative f_d(t_n); this is an algebraic inconsistency with Eq. (18)-(19), not a circular step, but it undermines the exactness claim. (2) Section 5.1 states 'we assume that the income function f_d has been reliably estimated' and Section 6.1.3 states 'Due to the lack of real-world datasets, we designed income functions'; this limits external validity but does not make the derivation circular. Hence the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- fairness trade-off γ =
swept in [0.0, 1000.0]
- uncertainty trade-off η =
swept in [0, 100]
- income function shapes f_d(t) =
periodic tilted square wave; aperiodic exponential decay, normalized to [0,1]
assumptions (6)
- domain assumption Click model p(c=1|d)=p(e=1|d)p(r=1|d), with examination probability depending only on position and being known.
- domain assumption Income fairness normative criterion: cumulative income proportional to cumulative relevance for all items.
- domain assumption Income function f_d(t) is known or reliably estimated.
- domain assumption Timestep n is aligned with real time t_n=n.
- ad hoc to paper Diagonal-only Hessian plus linear surrogate ΔE^2≈ΔE preserves the ranking optimum.
- ad hoc to paper Synthetic periodic/aperiodic income functions represent real income dynamics.
Cite this review
Pith. "Pith review of Beyond Exposure: Optimizing Ranking Fairness with Non-linear Time-Income Functions." pith.science (2026). https://pith.science/paper/N7NOJAS5
@misc{pith2026260203345,
author = {Pith},
title = {Pith review of: Beyond Exposure: Optimizing Ranking Fairness with Non-linear Time-Income Functions},
year = {2026},
howpublished = {\url{https://pith.science/paper/N7NOJAS5}},
note = {Machine review of arXiv:2602.03345}
}
read the original abstract
Ranking systems in web search and recommendation allocate attention among items and providers, and therefore need to balance relevance-based effectiveness with provider fairness. Existing fair-ranking methods commonly focus on exposure fairness, where cumulative exposure is allocated in proportion to item merit. However, exposure is often only an intermediate signal: the actual utility received by a provider may depend on context-dependent conversion from exposure to income, such as clicks, purchases, or advertising value. This paper studies fair ranking under context-dependent provider utility, which we refer to as income. We formalize income fairness by requiring cumulative provider income to be proportional to relevance, and define an income-unfairness metric based on this proportionality condition. We then propose DIDRF, a Dynamic-Income-Derivative-aware Ranking Fairness algorithm for income-fair ranking. DIDRF uses the quadratic structure of income-fairness violations to derive a state-aware scoring rule that jointly considers ranking effectiveness and the marginal effect of each ranking decision on cumulative income fairness. Experiments on standard learning-to-rank datasets with log-calibrated semi-synthetic income environments based on advertising and e-commerce logs show that DIDRF consistently improves income fairness over representative fair-ranking baselines while preserving competitive ranking effectiveness.
Figures
Reference graph
Works this paper leans on
-
[1]
Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018. Unbi- ased learning to rank with unbiased propensity estimation. InThe 41st interna- tional ACM SIGIR conference on research & development in information retrieval. 385–394
2018
-
[2]
Wayne M Alves and Peter H Rossi. 1978. Who should get what? Fairness judg- ments of the distribution of earnings.American journal of Sociology84, 3 (1978), 541–564
1978
-
[3]
Ashmi Banerjee, Gourab K Patro, Linus W Dietz, and Abhijnan Chakraborty
-
[4]
Asia J Biega, Krishna P Gummadi, and Gerhard Weikum. 2018. Equity of attention: Amortizing individual fairness in rankings. InThe 41st international acm sigir conference on research & development in information retrieval. 405–414
2018
-
[5]
Emily Black, Hadi Elzayn, Alexandra Chouldechova, Jacob Goldin, and Daniel Ho
-
[6]
Herm J Brascamp, Elliott H Lieb, and Joaquin Mazdak Luttinger. 1974. A general rearrangement inequality for multiple integrals.Journal of functional analysis 17, 2 (1974), 227–237
1974
-
[7]
Pedro G Campos, Fernando Díez, and Iván Cantador. 2014. Time-aware recom- mender systems: a comprehensive survey and analysis of existing evaluation protocols.User Modeling and User-Adapted Interaction24 (2014), 67–119
2014
-
[8]
Abhijnan Chakraborty, Saptarshi Ghosh, Niloy Ganguly, and Krishna P Gummadi
Show all 45 references
-
[9]
2022.Click models for web search
Aleksandr Chuklin, Ilya Markov, and Maarten De Rijke. 2022.Click models for web search. Springer Nature
2022
-
[10]
Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An ex- perimental comparison of click position-bias models. InProceedings of the 2008 international conference on web search and data mining. 87–94
2008
-
[11]
Fernando Diaz, Bhaskar Mitra, Michael D Ekstrand, Asia J Biega, and Ben Carterette. 2020. Evaluating stochastic rankings with expected exposure. In Proceedings of the 29th ACM international conference on information & knowledge management. 275–284
2020
-
[12]
Melissa M Favreault and Karen E Smith. 2016. A primer on the dynamic simulation of income model (DYNASIM3). (2016)
2016
-
[13]
Giovanni Gallo and Michele Raitano. 2023. SOS incomes: Simulated effects of COVID-19 and emergency benefits on individual and household income distribu- tion in Italy.Journal of European Social Policy33, 1 (2023), 101–116
2023
-
[14]
Ruoyuan Gao, Yingqiang Ge, and Chirag Shah. 2022. FAIR: Fairness-aware information retrieval evaluation.Journal of the Association for Information Science and Technology73, 10 (2022), 1461–1473
2022
-
[15]
Maria Heuss, Fatemeh Sarvi, and Maarten de Rijke. 2022. Fairness of Exposure in Light of Incomplete Exposure Estimation. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 759–769
2022
-
[16]
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems (TOIS)20, 4 (2002), 422–446
2002
-
[17]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay
-
[18]
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2019. An empiri- cal study of rich subgroup fairness for machine learning. InProceedings of the conference on fairness, accountability, and transparency. 100–109
2019
-
[19]
Tie-Yan Liu et al. 2009. Learning to rank for information retrieval.Foundations and Trends®in Information Retrieval3, 3 (2009), 225–331
2009
-
[20]
Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. 2016. Post-learning optimization of tree ensembles for efficient ranking. InProceedings of the 39th International ACM SIGIR conference on Research and Developme...
2016
-
[21]
InAcm Sigir Forum, Vol
Accurately interpreting clickthrough data as implicit feedback. InAcm Sigir Forum, Vol. 51. Acm New York, NY, USA, 4–11
-
[22]
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning.ACM Com- puting Surveys (CSUR)54, 6 (2021), 1–35
2021
-
[23]
Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020. Con- trolling fairness and bias in dynamic learning-to-rank. InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 429–438
2020
-
[24]
Harrie Oosterhuis. 2021. Computationally efficient optimization of plackett-luce ranking models for relevance and fairness. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1023–1032
2021
-
[25]
Masoud Mansoury. 2022. Understanding and mitigating multi-sided exposure bias in recommender systems.ACM SIGWEB Newsletter2022, Autumn (2022), 1–4
2022
-
[26]
Harrie Oosterhuis and Maarten de Rijke. 2021. Unifying online and counterfactual learning to rank: A novel counterfactual estimator that effectively utilizes online interventions. InProceedings of the 14th ACM international conference on web search and data mining. 463–471
2021
-
[27]
Gourab K Patro, Lorenzo Porcaro, Laura Mitchell, Qiuyue Zhang, Meike Zehlike, and Nikhil Garg. 2022. Fair ranking: a critical review, challenges, and future directions. In2022 ACM Conference on Fairness, Accountability, and Transparency. 1929–1942
2022
-
[28]
Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets.arXiv preprint arXiv:1306.2597(2013)
2013 arXiv
-
[29]
Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-aware unbiased learning to rank for top-k rankings. InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 489–498
2020
-
[30]
Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2219–2228
2018
-
[31]
Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking.Advances in neural information processing systems32 (2019)
2019
-
[32]
Alan Tsang, Bryan Wilder, Eric Rice, Milind Tambe, and Yair Zick. 2019. Group- fairness in influence maximization.arXiv preprint arXiv:1903.00967(2019)
2019 arXiv
-
[33]
Yuta Saito and Thorsten Joachims. 2022. Fair Ranking as Fair Division: Impact- Based Individual Fairness in Ranking. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1514–1524
2022
-
[34]
Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. InProceedings of the eleventh ACM international conference on web search and data mining. 610–618
2018
-
[35]
Tao Yang and Qingyao Ai. 2021. Maximizing marginal fairness for dynamic learning to rank. InProceedings of the Web Conference 2021. 137–145
2021
-
[36]
Tao Yang, Chen Luo, Hanqing Lu, Parth Gupta, Bing Yin, and Qingyao Ai. 2022. Can clicks be both labels and features? Unbiased Behavior Feature Collection and Uncertainty-aware Learning to Rank. InProceedings of the 45th International ACM SIGIR Conference on Research and Develo...
2022
-
[37]
Nicolas Usunier, Virginie Do, and Elvis Dohmatob. 2022. Fast online ranking with fairness of exposure. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2157–2167
2022
-
[38]
Tao Yang, Zhichao Xu, Zhenduo Wang, Anh Tran, and Qingyao Ai. 2023. Marginal-Certainty-aware Fair Ranking Algorithm. InProceedings of the Six- teenth ACM International Conference on Web Search and Data Mining. 24–32
2023
-
[39]
Quan Yuan, Gao Cong, Zongyang Ma, Aixin Sun, and Nadia Magnenat Thalmann
-
[40]
Meike Zehlike, Ke Yang, and Julia Stoyanovich. 2021. Fairness in ranking: A survey.arXiv preprint arXiv:2103.14000(2021)
2021 arXiv
-
[41]
Tao Yang, Zhichao Xu, Zhenduo Wang, and Qingyao Ai. 2023. FARA: Future-aware Ranking Algorithm for Fairness Optimization.arXiv preprint arXiv:2305.16637(2023)
2023 arXiv
-
[2013]
InProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Time-aware point-of-interest recommendation. InProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 363–372
-
[2017]
InProceedings of the 26th International Conference on World Wide Web
Optimizing the recency-relevancy trade-off in online news recommen- dations. InProceedings of the 26th International Conference on World Wide Web. 837–846
-
[2020]
In2020 IEEE International Conference on Big Data (Big Data)
Analyzing ‘Near Me’Services: Potential for Exposure Bias in Location-based Retrieval. In2020 IEEE International Conference on Big Data (Big Data). IEEE, 3642–3651
-
[2022]
InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency
Algorithmic fairness and vertical equity: Income fairness with IRS tax audit models. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1479–1503
2022
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.