Pith. sign in

REVIEW 2 minor 61 references

The Effect of Omitted Variables on the Sign of Regression Coefficients

T0 review · 0 major / 2 minor · reviewed 2026-05-24 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Omitted variables can reverse regression coefficient signs at much smaller values of Oster's delta than those needed to drive the coefficient to zero.

desk verdict Oster's delta can look large while omitted variables still flip signs at much smaller values, and the authors propose a modified threshold to target that. read the letter →

arxiv 2208.00552 v5 submitted 2022-08-01 econ.EM stat.ME

Matthew A. Masten , Alexandre Poirier This is my paper
classification econ.EMstat.ME
keywords omittedvariablesregressioncoefficientsrobustnessmeasuresOster'sdeltasignreversalproportionalselectionsensitivityanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that Oster's delta, a widely used robustness metric for omitted variables in regressions, has an asymmetric effect on coefficient signs versus magnitudes. Under the proportional selection assumption, the delta value sufficient to flip a sign is typically far below the value that would make the coefficient exactly zero. This implies that large reported delta values, which are taken as evidence that omitted variables are unimportant, can still permit sign reversals from modest omitted-variable effects. The authors therefore introduce a modified robustness measure focused on sign stability. They demonstrate the issue through four empirical applications and two meta-analyses, with code available in a Stata module.

What carries the argument

Oster's delta under the proportional selection assumption, which measures the relative selection on unobservables needed to nullify a result, extended to track the threshold for sign reversal instead.

What would settle it

In any given regression, compute both the delta that sets the coefficient exactly to zero and the (typically smaller) delta that changes its sign; if the sign-flip delta is consistently lower across applications, the central claim holds.

Watch

Extended reading notes

Core claim

Depending on the impact measure, omitted variables make it substantially easier to flip the sign of a regression coefficient than to drive it to zero, so that Oster's delta values large enough to suggest robustness can still be reversed by smaller selection effects; a modified measure is proposed to address this.

Load-bearing premise

The analysis assumes the standard linear regression setup together with the proportional selection assumption that links the correlation of omitted variables with the treatment to their correlation with the observed controls.

Editorial extensions

If this is right

  • Researchers reporting large Oster delta values should also report the lower threshold at which the coefficient sign reverses.
  • In the four empirical applications, the modified measure alters conclusions about robustness compared with the original delta.
  • Meta-analyses of published studies would likely show many results vulnerable to sign changes at modest omitted-variable strengths.
  • The Stata module allows direct computation of the sign-flip delta alongside the original measure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same asymmetry may appear in other omitted-variable bounds or sensitivity analyses that focus on magnitude rather than direction.
  • Re-analysis of existing papers citing large delta values could identify cases where signs are less stable than claimed.
  • Simulations that vary the degree of selection on unobservables could map out the full range of possible sign changes in typical datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript shows that, under the linear regression model and Oster's (2019) proportional selection assumption, the value of delta required for omitted variables to flip the sign of a coefficient is strictly smaller than the value required to drive the coefficient to zero. Consequently, large reported values of Oster's delta (suggesting robustness) can still be consistent with sign reversal at smaller selection strengths. The authors derive this relation, propose a modified robustness measure, illustrate the results in four empirical applications and two meta-analyses, and release a Stata module (regsensitivity).

Significance. If the central comparison of thresholds holds, the result identifies a previously under-appreciated feature of a widely used robustness diagnostic and supplies a direct fix. The empirical illustrations and open-source implementation provide concrete value to applied researchers. The analysis stays within the standard linear setup and proportional-selection framework of Oster (2019) without introducing new assumptions.

minor comments (2)
  1. [Abstract / Introduction] The abstract and introduction would benefit from an explicit statement of the modified measure's formula (parallel to Oster's delta) so readers can see the adjustment immediately.
  2. [Empirical applications] In the empirical applications, it would be helpful to report both the original Oster delta and the proposed modified measure side-by-side for each specification to facilitate direct comparison.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the supportive summary, positive assessment of significance, and recommendation of minor revision. We appreciate the recognition that our comparison of thresholds for Oster's delta identifies an under-appreciated feature of the robustness diagnostic and that the empirical illustrations and Stata implementation add value. As no specific major comments were raised, we have no point-by-point responses. We will incorporate any minor revisions needed to further clarify the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity detected

full rationale

The paper derives a direct mathematical relation between the value of Oster's delta that flips the sign of the coefficient and the value that drives it to zero, under the inherited proportional selection assumption. This follows algebraically from the standard omitted-variable bias formula in the linear model without any reduction to a fitted quantity, self-definition, or load-bearing self-citation. The proposed modified robustness measure is introduced as an explicit alternative threshold to address the derived property, not as a renaming or re-derivation of the inputs. The analysis is self-contained against the external Oster (2019) benchmark and contains no steps that equate outputs to inputs by construction.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Review based on abstract only; no free parameters, new entities, or non-standard axioms are described.

assumptions (1)
  • domain assumption Linear regression model with omitted variables correlated with included regressors under proportional selection
    This is the maintained framework inherited from Oster (2019) and required for the delta comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Effect of Omitted Variables on the Sign of Regression Coefficients." pith.science (2026). https://pith.science/paper/2208.00552

@misc{pith2026220800552,
  author       = {Pith},
  title        = {Pith review of: The Effect of Omitted Variables on the Sign of Regression Coefficients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2208.00552}},
  note         = {Machine review of arXiv:2208.00552}
}
read the original abstract

We show that, depending on how the impact of omitted variables is measured, it can be substantially easier for omitted variables to flip coefficient signs than to drive them to zero. This behavior occurs with "Oster's delta" (Oster 2019), a widely reported robustness measure. Consequently, any time this measure is large -- suggesting that omitted variables may be unimportant -- a much smaller value reverses the sign of the parameter of interest. We propose a modified measure of robustness to address this concern. We illustrate our results in four empirical applications and two meta-analyses. We implement our methods in the companion Stata module regsensitivity.

Figures

Figures reproduced from arXiv: 2208.00552 by the authors.

Figure 1
Figure 1. Example of Oster’s identified set BI (δ, R2 long) for the regression coefficient β as a function of the sensitivity parameter δ (see section 3 for a definition of this set and a description of the R2 long parameter). As we discuss in section 3.3, this identified set always has either one, two, or three elements. Both plots show the same set, but with different horizontal axis ranges. In this example, the explain awa… view at source ↗
Figure 2
Figure 2. The cumulative identified set for the example of figure [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The convex hull of the estimated identified set for [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Estimated identified sets BbI (δ, R2 long) for assessing the robustness of Satyanath et al. (2017) results to omitted variables. Left column: R2 long = 1. Right column: R2 long = 1.3Rb2 med. Top row: Treatment variable includes all associations. Middle row: Treatment v…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 61 canonical work pages

  1. [1]

    Altonji, J. G., T. Conley, T. E. Elder, and C. R. Taber (2019): Methods for using selection on observed variables to address selection on unobserved variables, Working paper

    work page 2019

  2. [2]

    Altonji, J. G., T. E. Elder, and C. R. Taber (2005): Selection on observed and unobserved variables: Assessing the effectiveness of Catholic schools, Journal of Political Economy, 113, 151--184

    work page 2005

  3. [3]

    --- -.1pt --- -.1pt --- (2008): Using selection on observed variables to assess bias from unobservables when evaluating swan-ganz catheterization, American Economic Review P&P, 98, 345--50

    work page 2008

  4. [4]

    rugged individualism

    Bazzi, S., M. Fiszbein, and M. Gebresilasse (2020): Frontier culture: The roots and persistence of “rugged individualism” in the United States, Econometrica, 88, 2329--2368

    work page 2020

  5. [5]

    Bellows, J. and E. Miguel (2009): War and local collective action in Sierra Leone, Journal of Public Economics, 93, 1144--1157

    work page 2009

  6. [6]

    Bonney, M

    Blandhol, C., J. Bonney, M. Mogstad, and A. Torgovitsky (2022): When is TSLS actually late? Working paper

    work page 2022

  7. [7]

    Breusch, T. S. (1986): Hypothesis testing in unidentified models, Review of Economic Studies, 53, 635--651

    work page 1986

  8. [8]

    Chen, B. and J. Pearl (2015): Exogeneity and robustness, Working paper

    work page 2015

Show all 61 references
  1. [9]

    Cinelli, C. and C. Hazlett (2020): Making sense of sensitivity: Extending omitted variable bias, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82, 39--67

    2020

  2. [10]

    Clarke, K. A. (2009): Return of the phantom menace: Omitted variable bias in political research, Conflict Management and Peace Science, 26, 46--66

    2009

  3. [11]

    Haenszel, E

    Cornfield, J., W. Haenszel, E. C. Hammond, A. M. Lilienfeld, M. B. Shimkin, and E. L. Wynder (1959): Smoking and lung cancer: Recent evidence and a discussion of some questions, Journal of the National Cancer Institute, 22, 173--203

    1959

  4. [12]

    (1835): De La Démocratie en Amérique

    de Tocqueville, A. (1835): De La Démocratie en Amérique

  5. [13]

    Diegert, P., M. A. Masten, and A. Poirier (2022): Assessing omitted variable bias when the controls are endogenous, arXiv preprint arXiv:2206.02303

    2022

  6. [14]

    Dippel, C. and S. Heblich (2021): Leadership in Social Movements: Evidence from the ``Forty-Eighters'' in the Civil War, American Economic Review, 111, 472--505

    2021

  7. [15]

    Gentzkow, and H

    Finkelstein, A., M. Gentzkow, and H. Williams (2021): Place-based drivers of mortality: Evidence from migration, American Economic Review, 111, 2697--2735

    2021

  8. [16]

    Frank, K. A. (2000): Impact of a confounding variable on a regression coefficient, Sociological Methods & Research, 29, 147--194

    2000

  9. [17]

    Gonzalez, F. and E. Miguel (2015): War and local collective action in Sierra Leone: A comment on the use of coefficient stability approaches, Journal of Public Economics, 128, 30--33

    2015

  10. [18]

    Gregg, A. G. (2020): Factory productivity and the concession system of incorporation in late imperial Russia, 1894--1908, American Economic Review, 110, 401--27

    2020

  11. [19]

    Hosman, C. A., B. B. Hansen, and P. W. Holland (2010): The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder, The Annals of Applied Statistics, 4, 849--870

    2010

  12. [20]

    Imbens, G. W. (2003): Sensitivity to exogeneity assumptions in program evaluation, American Economic Review P&P, 126--132

    2003

  13. [21]

    (2016): Bounding a linear causal effect using relative correlation restrictions, Journal of Econometric Methods, 5, 117--141

    Krauth, B. (2016): Bounding a linear causal effect using relative correlation restrictions, Journal of Econometric Methods, 5, 117--141

    2016

  14. [22]

    Masten, M. A. and A. Poirier (2018): Identification of treatment effects under conditional partial independence, Econometrica, 86, 317--351

    2018

  15. [23]

    --- -.1pt --- -.1pt --- (2020): Inference on breakdown frontiers, Quantitative Economics, 11, 41--111

    2020

  16. [24]

    (1990): Understanding LOVE (left out variables error): A method for estimating the effects of omitted variables

    Mauro, R. (1990): Understanding LOVE (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108, 314

    1990

  17. [25]

    Mian, A. and A. Sufi (2014): What explains the 2007--2009 drop in employment? Econometrica, 82, 2197--2223

    2014

  18. [26]

    Murphy, K. M. and R. H. Topel (1990): Efficiency wages reconsidered: Theory and evidence, in Advances in the Theory and Measurement of Unemployment, Springer, 204--240

    1990

  19. [27]

    (2019): Unobservable selection and coefficient stability: Theory and evidence, Journal of Business & Economic Statistics, 37, 187--204

    Oster, E. (2019): Unobservable selection and coefficient stability: Theory and evidence, Journal of Business & Economic Statistics, 37, 187--204

    2019

  20. [28]

    Putnam, R. D. (2000): Bowling Alone: The Collapse and Revival of American Community, Simon and Schuster

    2000

  21. [29]

    Robins, J. M., A. Rotnitzky, and D. O. Scharfstein (2000): Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models, in Statistical models in epidemiology, the environment, and clinical trials, Springer, 1--94

    2000

  22. [30]

    Rosenbaum, P. R. (1995): Observational Studies, Springer

    1995

  23. [31]

    --- -.1pt --- -.1pt --- (2002): Observational Studies, Springer, second ed

    2002

  24. [32]

    Rosenbaum, P. R. and D. B. Rubin (1983): Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome, Journal of the Royal Statistical Society, Series B, 212--218

    1983

  25. [33]

    Voigtl \"a nder, and H.-J

    Satyanath, S., N. Voigtl \"a nder, and H.-J. Voth (2017): Bowling for fascism: Social capital and the rise of the Nazi Party, Journal of Political Economy, 125, 478--526

    2017

  26. [34]

    Squicciarini, M. P. (2020): Devotion and development: Religiosity, education, and economic progress in nineteenth-century France, American Economic Review, 110, 3454--91

    2020

  27. [35]

    Caceres Bravo, P

    Abaluck, J., M. Caceres Bravo, P. Hull, and A. Starc (2021): Mortality effects and choice across private health insurance plans, Quarterly Journal of Economics, 136, 1557--1610

    2021

  28. [36]

    Abdulkadiroglu, A., P. A. Pathak, J. Schellenberg, and C. R. Walters (2020): Do parents value school effectiveness? American Economic Review, 110, 1502--39

    2020

  29. [37]

    Diamond, J.-P

    Allcott, H., R. Diamond, J.-P. Dub \'e , J. Handbury, I. Rahkovsky, and M. Schnell (2019 a ): Food deserts and the causes of nutritional inequality, Quarterly Journal of Economics, 134, 1793--1844

    2019

  30. [38]

    Allcott, H., B. B. Lockwood, and D. Taubinsky (2019 b ): Regressive sin taxes, with an application to the optimal soda tax, Quarterly Journal of Economics, 134, 1557--1626

    2019

  31. [39]

    Arbatl , C. E., Q. H. Ashraf, O. Galor, and M. Klemp (2020): Diversity and conflict, Econometrica, 88, 727--797

    2020

  32. [40]

    Burgess, A

    Bertrand, M., R. Burgess, A. Chawla, and G. Xu (2020): The glittering prizes: Career incentives and bureaucrat performance, Review of Economic Studies, 87, 626--655

    2020

  33. [41]

    Gartenberg, and A

    Blader, S., C. Gartenberg, and A. Prat (2020): The contingent effect of management practices, Review of Economic Studies, 87, 721--749

    2020

  34. [42]

    Blouin, A. and R. Macchiavello (2019): Strategic default in the international coffee market, Quarterly Journal of Economics, 134, 895--951

    2019

  35. [43]

    (2021): Team-specific human capital and team performance: Evidence from doctors, American Economic Review, 111, 3923--62

    Chen, Y. (2021): Team-specific human capital and team performance: Evidence from doctors, American Economic Review, 111, 3923--62

    2021

  36. [44]

    Petrencu, D

    Couttenier, M., V. Petrencu, D. Rohner, and M. Thoenig (2019): The violent legacy of conflict: evidence on asylum seekers, crime, and public policy in Switzerland, American Economic Review, 109, 4378--4425

    2019

  37. [45]

    Forty-Eighters

    Dippel, C. and S. Heblich (2021): Leadership in Social Movements: Evidence from the" Forty-Eighters" in the Civil War, American Economic Review, 111, 472--505

    2021

  38. [46]

    (2019): Kinship, cooperation, and the evolution of moral systems, Quarterly Journal of Economics, 134, 953--1019

    Enke, B. (2019): Kinship, cooperation, and the evolution of moral systems, Quarterly Journal of Economics, 134, 953--1019

    2019

  39. [47]

    --- -.1pt --- -.1pt --- (2020): Moral values and voting, Journal of Political Economy, 128, 3679--3729

    2020

  40. [48]

    Eugster, B. and R. Parchet (2019): Culture and taxes, Journal of Political Economy, 127, 296--337

    2019

  41. [49]

    Nardotto, and T

    Gavazza, A., M. Nardotto, and T. Valletti (2019): Internet and politics: Evidence from UK local elections and local government policies, Review of Economic Studies, 86, 2092--2135

    2019

  42. [50]

    Sorkin, and H

    Goldsmith-Pinkham, P., I. Sorkin, and H. Swift (2020): Bartik instruments: What, when, why, and how, American Economic Review, 110, 2586--2624

    2020

  43. [51]

    Grosfeld, I., S. O. Sakalli, and E. Zhuravskaya (2020): Middleman minorities and ethnic violence: anti-Jewish pogroms in the Russian empire, Review of Economic Studies, 87, 289--342

    2020

  44. [52]

    Grosjean, P. and R. Khattar (2019): It’s raining men! Hallelujah? The long-run consequences of male-biased sex ratios, Review of Economic Studies, 86, 723--754

    2019

  45. [53]

    Melnikov, and E

    Guriev, S., N. Melnikov, and E. Zhuravskaya (2021): 3G internet and confidence in government, Quarterly Journal of Economics, 136, 2533--2613

    2021

  46. [54]

    (2021): The origins of violence in Rwanda, Review of Economic Studies, 88, 730--763

    Heldring, L. (2021): The origins of violence in Rwanda, Review of Economic Studies, 88, 730--763

    2021

  47. [55]

    Hoffman, M. and S. Tadelis (2021): People management skills, employee attrition, and manager rewards: An empirical analysis, Journal of Political Economy, 129, 243--285

    2021

  48. [56]

    Murphy, R. and F. Weinhardt (2020): Top of the class: The importance of ordinal rank, Review of Economic Studies, 87, 2777--2826

    2020

  49. [57]

    Schindler, D. and M. Westcott (2021): Shocking racial attitudes: Black GIs in Europe, Review of Economic Studies, 88, 489--520

    2021

  50. [58]

    (2020): Gifts of the immigrants, woes of the natives: Lessons from the age of mass migration, Review of Economic Studies, 87, 454--486

    Tabellini, M. (2020): Gifts of the immigrants, woes of the natives: Lessons from the age of mass migration, Review of Economic Studies, 87, 454--486

    2020

  51. [59]

    (2019): The mission: Human capital transmission, economic persistence, and culture in South America, Quarterly Journal of Economics, 134, 507--556

    Valencia Caicedo, F. (2019): The mission: Human capital transmission, economic persistence, and culture in South America, Quarterly Journal of Economics, 134, 507--556

    2019

  52. [60]

    Verner, E. and G. Gy \"o ngy \"o si (2020): Household debt revaluation and the real economy: Evidence from a foreign currency debt crisis, American Economic Review, 110, 2667--2702

    2020

  53. [61]

    Weigel, J. L. (2020): The participation dividend of taxation: How citizens in Congo engage more with the state when it tries to tax them, Quarterly Journal of Economics, 135, 1849--1903

    2020

Pith tools

Reviewed May 24, 2026 · model on record in the stance chip above.