Shrinkage Estimators for the Intercept in Linear and Uplift Regression

Published in Volume XXXIII, Issue 1, 2023, pages 35-52, doi: 10.7561/SACS.2023.1.35

Authors: S. Jaroszewicz, K. Rudaś

Abstract

Shrinkage estimators modify classical statistical estimators by scaling them towards zero in order to decrease their prediction error. We propose shrinkage estimators for linear regression models which explicitly take into account the presence of the intercept term, shrinking it independently from other coefficients. This is different from current shrinkage estimators, which treat the intercept just as an ordinary regression coefficient. We demonstrate that the proposed approach brings systematic improvements in prediction accuracy if the true intercept term differs in magnitude from other coefficients, which is often the case in practice. We then generalize the approach to uplift regression which aims to predict the causal effect of a specific action on an individual with given characteristics. In this case the proposed estimators improve prediction accuracy over previously proposed shrinkage estimators and achieve impressive performance gains over original models.

Full Text (PDF)

References

[1] Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):7353-7360, 2016. doi:10.1073/pnas.1510489113.

[2] Christian Heumann, Thomas Nittner, Sandro Scheid, C.Radhakrishna Rao, and Helge Toutenburg. Linear Models: Least Squares and Alternatives. Springer New York, 2013. doi:10.1007/978-1-4899-0024-1.

[3] Paul W. Holland. Statistics and causal inference. Journal of the American Statistical Association, 81(396):945-960, 1986. doi:10.2307/2289064.

[4] Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, 1994. doi:10.1017/CBO9780511840371.

[5] Willard James and Charles Stein. Estimation with quadratic loss. In Jerzy Neyman, editor, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 361-379, 1961.

[6] Szymon Jaroszewicz and Piotr Rzepakowski. Uplift modeling with survival data. In ACM SIGKDD Workshop on Health Informatics (HI-KDD’14), 2014.

[7] Akio Namba and Kazuhiro Ohtani. MSE performance of the weighted average estimators consisting of shrinkage estimators. Communications in Statistics – Theory and Methods, 47(5):1204-1214, 2018. doi:10.1080/03610926.2017.1316860.

[8] Nicholas J. Radcliffe and Patrick D. Surry. Real-world uplift modelling with significance-based uplift trees. Portrait Technical Report TR-2011-1, Stochastic Solutions, 2011.

[9] Krzysztof Rudaś and Szymon Jaroszewicz. Linear regression for uplift modeling. Data Mining and Knowledge Discovery, 32(5):1275-1305, Sep 2018.

[10] Krzysztof Rudaś and Szymon Jaroszewicz. Shrinkage estimators for uplift regression. In Ulf Brefeld, Elisa Fromont, Andreas Hotho, Arno Knobbe, Marloes Maathuis, and Céline Robardet, editors, Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD’19), pages 607-623. Springer-Verlag, 2019. doi:10.1007/978-3-030-46150-8_36.

[11] Piotr Rzepakowski and Szymon Jaroszewicz. Decision trees for uplift modeling with single and multiple treatments. Knowledge and Information Systems, 32(2):303–327, 2012. doi:10.1007/s10115-011-0434-0.

Bibtex

@article{sacscuza:jaroszewicz23seilur,
  title={Shrinkage Estimators for the Intercept in Linear and Uplift Regression},
  author={Szymon Jaroszewicz, Krzysztof Rudaś},
  journal={Scientific Annals of Computer Science},
  volume={33},
  number={1},
  organization={Alexandru Ioan Cuza University, Ia\c si, Rom\^ania},
  year={2023},
  pages={35-52},
  publisher={Alexandru Ioan Cuza University Press, Ia\c si},
  doi={10.7561/SACS.2023.1.35}
}