Improving Data Analysis by Testing by Betting: Optional Continuation and Descriptive Statistics

Shafer, Glenn

doi:10.51387/23-NEJSDS55

The New England Journal of Statistics in Data Science

Improving Data Analysis by Testing by Betting: Optional Continuation and Descriptive Statistics

Volume 2, Issue 2 (2024), pp. 215–228

Glenn Shafer

https://doi.org/10.51387/23-NEJSDS55

Pub. online: 13 December 2023 Type: Statistical Methodology

Open Access

Accepted
2 December 2023

Published
13 December 2023

Abstract

When testing a statistical hypothesis, is it legitimate to deliberate on the basis of initial data about whether and how to collect further data? Game-theoretic probability’s fundamental principle for testing by betting says yes, provided that you are testing the hypothesis’s predictions by betting and do not risk more capital than initially committed. Standard statistical theory uses Cournot’s principle, which does not allow such optional continuation. Cournot’s principle can be extended to allow optional continuation when testing is carried out by multiplying likelihood ratios, but the extension lacks the simplicity and generality of testing by betting.

Testing by betting can also help us with descriptive data analysis. To obtain a purely and honestly descriptive analysis using competing probability distributions, we have them bet against each other using the principle. The place of confidence intervals is then taken by sets of distributions that do relatively well in the competition. In the simplest implementation, these sets coincide with R. A. Fisher’s likelihood ranges.

References

[1]

Alexander, N. (2015). What’s more general than a whole population? Emerging Themes in Epidemiology 12(1) 1–5.

[2]

Augustin, T., Coolen, F. P. A., de Cooman, G. and Troffaes, M. C. M., eds. (2014) Introduction to Imprecise Probabilities. Wiley. https://doi.org/10.1002/9781118763117. MR3236913

[3]

Barnard, G. (1947). Review of Sequential Analysis by Abraham Wald. Journal of the American Statistical Association 42(240) 658–665. MR0020764

[4]

Barnard, G. A. (1949). Statistical inference. Journal of the Royal Statistical Society B 11(2) 115–149. MR0034983

[5]

Berk, R. A. (2004) Regression Analysis: A Constructive Critique. SAGE.

[6]

Berk, R. A. and Freedman, D. A. (2003). Statistical assumptions as empirical commitments. In Law, Punishment, and Social Control: Essays in Honor of Sheldon Messinger, 2nd ed. 235–254. Aldine de Gruyter, Berlin.

[7]

Berk, R. A., Western, B. and Weiss, R. E. (1995). Statistical inference for apparent populations. Sociological Methodology 25 421–458.

[8]

Breiman, L. (1961). Optimal gambling systems for favorable games. In Fourth Berkeley Symposium on Probability and Mathematical Statistics (J. Neyman, ed.) 1 65–78. University of California Press. MR0135630

[9]

Bru, B., Bru, M.-F. and Bienaymé, O. (1997). La statistique critiquée par le calcul des probabilités: Deux manuscrits inédits d’Irenée Jules Bienaymé. Revue d’Histoire des Mathématiques 3 137–239. MR1620388

[10]

Cahusac, P. (2020) Evidence-Based Statistics: An Introduction to the Evidential Approach — from Likelihood Principle to Statistical Practice. Wiley.

[11]

Darling, D. A. and Robbins, H. (1967). Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences of the United States of America 58(1) 66–68. https://doi.org/10.1073/pnas.58.1.66. MR0215406

[12]

Dawid, A. P. (1984). Present position and potential developments: Some personal views. Statistical theory, the prequential approach. Journal of the Royal Statistical Society: Series A 147(2) 278–290. https://doi.org/10.2307/2981683. MR0763811

[13]

Dawid, A. P. (1991). Fisherian inference in likelihood and prequential frames of reference. Journal of the Royal Statistical Society: Series B 53(1) 79–109. MR1094276

[14]

Diaconis, P. and Skyrms, B. (2018) Ten Great Ideas about Chance. Princeton. MR3702017

[15]

Donoho, D. (2017). 50 years of data science. Journal of Computational and Graphical Statistics 26(4) 745–766. https://doi.org/10.1080/10618600.2017.1384734. MR3765335

[16]

Doob, J. L. (1953) Stochastic Processes. Wiley, New York. MR0058896

[17]

Edwards, A. W. F. (1972) Likelihood: An Account of the Statistical Concept of Likelihood and its Application to Scientific Inference. Cambridge. MR0348869

[18]

Ethier, S. (2010) The Doctrine of Chances: Probabilistic Aspects of Gambling. Springer, Berlin. https://doi.org/10.1007/978-3-540-78783-9. MR2656351

[19]

Feller, W. (1950) An Introduction to Probability Theory and Its Applications, 1st ed. Wiley, New York. MR0038583

[20]

Feller, W. K. (1940). Statistical aspects of ESP. The Journal of Parapsychology 4(2) 271–298. MR0004461

[21]

Fisher, R. A. (1952). Sequential experimentation. Biometrics 8 183–187.

[22]

Fisher, R. A. (1956) Statistical Methods and Scientific Inference. Hafner. Later editions in 1959 and 1973. MR0131909

[23]

Fourier, J. (1826). Mémoire sur les résultats moyens déduits d’un grand nombre d’observations. In Recherches Statistiques sur la Ville de Paris et le dÉpartement de la Seine (J. Fourier, ed.). Imprimerie Royale, Paris.

[24]

Fourier, J. (1829). Second mémoire sur les résultats moyens et sur les erreurs des mesures. In Recherches statistiques sur la ville de Paris et le département de la Seine (J. Fourier, ed.) Imprimerie royale, Paris.

[25]

Freedman, D. and Lane, D. (1983). A nonstochastic interpretation of reported significance levels. Journal of Business and Economic Statistics 1 292–298.

[26]

Freedman, D. A. (1991). Statistical models and shoe leather. Sociological Methodology (with discussion) 21 291–358.

[27]

Freedman, D. A. (2008). Randomization does not justify logistic regression. Statistical Science 23(2) 237–249. https://doi.org/10.1214/08-STS262. MR2516822

[28]

Freedman, D. A. (2009) Statistical Models: Theory and Practice, Revised Edition. Cambridge. https://doi.org/10.1017/CBO9780511815867. MR2489600

[29]

Greenwood, J. A. (1938). An empirical investigation of some sampling problems. The Journal of Parapsychology 3(2) 222–230.

[30]

Grünwald, P. (2007) The Minimum Description Length Principle. MIT.

[31]

Grünwald, P., de Heide, R. and Koolen, W. M. (2020). Safe testing. In 2020 Information Theory and Applications Workshop (ITA) 1–54. IEEE.

[32]

Grünwald, P., Ly, A., Pérez-Ortiz, M. and ter Schure, J. (2021). The safe logrank test: Error control under optional stopping, continuation and prior misspecification. Proceedings of Machine Learning Research 146 107–117.

[33]

Hendriksen, A. A. (2017). Betting as an alternative to p-values. Master’s thesis, University of Leiden, under the direction of Peter Grünwald.

[34]

Hexter, R., Pfuntner, L. and Haynes, J., eds. (2020) Appendix Ovidiana: Latin Poems Ascribed to Ovid in the Middle Ages. Harvard.

[35]

Howson, C. and Urbach, P. (2006) Scientific Reasoning: The Bayesian Approach, Third ed. Open Court.

[36]

Jozeau, M.-F. Géodésie au XIXème Siècle: De l’hégémonie française à l’hégémonie allemande. Regards belges (1997). PhD thesis, Université Denis Diderot Paris VII, Paris.

[37]

Kelly Jr., J. L. (1956). A new interpretation of information rate. Bell System Technical Journal 35(4) 917–926. https://doi.org/10.1002/j.1538-7305.1956.tb03809.x. MR0090494

[38]

Leahey, E. (2005). Alphas and asterisks: The development of statistical significance testing standards in sociology. Social Forces 84(1) 1–24.

[39]

Mason, W. M. (1991). Freedman is right as far as he goes, but there is more, and it’s worse. Statisticians could help. Sociological Methodology 21 337–351.

[40]

Matthews, J. R. (1995) Quantification and the Quest for Medical Certainty. Princeton.

[41]

Office of Diversity, U. o. M. Equity & Inclusion (2017). Results of the 2016 University of Michigan Faculty Campus Climate Survey on Diversity, Equity & Inclusion. https://diversity.umich.edu/wp-content/uploads/2017/11/DEI-FACULTY-REPORT-FINAL.pdf

[42]

Pawel, S., Ly, A. and Wagenmakers, E.-J. (2023). Evidential calibration of confidence intervals. American Statistician.

[43]

Ramdas, A., Grünwald, P., Vovk, V. and Shafer, G. (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science 38(4) 576–601. https://doi.org/10.1214/23-sts894. MR4665027

[44]

Rissanen, J. (1989) Stochastic Complexity in Statistical Inquiry. World Scientific. MR1082556

[45]

Royall, R. M. (1997) Statistical Evidence: A Likelihood Paradigm. Chapman & Hall. MR1629481

[46]

Shafer, G. (2007). From Cournot’s principle to market efficiency. In Augustin Cournot: Modelling Economics (J.-P. Touffut, ed.) 55–95. Edward Elgar.

[47]

Shafer, G. (2019). On the nineteenth-century origins of significance testing and p-hacking. Working paper 55, Game-Theoretic Probability Project.

[48]

Shafer, G. (2019). Pascal’s and Huygens’s game-theoretic foundations for probability. Sartoniana 32 117–145.

[49]

Shafer, G. (2020). Game-theoretic foundations for statistical testing and imprecise probabilities. SIPTA School 2020/2021, Slides and videos.

[50]

Shafer, G. (2021). Testing by betting: A strategy for statistical and scientific communication (with discussion). Journal of the Royal Statistical Society: Series A 184(2) 407–478. https://doi.org/10.1111/rssa.12647. MR4255905

[51]

Shafer, G. (2022). That’s what all the old guys said: The many faces of Cournot’s principle. Working Paper 60, Game-Theoretic Probability Project.

[52]

Shafer, G. and Vovk, V. (2001) Probability and Finance: It’s Only a Game. Wiley, New York. https://doi.org/10.1002/0471249696. MR1852450

[53]

Shafer, G. and Vovk, V. (2006). The sources of Kolmogorov’s Grundbegriffe. Statistical Science 21(1) 70–98. See also Working Paper 4, Game-Theoretic Probability Project.

[54]

Shafer, G. and Vovk, V. (2019) Game-Theoretic Foundations for Probability and Finance. Wiley, New York. https://doi.org/10.1002/0471249696. MR1852450

[55]

Shafer, G., Shen, A., Vereshchagin, N. and Vovk, V. (2011). Test martingales, Bayes factors and p-values. Statistical Science 26(1) 84–101. https://doi.org/10.1214/10-STS347. MR2849911

[56]

Troffaes, M. C. M. and de Cooman, G. (2014) Lower Previsions. Wiley. https://doi.org/10.1002/9781118762622. MR3222242

[57]

Vovk, V. (2023). The diachronic Baysian. Working paper 64, The Game-Theoretic Probability Project, http://probabilityandfinance.com.

[58]

Vovk, V. and Wang, R. (2021). E-values: Calibration, combination, and applications. Annals of Statistics 49(3) 1736–1754. https://doi.org/10.1214/20-aos2020. MR4298879

[59]

Wald, A. (1947) Sequential Analysis. Wiley, New York. MR0020764

[60]

Ziemba, W. T. (2015). A response to Professor Paul A. Samuelson’s objections to Kelly capital growth investing. The Journal of Portfolio Management 42(1) 153–167.

Full article Related articles

Open access article under the CC BY license.

Keywords

Game-theoretic probability Game-theoretic statistics Optional continuation Optional stopping Cournot’s principle Fundamental principle of testing by betting Ville’s inequality Descriptive statistics Kelly betting Likelihood Probability forecasting Convenience sample

Metrics

since December 2021

390

Article info
views

Full article
views

119

PDF
downloads

XML
downloads

RSS

Authors

Abstract

References

Export citation

Copy and paste formatted citation

Download citation in file