The New England Journal of Statistics in Data Science

Categorical data are prevalent in almost all research fields and business applications. Their statistical analysis and inference often rely on probit/logistic regression models. For these common models, however, there is no universally adopted measure for performing goodness-of-fit analysis. To this end, [26] proposed a so-called surrogate ${R^{2}}$ that resembles the ordinary least square (OLS) ${R^{2}}$ for linear regression models. The surrogate ${R^{2}}$ used the notion of surrogacy, namely, generating a continuous response S and using it as a surrogate of the original categorical response Y [24, 25, 8]. In this paper, we develop an R package $\mathbf{SurrogateRsq}$ to implement the surrogate ${R^{2}}$ method [43]. The package is compatible with existing model fitting functions (e.g., glm(), polr(), clm(), and vglm()), and its features are exhibited in a wine rating analysis. Our package can be used jointly with other R packages developed for variable selection and model diagnostics so as to form a complete model development process. This process is summarized and demonstrated in a categorical-data-modeling workflow that practitioners can follow. To exemplify an extended utility of the surrogate-${R^{2}}$-based goodness-of-fit analysis, we also use this package to illustrate how to compare different empirical models trained from different samples in the wine rating analysis. The result suggests that the package allows us to evaluate comparability across multiple samples/models/studies that address the same or similar scientific or business questions.

This article proposes an alternative to the Hosmer-Lemeshow (HL) test for evaluating the calibration of probability forecasts for binary events. The approach is based on e-values, a new tool for hypothesis testing. An e-value is a random variable with expected value less or equal to one under a null hypothesis. Large e-values give evidence against the null hypothesis, and the multiplicative inverse of an e-value is a p-value. Our test uses online isotonic regression to estimate the calibration curve as a ‘betting strategy’ against the null hypothesis. We show that the test has power against essentially all alternatives, which makes it theoretically superior to the HL test and at the same time resolves the well-known instability problem of the latter. A simulation study shows that a feasible version of the proposed eHL test can detect slight miscalibrations in practically relevant sample sizes, but trades its universal validity and power guarantees against a reduced empirical power compared to the HL test in a classical simulation setup. We illustrate our test on recalibrated predictions for credit card defaults during the Taiwan credit card crisis, where the classical HL test delivers equivocal results.