Severe event class imbalance is observed in many fields, such as insurance fraud, severe weather, traffic safety, and rare disease, and has far-reaching significance when the generalization of minority event classes is of primary interest. While existing solutions mostly focus on differential sampling or sample re-weighting approaches to alleviate the imbalance issue, we take a novel alternative view on promoting generalization. Our proposal is formulated under the generative Bayesian framework, positing that predictors are the stochastic proxies of latent causes with limited samples, whose exceedance leads to extreme events. To accurately capture the extended tail, our solution adopts the Gumbel distribution as prior and is modeled with a variational inference framework. Following the assertion that exceedance leads to extremes, we devise a disentangled additive monotonic neural architecture to predict the risk. The proposed model acknowledges representation uncertainties while embracing improved interpretability, generalization, and robustness. We provide theoretical insights to show the merits of the proposed approach. To verify the effectiveness in empirical settings, we conducted studies on various real-world data against the state-of-the-art counterparts, with encouraging results reported.
Rare event modelingVariational inferenceGenerative Bayesian modelsExtreme value theoryIntroduction
Rare event modeling (REM) deals with situations with an extremely low prevalence of events but influential consequences. Common examples of high social-economic value events include vehicle collisions in traffic safety [10], autonomous vehicle testing [34], life-threatening rare diseases in healthcare, and fraudulent financial activities [32]. Accurate and robust modeling of rare events is critical for the identification of the relevant risk factors so as to predict, and hopefully prevent, associated negative outcomes via intervention. As an example, it is estimated that an adult driver in the U.S. will, on average, travel approximately 6.8 million miles before experiencing a single traffic crash [10]. Given that traffic crashes are relatively rare events compared to normal driving, the precise identification and prediction of such incidents are of utmost importance for reducing fatalities and enhancing overall safety [39].
Characterized by severe event class imbalance and the lack of minority labels, rare event modeling falls outside the comfort zone of standard statistical approaches [46]. Without explicit statistical adjustments, the imbalance drives a learning agent to bias toward the majority class; at the same time, the absence of adequate minority examples causes unprotected models to wrongly capture spurious features that do not generalize. Without adequate generalization ability, the implemented REM model would substantially diminish effectiveness in detecting rare events, which are usually associated with risks that people aim to avoid. In the context of traffic crashes, inaccurate predictions by the REM model could potentially result in severe accidents occurring.
To establish reliability for REMs, the implemented model should have sufficient robustness to handle future out-of-sample predictions for the extreme event, i.e. the data which has not been exposed to during the training of REM. [48] point out that learning from less represented events is vulnerable to data perturbations. These generated misjudgments might lead to safety-critical situations. For instance, if practitioners employ the REM model to assess the safety functions of autonomous vehicles, including decisions about whether to execute maneuvers to prevent traffic crashes based on crash predictions, a less robust REM model could potentially permit self-driving cars to take unforeseen actions, possibly resulting in collisions [5].
Pioneered by the work of [23], there has been extensive research to overcome the potential pitfalls of rare-event modeling in the classical statistical regression framework. Prominent examples include coefficient bias correction [23] and penalized maximum likelihood estimation [14]. However, difficulties arise when applying such correction techniques in real-world settings: (i) the estimation efficiency for rare event modeling is dictated by the number of events of interest given fixed dimensions, and high-dimensional inputs render the conclusions unreliable and ungeneralizable [46, 43]; (ii) heavy tails are commonly expected in rare event data, and most existing solutions fail to account for such characteristics [48].
Statistical re-sampling and re-weighting are the two most popular sample adjustment strategies to mitigate severe event class unbalancing issues. Re-sampling approaches alter the learner’s exposure frequency by over-sampling the minority class and down-sampling the majority class. The re-weighting approach adjusts the relative importance by assigning more weight to the less-represented examples. In line with these practices, recent work represented by [4] and [29] also considers modifications to derive class-sensitive loss that properly penalizes the minority event class, allowing the model to learn efficient representation without getting trapped in the complications associated with sample adjustments.
The re-sampling and re-weighting statistical approaches have limited capability to handle the situation when inputs are outside the norm. Re-sampling typically involves either over-sampling or under-sampling [3]. The former is often subjected to the loss of estimation efficiency [46], the latter is often associated with compromised generalization [4]. The re-weighting schemes are often criticized for the numerical instability [35]. Recent works by [4, 29] modify the hinge loss and entropy loss to capitalize the minority class prediction. However, research points out this might result in overfitting issues due to training bias and label noise [35].
An alternative way to overcome the difficulties of rare event modeling is to solicit inductive bias, thereby imposing structural constraints to suppress the spurious associations and identify robust predictive features. Many such models are framed under the few-shot learning setup, capitalizing on its superior ability to generalize in the small sample regime. These models often operate under relatively strong assumptions that the majority event classes sufficiently characterize the feature encoder and meta-predictors, leaving only a handful of parameters to tune for the minority classes, thereby boosting efficiency and generalization. Sharing an aggregating knowledge across different learning tasks defined in majority classes contributes significantly to the success of few-shot learning on generalization [13]. Few-shot learning relies on the assumption that all classes in the data share common features and leverages knowledge transfer from majority classes to enhance learning for minority classes. However, in reality, this assumption may not always hold true, as different classes can have varying data generation processes. For example, traffic crashes and normal driving segments often display distinct characteristics [16]. The failure to meet the assumptions of few-shot learning methods can potentially lead to a degradation in their performance [11].
Adding to the difficulty is the challenge of discerning spurious correlations as rare events do not always associate with stable, recognizable patterns. Instead, feature irregularities are common for rare events, which makes it unattainable to confidently model the rare event behavior. In such scenarios, rare event modeling can be formulated by “establishing the norm” using the majority of examples and to test if input is out of the ordinary, i.e., the anomaly detection approach with abundant tools developed based on various heuristics [8, 37]. Unsupervised anomaly detection approaches often involve density modeling, which has been a long-standing challenge for high-dimensional inputs [8, 37]
Despite the varying degree of empirical success of existing rare event modeling schemes, a few limitations have been largely overlooked. First, assumptions made by different modeling schemes are often at odds, implying performance depends on whether underlying assumptions have been satisfied. Second, representation has not been sufficiently valued in existing research, potentially taking a toll on the minority class generalization. In light of the above, it is imperative to develop robust and interpretable learning strategies that can fully recognize the unique characteristics of rare-event data.
In recognition of the limitations of current rare-event modeling discussed above, we present a novel learning framework, named Variational Inference for Rare Event Modeling (VI-REM), that explicitly addresses some of the weaknesses of existing schemes. Our main contributions include (i) formulation of a variational representation learning scheme based on information theoretic model, allowing disentangled extreme representations for rare-event prediction; (ii) development of a robust and interpretable prediction arm that joins the strength of a generalized additive model and an isotonic neural net; (iii) presentation of theoretical insights that justify the empirical gains through multiple numerical experiments of VI-REM.
Background
Denote the input features as x∈Rd, the latent variables as z∈Rk, and class label y∈{0,1} for which y=1 represents the rare-event of interest with probability of occurrence p, and y=0 as Normality. Training instances are given by n paired samples D={x(i),y(i)}i=1n with sample size n1 in class Event and n0=n−n1 in class Normality. The goal is to predict the conditional likelihood of an event given the features, i.e., Pr(y|x) with accuracy and robustness. The rare-event problem arises when p≪1.
Information Theoretic Generative Model
The generative Bayesian model has gained popularity due to its scalability, interpretability, and generalizability [24]. Under the full Bayesian setup, the generative process for the data likelihood of y given x can be obtained by marginalization over latent variables z:
Pr(y|x)=∫Pr(y,x,z)dzPr(x)=∫Pr(y|x,z)Pr(z|x)dz
A common assumption in such generative process is z contains sufficient information of x for y, i.e Pr(y|x,z)=Pr(y|z). Similar assumption can be found in [41]. With this, Equation 2.1 can be simplified as
Pr(y|x)=∫Pr(y|z)Pr(z|x)dz
For REM which belongs to supervised learning tasks, the core of the generative Bayesian model is to reconstruct the conditional probability of response variable y given predictors x, which is Pr(y|x), with high accuracy [46, 43].
For REM, maximizing over Equation 2.2 is challenging because of the computational intractability of the integration as well as the difficulty to find the latent z given high-dimensional x.
To address the computational intractability issue for latent z, the Variational Information Bottleneck (VIB) [1] utilizes the information-theoretic methods to find a meaningful representation z from x by maximizing the following equations:
argmaxzI(z,y)−βI(z,x)
where I(z,y) measures the mutual information between z and y, and the parameter β>0 is a hyper parameter.
Under the generative Bayesian setup, maximizing Equation 2.4 is identical to minimize the following loss function based on Variational Inference (VI) [24]:
LossVIB=−DKL(qϕ(z|x)‖Pr(z))+βEqϕ(z|x)[logpθ(y|z)]
where Pr(z) denotes a prior distribution on z, pθ(y|z) denotes a conditional distribution of y given z parameterized by θ, and qϕ(z|x) is a family of distributions parameterized by ϕ to approximate the true posterior density [24].
The first term DKL(qϕ(z|x)‖Pr(z)) is the KL divergence between Pr(z) and qϕ(z|x). It shows the complexity of the posterior distribution. Smaller values mean that qϕ(z|x) is similar to Pr(z), which implies a simpler posterior. The second term Eqϕ(z|x)[logpθ(y|z)] shows how well latent z can reconstruct label y, i.e. the task performance.
The parameter β is used to keep a balance between the performance on the task and the complexity of the latent representation z [50]. If β=1, then Variational Information Bottleneck in Equation 2.4 is the same as maximizing the evidence lower bound (ELBO) of Pr(y|x), given latent variable z [41]; if β≠1 Variational Information Bottleneck will be the β-Variational Auto Encoder (VAE) [21] in the version of supervised learning.
For supervised learning tasks, maximizing over the VIB objective can improve model’s generalizability [41], because the DKL(qϕ(z|x,y)‖Pr(z)) regularizes the Kullback–Leibler (KL) divergence between approximated posterior qϕ(z|x,y) and prior distribution Pr(z) and helps to prevent qϕ(z|x,y) being overfitted because of simplicity in prior distribution. The generative Bayesian model also shows superior robustness over the deterministic models, especially in handling testing data with considerable noises [1].
Challenges and Motivation
When applying the generative Bayesian model for REM, there are three challenges from statistical perspectives. The first challenge is setting an appropriate prior Pr(z) for the less represented events. Within the current state of the art (SOTA) approaches, most work uses the isotropic Gaussian [41, 24] or non-parametric based approaches [45]. However, neither of them could be a good fit in REM, where the events of interest are always associated with heavy tail and rarity of observations.
The second challenge in REM comes from the model’s generalization ability to unseen data [48], as the events of interest do not always associate with robust patterns. The disentanglement learning approach proves its effectiveness in representing information with stability, especially for the patterns that are less stable [2]. Promising work includes generalized additive models (GAM) [19] and β-Variational Auto Encoder (VAE) [21]. A feasible approach for disentanglement is to assume sets of statistically independent variables for modeling [7]. In this work, we follow the work in [21] to assume sets of statistically independent variables assuring the model’s disentanglement in REM. This setup helps to decouple the representations into individual features that each capture a unique aspect of data.
The third challenge arises from how we can build an interpretable prediction model. Isotonic regression, also known as monotonic regression, is an important class of constrained regression techniques that works under the assumption of monotonicity [18, 47]. Research has verified its appropriateness in preventing overfitting and enhancing model interpretability [18]. Specifically, for a scalar predictor x, the relationship of x toward y=g(x) is assumed to be non-decreasing with regard to x; i.e., g(x1)≤g(x2) if x1≤x2. Such relation is ubiquitous in practice, and monotonicity allows extra flexibility compared to linear relations yet retains a similar level of interpretability of the predictive features and modeling robustness. In this paper, we leverage the recent introduction of the unconstrained monotonic neural net (UMNN) [47] to approximate flexible mapping functions in the generative Bayesian model.
Methodology
We formulate VI-REM under the latent variable model framework, in which the event class labels can be predicted from the latent z. The key idea of the VI-REM is to amortize the difficulty of direct prediction of rare-events to the representation learning stage. The premise is the extreme latent representations lead to extreme events. The approach partitions the representation space into normal and extreme regions, where the prevalence of target events in the latter far exceeds those in the former. This alleviates the majority bias issue that afflicts conventional schemes, as the event distribution is more balanced for the extreme region.
Extreme Prior Distribution
The Extreme Value Theory (EVT) studies the statistical behavior of extremes [9]. To model the extreme behavior of statistical features, maximum model within the domain of EVT describes the maximal value in a sequence of independent observations. Modeling extremes poses a challenge due to the limited number of observations. However, extreme value theory (EVT) asserts that, under mild conditions [9], extreme values actually belong to the Generalized Extreme Value (GEV) family of distributions. For a sequence of independent and identically distributed random variables, denoted as X1,X2,…,XN, the asymptotic distribution of the maximum value max(X1,X2,…,XN) follows:
limN→∞Pr(maxX1,…,XN≤x)=GEV(μ1,σ1,ξ)
where GEV(μ1,σ1,ξ) denotes a GEV distribution with μ1 as the location, σ1 as the scale and ξ as the extreme value index. Equation 3.1 is known as the Fisher–Tippett–Gnedenko theorem [9], allowing for accurate modeling of extreme values with very limited examples.
The GEV has three forms depending on the parameter ξ: Fréchet distribution (ξ>0), Gumbel distribution(ξ=0), and reversed Weibull distribution (ξ<0). The Fréchet and Weibull distributions have bounded support, e.g. x>μ1−σ1ξ for Fréchet distribution. To cover the whole space, we use the Gumbel distribution, denoted as G(μ1,σ1), as the prior because the support for Gumbel is x∈R.
In VI-REM, we assume extreme latent in the representation space lead to the happening of Event cases. As these extreme representation usually deviate from the center of normality, we leverage maximum model in Equation 3.1 to represent these extreme cases. We extend standard Gaussian prior [24, 41] via incorporating a Gumbel based prior to accommodate heavy tails. We model the normal regular representations with a Gaussian distribution, and the extreme representation with a Gumbel distribution. We assume the representation space of latent variable of z can be partitioned into the into normal and extreme regions. So that the probability density can be:
Pr(z)=Pr(z∈Normality)×N(μ2,σ2)+Pr(z∉Normality)×G(μ1,σ1)
where N(μ2,σ2) is a Gaussian distribution with parameter μ2, σ2. For simplicity, we set λ=Pr(z∈Normality). We define the extreme prior (EP) as a weighted mixture of a Gumbel distribution and Gaussian distribution in equation 3.2, and its associated probability density function is formulated as:
pκEP(z)=λ2πσ22exp{−(z−μ2)22σ22}+1−λσ1exp{−z−μ1σ1−exp(−z−μ1σ1)}
Illustration of the Extreme Prior distribution. z1 and z2 are simulated independently in the figure. (a) The scatter plot for different EP distributions and the red points come from the Gumbel distribution part and the green points come from the Normal distribution part. For case 1, both of the latent variables z1,z2∼EP(10,2,0.995,0,2). (b) We set both the latent factors z1,z2∼EP(10,0.2,0.995,0,2) and as z1, z2 comes from the same distribution, we only show the results for the probability density estimation for z1. We consider the EP distribution, kernel density estimation (a non-parametric approach), and Normal distribution to fit the distribution of z1. The black line represents the estimated entire distribution functions, and the blue displays the distribution function for the tailed part.
To induce more flexibility into the model, we set the parameters κ=(λ,μ1,σ1,μ2,σ2) of EP prior are all learnable during the optimization process. The parameters μ1, μ2 controls the center of Normal and Gumbel distribution. The parameter σ1 controls the shape of distribution tail. Figure 1a displays the scatter plot for different EP distributions. Theorem 3.1 states that as long as σ1,σ2>0, the proposed EP prior is a proper prior. If the tailed behavior does not actually lead to rarity, the EP can fall back to a standard Gaussian distribution.
The proposed EP distribution is a proper prior, satisfying:
∫z∈RpκEP(z)dz=1
where κ=(λ,μ1,σ1,μ2,σ2) are the parameters for the EP prior and σ1,σ2>0.
Figure 1b compares the performance of different estimators in modeling the tail part of the distributions. The fitted Normal distribution considerably underestimates the tail portion, while non-parametric kernel density estimation tends to overfit the observed samples. In scenarios involving heavy-tailed distributions, which are commonly observed in risk-related analyses [9], traditional parametric methods (such as fitted Normal distribution) and non-parametric methods (like kernel density estimation) may struggle to accurately capture the tail probabilities due to the limited number of rare events. However, extreme value theory-based estimation (EP distribution) excels in handling heavy-tailed scenarios effectively. It not only addresses the challenges posed by limited rare events but also demonstrates substantial generalization ability in accurately describing the tails of distributions.
EP distribution can be generalized to multi-dimensional latent components by applying the above setup to each individual dimension. We assume mutual independence among multi-dimensional latents z∈Rk, so as the EP distribution for z with parameter ω∈R5k can be extended as where zi∈R:
pωEP(z)=∏i=1kpκiEP(zi)
where κi=(λi,μ1,i,σ1,i,μ2,i,σ2,i) and ω=(κ1,…,κk).
The decomposition above assumes that the prior distribution is independent, which implies a possible disentanglement of the latent factor z [21]. One can also consider dependent priors for z. For example, a Gaussian process-based prior Pr(z)=N(0,Kzz) with a kernel matrix Kzz [6] to account for dependence in the prior distribution, or to introduce a hierarchical structure to the prior with Pr(z)=Pr(z1)∏i=2kPr(zi∣zi−1) [42]. However, more complex priors will increase the number of parameters to be optimized, thus to increase the overfitting risk in REM [45]. We focus on the independent prior in this work, and defer more flexible priors to future research.
Monotonic Additive Neural Network
To facilitate interpretability and generalization for REM, it is beneficial to impose constraints on pθ(y|z). We combine two prominent techniques: the generalized additive model (GAM) [19] and isotonic regression for pθ(y|z), and we name it as Monotonic Additive Neural Network (MANN). Let y come from a Bernoulli(p) distribution and ψθ(y|z) be the Logit function and the pθ(y|z) can be the following:
pθ(y|z)=11+exp(−ψθ(y|z))
We assume sets of statistically independent variables [7] in MANN to assure model’s disentanglement. Each fj takes input from the j-th dimension of the latent to model the generation of y. This additive decomposition encourages disentangled representation. The Logit function ψθ(y|z) is based on GAM:
ψθ(y|z)=logpθ(y|z)1−pθ(y|z)=α(θ)+∑j=1kfj(zj)
where each of the individual function is in Equation 3.8, j=1,..k and k is the dimension of latent variables.
fj(zj)=βj(θ)∫0zjexp(hj(t;θ))dt
It is easy to see all fj are monotonic, with the direction dictated by the sign of βj. We model hj(t;θ) above using MLP based deep neural networks, and compute fj(zj) via Clenshaw-Curtis numerical integration [47].
We use a toy examples to illustrate the advantages of monotonicity in dealing with REM. We set y from a Bernoulli distribution and the conditional probability follows
Pr(y|z1,z2)=11+exp(−z13−0.5z23)
where the random variables z1∼EP(2,0.5,0.99,−2,0.5) and z2∼EP(1,0.5,0.99,−1,0.5). Through this, the probability for Pr(y=1)=0.02. We set a 2×500 fully connected layers for both MANN and MLP. From Figure 2, the MANN models shows better fittings for the low data density region.
The marginal relationship of z1, z2 with the Logit value of condition probability, that is log(Pr(y|z1)1−Pr(y|z1)) and log(Pr(y|z2)1−Pr(y|z2)) for z1 and z2. The black points represent the observed samples of the Gumbel part.
Variational Inference Learning
Our VI-REM model combines the MANN and EP prior distribution. We establish VI-REM on the Multilayer Perceptron (MLP), which is flexible to handle different learning tasks. We adopt the setup from VIB [1], the loss function to be minimized for VI-REM is:
L(θ,ϕ,ω)=DKL(qϕ(z|x)||pωEP(z))−βEqϕ(z|x)(logpθ(y|z))
where β is empirically set to be n0/n1 for models to be better optimized. pωEP(z) is the EP distribution with parameters ω.
The qϕ(z|x) is a family of distributions to approximate the true posterior density [24]. Practitioners usually refer qϕ(z|x) as Encoder structure as it reduces the original high-dimensional features x into lower dimensions z. The qϕ(z|x) can be set as isotropic Gaussian distributions through:
qϕ(z|x)=N(z∣μϕ(x),σϕ2(x)×I)
where I is the identity matrix. The parameters μϕ(x),σϕ2(x)=gϕ(x) and gϕ(x) is established based on MLP parameterized by ϕ. This is a standard choice in literature of VI [25]. The usage of isotropic Gaussian distributions imply the individual components of the posterior qϕ(z1|x), ..., qϕ(zK|x) are mutually independent [25]. For more complex cases with dependent posterior, we can use distributions with tractable likelihood, such as normalizing flows [36] or inverse autoregressive flows [26]. Our results show that isotropic Gaussian distributions perform well. We use them in the following analysis. logpθ(y|z) is based on MANN parameterized by θ:
logpθ(y|z)=−log{1+exp(−α(θ)−∑j=1kfj(zj))}
Updating the parameters θ, ϕ, ω in VI-REM is based on stochastic gradient descent. Updating the parameter ϕ is the same as the strategies in [41, 24]. As for parameters ω∈R5k for pωEP(z) and we calculate the derivatives with Equation 3.5:
∇ωL(θ,ϕ,ω)=∇ωDKL[qϕ(z|x)||pω(z)]=−Eqϕ(z|x)(∑i=1k∇ωlogpκi(zi))
The parameters λ1,…,λk∈(0,1) determine the weight for the Gumbel distribution within the EP prior. Their derivatives can be calculated as follows:
∇λiL(θ,ϕ,ω)=−Eqϕ(zi|x)(∇λipκi(zi)pκi(zi))=−Eqϕ(zi|x){h1,i−h2,iλih1,i+(1−λi)h2,i}
where i=1,…,k, and k is the dimension of the latent variable z; zi is the ith component of latent z; λi is the parameter λ for latent zi; h1,i=12πσ2,i2exp(−(zi−μ2,i)22σ2,i2); and h2,i=1σ1,iexp(−(zi−μ1,i)σ1,i−exp(−(zi−μ1,i)σ1,i)).
During the training process, parameter λi might collapse to zero due to sparse parameter regime. To avoid this issues, we set parameter λi through a logistic function as λi=11+exp(−Λi). This transformation enables us to update the weight for λi∈(0,1) by updating the real-valued parameter Λi∈R. The logistic function helps prevent the parameter λi from collapsing to zero. To obtain λi, we utilize the chain rule on Equation 3.12 to compute the derivatives of ∇ΛiL(θ,ϕ,ω). The effectiveness of this approach is confirmed through numerical studies in Sections 5 and 6.
The calculation of derivatives θ involves the derivatives of each individual function fj(zj) in Equation 3.8. Based on Leibniz integral rule, we can have:
∇θfj(zj)=∫s0zj∇θexp(hj(t;θ))dt+∇θβj(θ)
Based on Equation 3.13, the derivatives of the Logit function ψθ(y|z) with respect to θ can be obtained through:
∇θψθ(y|z)=∇θα(θ)+∑j=1k∇θβj(θ)+∑j=1k∫s0zj∇θexp(hj(t;θ))dt
With z∼qϕ(z|x), the derivatives w.r.t parameters θ are:
∇θL(θ,ϕ,ω)=Eqϕ(z|x)[∇θlogpθ(y|z)]=Eqϕ(z|x)[exp(−ψθ(y|z))1+exp(−ψθ(y|z))×∇θψθ(y|z)]
With the optimized parameters θ, ϕ through the introduced learning process, we can generate the predictions for Pr(y|x) through a Monte Carlo estimation where L is number of generated samples, where z(l)∼qϕ(z|x):
Pr(y|x)=Eqϕ(z|x)(logpθ(y|z))≈1L∑l=1Lpθ(y|z(l))
Theoretical Justifications of VI-REM
We theoretically discuss modeling the tail in REM can improve generalization power. The generalization power of Pr(y|x) involve qϕ(z|x) and pθ(y|z) as in Equation 3.16. The asymptotic behavior of qϕ(z|x) through VI has been extensively discussed in [49]. This paper focuses on pθ(y|z).
We use generalization gap to measure model’s generalization ability, which is defined as the difference between the model’s performance on training data and on unseen testing data. Denote Pn(g(z)) and Pr(g(z)) respectively as the empirical measure and underlying ground truth for some function g(z). Define F as the set of all monotonic functions, let f be the candidate function belonging to function space F, and f0∈F is the ground truth. We use Lf(z,y) to denote the loss measured by function f taking value at z for the response y. We identify Ωη≜{‖z‖<η} as an η-ball where the bulk of the distribution mass resides, i.e., Pr(Ωη)≥1−ϵ, ϵ≪1, and consequently ΩηC≜Rd∖Ω contains the extremes.
We point out the theoretical generalization gap with the introduction of cutoff points in Theorem 3.2. The following regularized assumptions are needed to establish Theorem 3.2:
The function f(z) follows the Lipschitz property:
|f(z)−f(z′)|≤kn‖z−z′‖
where kn is some positive constant.
The ground truth function f0 satisfies the condition:
f0=argminf∈FE(Pr(Lf0(z,y)))
Function f(z) satisfies the condition for nonzero z:
supf∈F|f(z)−f0(z)|≤supf∈F‖f−f0‖‖z‖
Under the assumptions in C.1–C.3 described above, the following generalization gap holdsE(supf∈F|Pn(Lf(z,y))−Pr(Lf0(z,y))|)≤log|F|+log(1/δ)2n+γh1/2η1/2supf∈F‖f−f0‖z∈Ωη1/2+γsupz∈ΩηC‖z‖1/2(1−h)1/2supf∈F‖f−f0‖z∈ΩηC1/2whereδ=|F|e−2nϕ2and|F|is the cardinality of spaceF. ϕ, γ are some constant,Iz∈Ωη:||z||≤ηis parameter λ in EP distribution, and h is the empirical value ofIz∈Ωη:||z||≤η.
Through Theorem 3.2, we decompose the generalization gap into estimation error, the approximation error for the non-tailed part, and the approximation error for the tailed part over a certain cutoff point as stated in the following Remark.
Assumef1,f2∈Fare two functions belonging the spaceF. We setf1as thelogpθ(y|z)which is obtained by maximizing Equation3.9. Similarly, we setf2aslogpθ′(y|z′)is obtained by optimizingβEqϕ′(z′|x)(logpθ′(y|z′))−DKL(qϕ′(z′|x)∣pGauss(z′))wherepGauss(z)is a isotropic Gaussian distribution. DenoteΔΩηC=||f2−f0||z∈ΩηC1/2−||f1−f0||z∈ΩηC1/2andΔΩη=||f1−f0||z∈Ωη1/2−||f2−f0||z∈Ωη1/2. As long as the follows holds:ΔΩηC/ΔΩη>η1/2rsupz∈ΩηC‖z‖1/2(h1−h)1/2where r is some constant. Then based on conditions C.1–C.3 and Theorem3.2, the generalization gap over functionf1,f2can have the following:E(supf1∈F|Pn(Lf1(z,y))−Pr(Lf0(z,y))|)<E(supf2∈F|Pn(Lf2(z′,y))−Pr(Lf0(z′,y))|)
Remark 1 supports that the proposed extreme prior can improve model generalization. The misrepresented tailed part in REM amplifies the approximation error, thus restricting the model’s generalizability. Intuitively from Figure 1b, utilizing the heavy-tailed prior distribution remedies the misrepresentation problem, thus helps to tighten the generalization bound.
Model ImplementationsModel Interpretation
The interpretation of VI-REM, i.e., the relationship between x and label y, needs to be made through latent factor z. The MANN component in recognizing the pθ(y|z) makes the relationship between latent factors z and response y easy to assess. However, interpreting the relationship between x and z is challenging due to the black-box neural network. We frame the model interpretability as a variable selection process, i.e., identifying the most relevant x for each of the latent factors.
The core of this interpretation approach is the evaluation of the statistical dependence between x and z. Recent studies suggest that distance correlation [44] is a powerful non-parametric tool to measure the statistical correlation between two random variables. The MANN structure guarantees that the latent factors z in VI-REM are largely statistically independent. Based on this fact, we can use the Distance Correlation [44] to select the critical x in determining z1, ...zk.
Considering the high dimensionality of the input features x, we adopt the Distance Correlation Sure Independence Screening (DC-SIS) framework [27], which is the SOTA approach to use Distance Correlation to select high-dimensional features. The essence of a DC-SIS is to rank variables based on the Distance Correlation. Under our setup, we can use the Distance Correlation between zi and each of the input features x1, ..., xd to rank the importance of input features and select the most relevant ones. However, when directly using the DC-SIS to the proposed VI-REM model, the O(n2) computational complexity for Distance Correlation is intolerable for a large sample size [28], especially when hundreds of thousands of training data. Therefore, we adopted a Divide and Conquer algorithm [28] to overcome the computation challenge.
For latent factor zj, we equally spilt the n samples into G groups with s=n/G data points in each subgroup. For each subgroup g∈[1,…G] with s samples, we measure the Distance Correlation of xlg and zjg, where xlg represents the input feature xl within subgroup g and zjg represents the latent factor zj within subgroup g. For simplicity, denote u=xlg and v=zjg, du, dv as the dimension for u and v, and u(i), v(i) as the i th sample of u and v correspondingly. The detailed formula for estimation of distance correlation is as follows:
DCˆlg=dcorrˆ(xlg,zjg)=dcovˆ(u,v)dcovˆ(u,u)dcovˆ(v,v)
where
dcovˆ2(u,v)=Sˆ1g+Sˆ2g−2Sˆ3g
and Sˆ1g, Sˆ2g, Sˆ3g for dcovˆ2(u,v) are as follow:
Sˆ1g=1s2∑i=1s∑j=1su(i)−u(j)duv(i)−v(jdvSˆ3g=1s3∑i=1s∑j=1s∑l=1su(i)−u(l)duv(j)−v(l)dvSˆ2g=1s2∑i=1s∑j=1su(i)−u(j)du1s2∑i=1s∑j=1sv(i)−v(j)dv
For each subgroup, we conduct the DC-SIS [27] to select the set with the most contribution of input features x toward the latent factor zj. In practice for each of the subgroup g, the selection process for a given size q<s=n/G can be:
Dˆg=l:DCˆlgis among topqlargest ofd,
where l∈[1,…,d] and d is the dimension of input features x.
After conducting these G (in total we have G subgroups) independent calculations and selections, the final component of zj denoted by Dˆzj is established by: Dˆzj=∩g∈{1,…G}Dˆg. In practice, we can simply uses the most common selected relevant features x1, ..., xd in each of the subgroup procedures. Repeat the above process for all zj and calculate the Dˆzj; i.e., the identified component of latent factors.
DenoteDzj∗as the true components associated with latentzj, andDˆzjas the set identified by the previously mentioned model’s interpretation strategy. Then:PrDzj∗⊆Dˆzj≥{1−O{dexp[−c1nα−2α(u+v)]+dnαexp(−c2nv)}}G→1whereG=n1−α,α>0, d is the dimension of input featuresx,c1,c2, u, v are some constant andu+v<1/2.
Theorem 4.1 shows that as long as the sample size n is getting sufficiently large, the true risk modulators for the learnt latents have high confidence to be selected by the given procedure latent factor identification.
Robustness Evaluations
A robust prediction model should be less vulnerable to attacks and remain valid when testing data is out of domain [5]. Building a robust model is critical for deploying deep learning models in the application of safety-critical situation [48]. Robustness measures the model’s ability to handle data perturbations and helps understand the boundary of successful applications. Mathematically for a model f and its predictions over xf(x), robustness evaluations aims to find how the model f can handle f(x′), where x′ has minor perturbations to x. if f(x′) does not show much difference with f(x), then model f is robust and trustworthy to deploy [48].
Researchers have proposed various approaches under different application scenarios to generate x′ to evaluate robustness. Regarding natural language processing tasks, [51] summarizes the current well-established methods and suggests mixing the words from different documents [22, 51] could be a simple and efficient approach. For computer vision studies, image blurring and image rotation are all useful in practice. In this work, we use the rotation operator to investigate whether the model can efficiently extract invariant features from rare events.
Extension to General Setups
The previous VI-REM model mostly focuses on binary classification, and we generalize it to multi-label classification in risk analysis. Suppose we have L+1 classified labels, and we treat one dominant class as class Normality, the rest L minority classes as Event. For example, traffic crashes are rare and can be classified from the most severe fatal crashes to injury crashes to minor property-damage only crashes. To mimic the general settings in risk analysis, we assume the severity of the L minority class ranges from low to high.
Without generality, we set 1, ..., L as labels for the L minority classes, and class 1 as the least severe Event and class L is the most severe one. As the response variable, the severity of an event increases from class 1 toward class L and belongs to the ordinal variables, we applied the setup from the Ordered Logit Model (OLM). Under the OLM setup, we want to model Pr(y≥m|x) given different event severity where m=1,,,.L. So that under the VI-REM, we modify the Logit function in Equation 3.6 through:
ψθ(y≥m|z)=logpθ(y≥m|z)1−pθ(y≥m|z)
and the Logit function in the OLM based VI-REM follows:
ψθ(y≥m|z)=αm(θ)+∑j=1kfj(zj)
where the coefficients αm(θ) are intercepts for different event categories and 0<α1(θ)<⋯<αL(θ). The individual function fj(zj) function follows the same as Equation 3.8.
Experiment Setup
We considered the following machine learning approaches for comparison: (1) Sampling MLP, short as S-MLP, is a re-sampling scheme based on Multilayer Perceptron (MLP) over-samples from the less-represented class [3]. The hyperparameter r in the S-MLP is used to control the over-sample ratio, which is equal to the expected n1 over n0 after over-sampling. We use grid search cross-validation to select the sampling ratio r∈(0.01,0.05,0.1,0.2). (2) Focal MLP, short as F-MLP, uses a re-weighting based loss that adapts the cross entropy loss putting more penalization on the less-represented class [29] based on MLPs. We use grid search cross-validation to select the hyperparameters α∈(0.25,0.5,0.75), γ∈(2,5). (3) MAML is a few-shot learning scheme from [13] and it focuses on improving the generalization performance for less represented data samples and we use 5-shot learning.
We also consider the Deep-SVDD [37], short as D-SVDD, which is a unsupervised anomaly detection approach. D-SVDD will train a MLP based neural network to represent the majority of the data. The data points falling outside the majority will be considered as outliers. We also consider GBDT, a tree based classifier optimizing objective function through a multiple boosting process, and it is a good fit for REM [31].
In terms of the evaluation metrics, we use the F1 score to evaluate the model’s classification ability, and we set 0.5 as the decision threshold. Another adopted evaluation metric is the Precision-Recall AUC (PR-AUC), which focuses on the trade-off between precision and recall ability [20]. Besides the F1 and PR-AUC score for classification ability, we also care about the model’s ability with uncertainty prediction as the model results provide support for decision-making [40]. We use the negative log likelihood (NLL), Brier score [40]. Considering the extreme imbalanced label in REM, we modify the Brier score as the Brier Skill Score (BSS) in the following:
BSS=1−1n∑i=1n(Pr(y(i)|x(i))−y(i))21n∑i=1n(1n∑iny(i)−y(i))2
where Pr(y(i)|x(i)) is predicted probability of label y(i) and BSS∈[0,1]. Similar as the BSS, we modify the NLL with the relative NLL (short as R-NLL) given the imbalanced label:
R‐NLL=∑i=1nNLL(Pr(y(i)|x(i)),y(i))∑i=1nNLL(1n∑iny(i),y(i))
where NLL(Pr(y(i)|x(i)),y(i)) measure the NLL between Pr(y(i)|x(i)) and label y(i).
Benchmark Dataset
We investigated VI-REM’s generalization performance on unstructured image and natural language processing tasks and evaluated robustness. We conducted the experiments via Pytorch on an NVIDIA GeForce RTX 4080 Laptop GPU. To ensure fair comparisons and convenience of visualization, we set the number of latents to 2 in all experiments. As the two considered benchmark datasets are well established in practice, we design an ablation study to understand the functionality of each component in VI-REM. For model VI-REM-V1, we alternate the EP distribution with the Gaussian distribution. Model VI-REM-V2 further replaces the MANN component with the fully connected MLPs. Model VI-REM-V3 alternates the weight β=n0/n1 in Equation 3.9 with a small number. For simplicity, we set the weight β=5 in VI-REM-V3.
Benchmark Dataset: Imbalanced Digit Recognition
Learned representations via VI-REM under different tasks on original testing. Red points represent class Event, green points represent class Normality. Red points represent the class Event, green points represent the class Normality.
We applied the VI-REM to MNIST1
http://yann.lecun.com/exdb/mnist/
dataset. The MNIST dataset covers handwritten digit numbers 0 to 9, corresponding to class 0 to class 9 as the label. Following a similar setup in [3], in each task, one of the ten classes is set as Event, while the samples from the remaining nine classes represent Normality. In the model training part, 100 images are randomly selected from the Event class, and nearly 600,000 images are selected from Normality class. As the selection of Event cases involves randomness, we repeat the experiment ten times for each task. In total, we conduct 100 individual runs of experiments. For model testing part, we create balanced testings consist of 5,000 images from the class Event and 5,000 images from the class Normality. Regarding robustness evaluations, we applied the random rotation operator to the original testing data. Each of the original testing images will rotate at a different angle, and the rotated angle is a random value from a uniform distribution [−60∘,60∘].
For VI-REM, the experiment sets 500×500×2 fully connected (FC) MLPs for qϕ(z|x) and 500×500×500 for MANN in recognizing pθ(y|z). We used Leaky ReLU for the activation function for both qϕ(z|x) and pθ(y|z). For S-MLP, F-MLP and MAML, we set the neural network to be 500×500××2×500×500×500×2 FC layers; for D-SVDD, we set the neural network to be 1000×1000×1000×2 FC layers. For the GBDT model, we set the maximum depth of GBDT to be 10 and number of estimators in GBDT to be 50.
Digit Number Detection Overall Performance Comparison.
R-NLL↑
BSS↑
PR-AUC↑
F1 Score↑
Original
GBDT
0.28
−0.49
0.75
0.24
MAML
1.32
0.47
0.98
0.82
S-MLP
0.74
0.31
0.98
0.78
F-MLP
0.83
0.40
0.98
0.81
D-SVDD
0.38
−0.80
0.60
0.01
VI-REM
1.13
0.51
0.96
0.86
VI-REM-V1
1.25
0.54
0.95
0.86
VI-REM-V2
0.63
−0.22
0.78
0.45
VI-REM-V3
0.55
−0.43
0.70
0.32
Rotation
GBDT
0.24
−0.76
0.65
0.24
MAML
0.23
−0.82
0.69
0.12
S-MLP
0.31
−0.50
0.77
0.33
F-MLP
0.25
−0.70
0.78
0.22
D-SVDD
0.38
−0.81
0.54
0.01
VI-REM
0.87
0.21
0.89
0.71
VI-REM-V1
0.34
−0.45
0.72
0.36
VI-REM-V2
0.26
−0.75
0.65
0.17
VI-REM-V3
0.23
−0.92
0.57
0.05
↑ for higher is better and ↓ for lower is better.
Figure 3 shows the learned representations via VI-REM under different tasks under the original testing. It’s clear that all the class Event deviate from the distribution and mostly lie on the tail part. This pattern is consistent with the EP prior from the VI-REM model. Table 1 reports the average of the 100 experiment results for both original testings and robustness evaluations. The proposed VI-REM model consistently outperformed the competing models regarding model accuracy and robustness. Under random rotation tasks, the VI-REM shows significantly improved model robustness compared to other SOTAs. Through ablation study, the MANN structure largely increases the model’s generalization ability under original testings, while the EP distribution contributes much larger in robustness.
Digit Number Detection Overall Performance Comparison with Different Number of Latent Variables.
Dim
R-NLL↑
BSS↑
PR-AUC↑
F1 Score↑
Original
2
1.13
0.51
0.96
0.86
3
0.33
0.31
0.96
0.77
5
0.48
0.56
0.96
0.87
Rotation
2
0.87
0.21
0.89
0.71
3
0.13
−0.42
0.77
0.38
5
0.13
−0.35
0.77
0.43
↑ for higher is better and ↓ for lower is better.
The dimension of latent variable is an important hyper parameter in the VI-REM. To examine how the latent variable dimension affects the performance of our model, we vary it from 2 to 3 and 5. The results are reported in Table 2, and the column “Dim” displays the dimension for the latent variables. We find that setting the dimension of the latent variable can generate satisfying results on both original testing and rotation testing.
Benchmark Dataset: Imbalanced News Detection
This experiment applied the VI-REM to the standard natural language processing (NLP) benchmark dataset AG News Classification.2
The AG News dataset covers various news articles from “Sports”, “World”, “Business”, and “Sci/Tech” these four different categories. For implementing the applications, we used pre-trained fastText [33] and SWEM [38] to extract raw text representations. For the original input text data, we firstly apply the pre-trained fastText model to transform raw word input into the word embedding, where we set the dimensionality to 300. Based on the extracted word embedding, we used the average operator from SWEM [38] to combine word embedding and generate a 300×1 vector for an individual news article. In terms of robustness evaluations, we randomly select 20% words from class Event and class Normality, and exchange them. Figure 4 gives an illustrative example of the data processing and robustness evaluations.
We created an extremely imbalanced text classification task. For the training task, we treated 30,000 “Sports” news articles as Normality and 100 randomly selected news articles from other categories as Event types. Within each task, we repeat the random samplings for Event types ten times to alleviate randomness. In task 1, we use the news articles from the “World” category as class Event while in task 2 and 3, we set “Business” and “Sci/Tech” separately as Event types. For out-of-sample model performance evaluation, 1,900 news articles for each category made up the testing sample. For VI-REM, we set a 500×500×2 FC neural layer for qϕ(z|x) and 500×500×1 for the individual function fj(zj) of MANN in pθ(y|z). For S-MLP, F-MLP and MAML, we set the neural network as 500×500×2×500×500×1 FC layers; for D-SVDD, we set a 1000×1000×1000×2 network.
Table 3 reports average performance under the original testings and robustness evaluations, and VI-REM shows better generalization ability and more robust model performance in dealing with data perturbations. The MANN structure largely increases the model’s generalization ability through the ablation study in Table 3. We also check how the model performances will change when setting a different number of latent dimension in Table 4.
Illustration of Robustness Evaluations for Imbalanced News Detection.
News Detection Model performance Comparisons.
Method
R-NLL ↑
BSS ↑
PR-AUC↑
F1-Score ↑
Original
GBDT
0.06
−0.61
0.81
0.33
MAML
0.60
0.44
0.99
0.82
F-MLP
0.34
0.43
0.99
0.83
S-MLP
0.35
0.43
0.98
0.83
D-SVDD
0.45
−0.69
0.60
0.02
VI-REM
1.64
0.58
0.98
0.87
VI-REM-V1
1.43
0.52
0.98
0.84
VI-REM-V2
0.68
0.20
0.95
0.63
VI-REM-V3
0.50
−0.03
0.92
0.50
Mixture
GBDT
0.05
−0.95
0.66
0.06
MAML
0.34
0.02
0.97
0.65
F-MLP
0.11
−0.49
0.86
0.39
S-MLP
0.12
−0.44
0.85
0.43
D-SVDD
0.42
−0.75
0.57
0.01
VI-REM
0.92
0.24
0.95
0.74
VI-REM-V1
0.80
0.15
0.95
0.69
VI-REM-V2
0.42
−0.18
0.87
0.47
VI-REM-V3
0.32
−0.38
0.85
0.32
↑ for higher is better and ↓ for lower is better.
News Detection Overall Performance Comparison with Different Number of Latent Variables.
Dim
R-NLL↑
BSS↑
PR-AUC↑
F1 Score↑
Original
2
1.64
0.58
0.98
0.87
3
1.06
0.42
0.96
0.85
5
0.91
0.06
0.99
0.27
Mixture
2
0.92
0.24
0.95
0.74
3
0.65
0.04
0.91
0.70
5
0.69
−0.17
0.95
0.19
↑ for higher is better and ↓ for lower is better.
Real Data Applications
Two real data applications are considered. First application is to detect the fraudulent credit card transactions. The second application identifies traffic crash events based on driving data.
Application Case 1: Fraud Detection
We implemented the VI-REM model on the fraud detection applications3
https://www.kaggle.com/mlg-ulb/creditcardfraud
where fraudulent credit card transactions are the rare events of interest. The dataset to which the VI-REM was applied consists of about 284,000 financial transactions associated with 28 predictive features, among which only 492 examples are labeled as Event resulting in an extremely low prevalence of 0.2%. We adopted a 5:5 train test split. To further evaluate model robustness in handling massive data noise, we created an artificially noisy setting by injecting Gaussian noise N(0,1) to the testing data, labeled as “Noisy 1” in Table 5. Further, we impose Gaussian noise N(0,2) to testing data and we refer it as “Noisy 2” in Table 5. The neural network structure is the same as Imbalanced News Detection.
Figure 5a and Figure 5b display the learnt representations for the fraud detection dataset in the original testing and noisy testings. Figure 5a and Figure 5b confirm that for the VI-REM model, the learnt representation for the Event cases lie on the tailed part while Normality mostly lie in the center.
The latent representation by different model in testing Set for Fraud Detection. PCA is short as Principal Component Analysis. For VI-REM model, the latents comes from z∼qϕ(z|z). The red points represent the fraud transactions, the green points represent the normal ones.
Fraud Detection Model Comparison.
Method
R-NLL ↑
BSS ↑
PR-AUC↑
F1-Score ↑
Original
GBDT
0.54
0.42
0.64
0.71
MAML
1.06
−0.31
0.71
0.49
F-MLP
2.23
0.63
0.79
0.79
S-MLP
0.89
0.56
0.76
0.77
D-SVDD
1.09
−0.01
0.03
0.00
VI-REM
3.72
0.68
0.80
0.81
Noisy 1
GBDT
0.07
−5.49
0.36
0.17
MAML
0.54
−2.16
0.67
0.26
F-MLP
0.76
−0.61
0.68
0.44
S-MLP
0.31
−1.56
0.46
0.32
D-SVDD
0.99
−0.01
0.02
0.00
VI-REM
2.83
0.57
0.72
0.73
Noisy 2
GBDT
0.02
−30.2
0.33
0.04
MAML
0.30
−4.46
0.53
0.17
F-MLP
0.28
−2.82
0.41
0.22
S-MLP
0.13
−5.10
0.21
0.14
D-SVDD
0.55
−0.26
0.01
0.00
VI-REM
1.32
0.16
0.53
0.56
↑ for higher is better and ↓ for lower is better.
Table 5 reports results for both original and noisy settings. The proposed VI-REM model consistently outperformed the competing models in terms of accuracy and robustness. Under noisy testing tasks, which sometimes can be associated with data quality issues in real-world settings, the VI-REM shows significantly improved model robustness compared to all SOTAs. VI-REM’s supreme performance on robustness to data noises indicates it can learn ubiquitous discriminative information. We notice that under “Noisy 1” testings, our model VI-REM still has reliable prediction performance while the other competitors are like a random guesses.
Next, we investigate the relationship between latents z and label y. Figure 6a plots the decision boundary and associated predicted probability for determining the Event cases under original testings. It is clear that the larger distance with the distribution center, the more likely it will be class Event. Figure 6b details the relationship between latent factors z1, z2 and label y. Latent z1 has a negative relationship with the occurrences of the events while z2 has a positive relationship.
We identify the relationship between input features x and latent z. We consider the 28 individual features and the interaction of any two of the input features. In total, 281+282=406 combinations are considered for model interpretations. We implement the strategy discussed in Section 4.1 where we q=5 for each round of selections. Table 6 reports the major components of latent factor z1 and z2 respectively, where V22 represents the 22th variable of the input features. The most contributed part of z1 and z2 do not overlap, confirming the disentanglement. We also fit a linear regression between z1 and z2 and their selected major components, and it is reported as column “Coefficients” in Table 6. For example, the fitted coefficient between the combination of V16, V24 and latent z1 is 0.024, indicating they have a positive relationship.
Model Interpretations for Latent Factor z1 and z2 in Application Case 1 and Application Case 2.
Selected Variable
Coefficient
Case 1
Latent Variable z1
V22×V26
−0.001
Latent Variable z1
V16×V24
0.024
Latent Variable z1
V16×V18
−0.041
Latent Variable z1
V13×V24
−0.006
Latent Variable z1
V16×V23
0.011
Case 1
Latent Variable z2
V3×V13
0.010
Latent Variable z2
V7×V13
−0.002
Latent Variable z2
V13×V19
0.040
Latent Variable z2
V3×V16
0.008
Latent Variable z2
V13×V20
0.023
Case 2
Latent Variable z1
acc-x26×acc-y26
−0.747
Latent Variable z1
acc-z26×acc-z29
−2.042
Latent Variable z1
acc-z26×acc-z28
0.133
Latent Variable z1
acc-z25×acc-z27
−0.973
Latent Variable z1
acc-z26×acc-z30
1.896
Model interpretations for VI-REM in fraud detection. (a)The decision boundary and associated predicted probability for determining the Event cases under original testings. (b) Latent factors plot. Part (1) and part (3) of displays the probability density functions of latent variable 1 and latent variable 2, where the red lines represent the Event class and the green lines represent the Normality class.Part (3) and (4) presents the marginal relationship of the latent factors z1, z2 with the Logit value of condition probability, that is log(Pr(y|z1)1−Pr(y|z1)) and log(Pr(y|z2)1−Pr(y|z1)) for z1 and z2.
Application Case 2: Traffic Crash Identification
Traffic crashes are rare events, with an average rate of one crash 6.8 per million vehicle miles traveled in the US [10]. The naturalistic driving study (NDS) provides an unprecedented opportunity to evaluate crash risk [10]. NDSs are characterized by continuously recording driving information, such as three-dimensional Inertial Measurement Unit acceleration and multi-channel video recordings. Accurately identifying crashes with robustness benefits further understanding of driving behavior with the overall goal of reducing traffic accidents [10], identifying high risky segments at road network [52] or improving the validation efficiency for autonomous vehicles [34].
This experiment applied the VI-REM to the Second Strategic Highway Research Program (SHRP2) NDS [10], which is largest NDS to-date, with more than 1 million hours of continuous driving data. SHRP2 Insight website4
https://insight.shrp2nds.us/
gives a more detailed introduction. We applied the VI-REM to identify traffic crashes based on the high-frequency longitudinal, lateral and vertical tri-axial acceleration for the vehicles.
SHRP2 crash events have four different severity levels [10], ranging from level 4 to level 1. Level 4 (L4) crash events are low-risk tire strikes, level 3 (L3) crash events are minor crashes, level 2 (L2) crash events are police reportable crashes with at least 1,500 dollars worth of damage, and level 1 (L1) are fatal crash events. Our data consists of 100 level 1, 150 level 2, 578 level 3, and 588 level 4 crash events. After a random selection process, we also collected nearly 120,000 safe driving segments from SHRP2. These selected safe driving segments have similar maximum and minimum speeds with crashes. Table 7 summarizes the detail of different events and the last column calculates the imbalance ratio.
Summary Description of Level 1 to Level 4 Crash Events and Normal Drivings.
Description
Number
Ratio
L1 Crash
Fatal Crash
100
1200:1
L2 Crash
Police Reportable Crashes
150
800:1
L3 Crash
Minor Crash and MC
578
208:1
L4 Crash
Low-risk Tire Strikes
588
204:1
Normal Driving
Routine Safe Driving
119,996
1:1
We applied the ordinal-based VI-REM model to the SHRP2 data, as the label of events has different severity levels. To standardize the data, we put the most volatile time in the middle of each driving segment. Each driving segment has 5.1 seconds and as the collection frequency for acceleration is 10 HZ, so the length for each segment is 5.1×10=51. As the acceleration has three directions, the dimensionality for the input features will be 51×3=153. Figure 7a shows several driving segments. Our experiment adopted a 5/5 split for training and testing, and the neural network structure for each model is the same as Imbalanced Digit Recognition.
Several plots for identifying traffic crash events in SHRP2 data (a) Several driving segments are associated with the three-dimensional accelerations under different event categories. (b) Part (1) and Part (2) represent the kernel density estimation of latent z1 and z2 given different event categories. Part (3) and Part (4) display the associated probability of different event categories given different values on z1 and z2.
Our work transforms the assessment of an ordinal model into assessing the performance of a classification model based on different selections on the severity level. For example:
PR-AUC(Prˆ(Y≥j),I(Y≥j))andj∈[L1, L2, L3, L4]
where Y=(y1,…,yN) and Prˆ(Y≥j) is the predicted probability from model in Equation 4.7. Similar transformations will also be applied to the other metrics. Table 8 compares the performances of different models for identifying different severity traffic crashes, where we skip the Deep-SVDD model as it is difficult to extend to ordinal data conditions. Table 8 shows VI-REM’s supreme performances over other competitive models, especially for severe level 1, level 2 and level 3 crash events. Overall for identifying crash events (level 4 up to level 1), our proposed VI-REM can have 0.77 in F1 score, while the most competitive F1 score from benchmark model is 0.68 by Sampling MLP.
Analytical modeling for identifying traffic crashes is sensitive to kinematic noise (such as signal connection issues or different driving environments), where model performance can largely deteriorate as noise increases [30]. Considering this, where model robustness is fundamentally important for decision making, we created an artificially noisy setting where Gaussian noise N(0,0.05) was injected to the input accelerations. The model performance metrics are presented in Table 9. As the table shows, the proposed VI-REM significantly outperforms alternative models based on the evaluation metrics. Compared with benchmarks, the VI-REM shows much more robust disincentive power under noisy testing conditions.
As for the model interpretations, Figure 7b shows the mutual relationship between z and label y. z1 has larger discriminative over different crash events, and larger values in z1 are associated with more considerable severity. We implement the same strategy in application case 1 to identify the relationship between input features x and latent z. We report the identified major components of z1 in Table 6, as latent z2 does not have strong distinguishing ability. In Table 6, acc-x26 represents the longitudinal acceleration at point 26, and acc-y, acc-z represent the lateral and vertical acceleration respectively. All the major contributing input features are around the middle of the driving segment through the model interpretations. This phenomenon is consistent with our experiment setup. We also find that vertical accelerations could be the most influential factor for identifying traffic crashes and determining their associated severity level. This observation is consistent with various empirical studies [30]
Performance Comparisons in Traffic Crash Identifications.
Method
R-NLL ↑
BSS ↑
PR-AUC↑
F1-Score ↑
VI-REM
Y≥L4
1.58
0.58
0.70
0.77
Y≥L3
2.50
0.52
0.69
0.68
Y≥L2
3.32
0.59
0.77
0.74
Y≥L1
2.71
0.56
0.78
0.71
GBDT
Y≥L4
0.65
−0.23
0.49
0.52
Y≥L3
0.48
−0.86
0.27
0.31
Y≥L2
0.62
−0.61
0.38
0.36
Y≥L1
0.94
−0.21
0.50
0.42
MAML
Y≥L4
1.91
0.42
0.64
0.56
Y≥L3
2.16
0.44
0.67
0.64
Y≥L2
1.67
0.40
0.65
0.61
Y≥L1
1.42
0.30
0.53
0.49
S-MLP
Y≥L4
1.43
0.26
0.57
0.65
Y≥L3
1.36
0.08
0.63
0.66
Y≥L2
1.68
0.24
0.54
0.54
Y≥L1
1.44
0.09
0.57
0.51
F-MLP
Y≥L4
0.71
−0.01
0.70
0.68
Y≥L3
1.25
0.40
0.65
0.60
Y≥L2
1.15
0.55
0.72
0.68
Y≥L1
0.66
0.49
0.62
0.58
↑ for higher is better and ↓ for lower is better.
Robustness Evaluations in Traffic Crash Identifications.
Method
R-NLL ↑
BSS ↑
PR-AUC↑
F1-Score ↑
VI-REM
Y≥L4
1.48
0.48
0.69
0.73
Y≥L3
2.43
0.47
0.68
0.66
Y≥L2
3.28
0.59
0.77
0.74
Y≥L1
2.57
0.57
0.78
0.72
GBDT
Y≥L4
0.38
−1.09
0.42
0.38
Y≥L3
0.38
−1.42
0.24
0.23
Y≥L2
0.38
−0.84
0.31
0.31
Y≥L1
0.67
−0.64
0.43
0.38
MAML
Y≥L1
1.85
0.40
0.58
0.56
Y≥L2
1.99
0.39
0.64
0.62
Y≥L3
1.60
0.38
0.61
0.60
Y≥L4
1.43
0.30
0.52
0.49
S-MLP
Y≥L4
0.87
0.05
0.58
0.61
Y≥L3
0.66
−0.65
0.62
0.63
Y≥L2
0.87
−0.42
0.45
0.48
Y≥L1
0.63
−1.34
0.45
0.29
F-MLP
Y≥L4
0.55
−0.53
0.64
0.60
Y≥L3
1.00
0.27
0.62
0.60
Y≥L2
1.14
0.55
0.71
0.68
Y≥L1
0.65
0.48
0.61
0.61
↑ for higher is better and ↓ for lower is better.
Discussion on the Running Time
Before concluding our paper, we present the average running time of our method VI-REM and other competitive methods across different tasks in Table 10. Column “CV” represents the task for imbalanced digit recognition in Section 5.1, column “NLP” represents the task for imbalanced detection in Section 5.2, and column “Fraud” represents the task for fraud detection in Section 6.1, and “Crash” represents the task in Section 6.2. Our experiments show that MAML is the most robust among the competitive methods. In terms of efficiency, VI-REM is faster than MAML, especially under the MLP and CV task. We also find that most of the running time for the VI-REM model comes from the MANN part. Designing a more efficient regularized neural networks could be an interesting topic to explore in the future. The GBDT method performs fast under CV and Fraud task, but may experience longer iteration (with an average of 2113 seconds) to converge when dealing with complex text data. Under the OLM setup in Section 6.2, the VI-REM is even faster than S-MLP.
Overall Computation Time across Different Methods on Different Tasks. The unit is ×100 Second.
Model
CV
NLP
Fraud
Crash
GBDT
1.05
21.13
1.01
6.44
MAML
4.82
1.53
2.78
3.79
F-MLP
0.30
0.11
0.35
0.32
S-MLP
0.51
0.14
0.32
3.53
D-SVDD
0.25
0.19
0.32
-
VI-REM
3.17
0.80
2.40
3.04
VI-REM-V1
2.95
0.77
2.10
2.89
VI-REM-V2
0.28
0.20
0.16
0.27
Summary and Discussion
This study proposes a Variational Inference toward extreme approach for rare event modeling (short as VI-REM). Our model induces uncertainty to representation learning through a novel EP distribution and addresses the over-fitting issue through MANN, which leads to a more robust model for rare events. We provide theoretical properties on the efficacy of the proposed model. Extensive empirical experiments are conducted to confirm the model performance over various application scenarios with promising results, especially with respect to model generalization, robustness and interpretability.
The VI-REM framework greatly improves the generalizability and interpretability of rare event modeling, two challenging issues associated with limited numbers of events. The application of VI-REM could accurately depict the risk associated with rare-events and allow the general public and decision-makers to set realistic expectations for rare events. The identification of adverse events at an early stage may allow mitigation of the damage and loss associated with the events. The features identified through the model are crucial for researchers and practitioners to identify the causes of rare events and take proper countermeasures to prevent and reduce the occurrence of future adverse events.
Proof of Theorem <xref rid="j_nejsds071_stat_002">3.2</xref>
As the label y is binary variable, we set a cross entropy loss function, and we can specify Lf(z,y) as:
Lf(z,y)=ylogf(z)+(1−y)log(1−f(z))
So as when y=1 or when y=0, we can have for z, z′|Lf(z,y)−Lf(z′,y)‖≤|logf(z)−logf(z′)|=|log1+f(z)f(z′)−1|≤|f(z)−f(z′)|min(f(z),f(z′))
Based on condition C.1, the function Lf(z,y) follows:
‖Lf(z,y)−Lf(z′,y)‖≤kn′‖z−z′‖
where kn′ is some constant kn′=knmin(f(z),f(z′))
Decompose generalization bound with triangle inequality:
E(supf∈F|Pn(Lf(z,y))−Pr(Lf0(z,y))|)≤E(supf∈F|Pn(Lf(z,y))−Pr(Lf(z,y))|)+E(supf∈F|Pr(Lf(z,y))−Pr(Lf0(z,y))|)
where we know that f0 minimize of the loss function form assumption C.2, then we can have the upper bound for:
E(supf∈F|Pr(Lf(z,y))−Pr(Lf0(z,y))|)≤E((Pr−Pn)(supf∈F|Lf(z,y)−Lf(z,y)|))
where we denote this upper bound to be
E(G(z))=E((Pr−Pn)supf∈F|Lf(z,y)−Lf(z,y)|)
Thus the generalization gap can be transferred as:
E(supf∈F|Pn(Lf(z,y))−Pr(Lf0(z,y))|)≤E(supf∈F|Pn(Lf(z,y))−Pr(Lf(z,y))|)+E(G(z))
And the function G(z) can be decomposed as:
G(z)=(Pr−Pn)supf∈F|Lf(z,y)−Lf(z,y)|×(In++In−)
where In− as the empirical value of Iz∈Ωη:‖z‖<η and In+ as the empirical value of Iz∈ΩηC:‖z‖≥η. It is obvious In++In−=1. Thus the expectation generalization bound can be formed as:
E(G(z))=E((Pr−Pn)supf∈F|Lf(z,y)−Lf(z,y)|}×In+)+E((Pr−Pn)supf∈F|Lf(z,y)−Lf(z,y)|}×In−)=E(G+(z))+E(G−(z))
From Lemma 2 and Lemma 3 in [12] and Equation A.3, we can have the following bound, where j∈[+,−]:
E(Gj(z))≤4kn′Esupf∈F|Pnϵ(f(z)−f0(z))Inj|
where kn′ is some positive constant satisfying Lipschitz property, ϵ is the Rademacher sequence.
Similar to the Lemma 5 proof in [12], Cauchy–Schwarz inequality can bound it further by:
4kn′Esupf∈F|Pnϵ(f(z)−f0(z))Inj|≤4kn′E(PnϵInj)12supf∈F|f(z)−f0(z)|12
Based on the property in Rademacher sequence, we can have the following bound where c is some positive constant:
E(Pnϵ)≤c2/n
Denote In+=h and In−=1−h. Follows the assumption C.3, we can have the bound toward E(G−(z)):
E(G−(z))≤4kn′cn−12‖z‖1/2(1−h)1/2supf∈F‖f−f0‖z∈ΩηC1/2
Similarly, we can have the bound toward E(G+(z)):
E(G+(z))≤4kn′cn−12h1/2η1/2‖f−f0‖z∈Ωη1/2
For simplicity, we set γ=4kn′cn−1/2 and we can have:
E(G(z))≤γh1/2η1/2supf∈F‖f−f0‖z∈Ωη1/2+γsupz∈ΩηC‖z‖1/2(1−h)1/2supf∈F‖f−f0‖z∈ΩηC1/2
And similar to the classification generalization bound:
E(supf∈F|Pn(Lf(z,y))−Pr(Lf(z,y))|)≤log|F|+log(1/δ)2n
where δ=|F|e−2nϕ2 where ϕ is some constant. Combine Equation A.15 and Equation A.16 based on Equation A.7, we can have Theorem 3.2. □
Proof of the Remark <xref rid="j_nejsds071_stat_003">1</xref>
To simplify the proof, we denote the generalization gap for function f as Ef. Then, the difference of the generalization bound by f1, f2 can be expressed as:
Ef1−Ef2≃Δ2−Δ1
where Δ2=γh1/2η1/2ΔΩ. and
Δ1=maxz∈ΩηC(sup‖z‖1/2,sup‖z′‖1/2)γ(1−h)1/2ΔΩηC
As supz∈ΩηC‖z‖1/2 and supz′∈ΩηC‖z′‖1/2 are at the same magnitude, we assume the following condition:
maxz∈ΩηC(sup‖z‖1/2,sup‖z′‖1/2)=rsupz∈ΩηC‖z‖1/2
where r is some constant. So that:
Ef1−Ef2≃γh1/2η1/2ΔΩ−rsupz∈ΩηC‖z‖1/2γ(1−h)1/2ΔΩηC
Intuitively, as long as Δ2<Δ1, the overall model performance will be improved. As long as the Equation 3.18 satisfies, we can have the statement in Remark 1. □
Proof of Theorem <xref rid="j_nejsds071_stat_004">4.1</xref>
For each round of the selections for model interpretation as in Equation 4.4, we have the following from[27]:
Pr{Dz∗⊆Dˆg}≥1−O{dexp[−c1s1−2(u+v)]+d×sexp(−c2sγ)}
where s is the sample size group g, d is the dimension of input features, and c1, c2, u, v are some constant and u+v<1/2.
Thus:
PrDzj∗⊆Dˆzj=Pr{Dz∗⊆Dˆ1}×⋯×Pr{Dz∗⊆DˆG}≥{1−O(dexp[−c1s1−2(u+v)]+d×sexp(−c2sγ))}G
Replacing s=n−α where α>0, then we have Theorem 4.1. □
Acknowledgements
This study is partially funded by the Safety through Disruption (Safe-D) National University Transportation Center, a grant from the U.S. Department of Transportation – Office of the Assistant Secretary for Research and Technology, University Transportation Centers Program.
ReferencesAlemi, A. A., Fischer, I., Dillon, J. V. and Murphy, K. (2016). Deep variational information bottleneck. arXiv:1612.00410.Bengio, Y., Courville, A. and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence35(8) 1798–1828.Byrd, J. and Lipton, Z. C. (2018). What is the effect of importance weighting in deep learning? arXiv:1812.03372.Cao, K., Wei, C., Gaidon, A., Arechiga, N. and Ma, T. (2019). Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems1565–1576.Carlini, N. and Wagner, D. (2017). Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy39–57. IEEE.Casale, F. P., Dalca, A., Saglietti, L., Listgarten, J. and Fusi, N. (2018). Gaussian process prior variational autoencoders. Advances in neural information processing systems31.Chen, R. T., Li, X., Grosse, R. B. and Duvenaud, D. K. (2018). Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems31.Chen, Y., Zhou, X. S. and Huang, T. S. (2001). One-class SVM for learning in image retrieval. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205)134–37. IEEE.Coles, S., Bawa, J., Trenner, L. and Dorazio, P. (2001). An introduction to statistical modeling of extreme values208. Springer. https://doi.org/10.1007/978-1-4471-3675-0. MR1932132Dingus, T. A., Guo, F., Lee, S., Antin, J. F., Perez, M., Buchanan-King, M. and Hankey, J. (2016). Driver crash risk factors and prevalence evaluation using naturalistic driving data. Proceedings of the National Academy of Sciences113(10) 2636–2641.Dong, Q., Gong, S. and Zhu, X. (2018). Imbalanced deep learning by minority class incremental rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence41(6) 1367–1381.Fan, J., Song, R.et al. (2010). Sure independence screening in generalized linear models with NP-dimensionality. The Annals of Statistics38(6) 3567–3604. https://doi.org/10.1214/10-AOS798. MR2766861Finn, C., Abbeel, P. and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning1126–1135. PMLR.Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika27–38. https://doi.org/10.1093/biomet/80.1.27. MR1225212Fortuin, V. (2022). Priors in bayesian deep learning: A review. International Statistical Review90(3) 563–591. https://doi.org/10.1111/insr.12502. MR4524825Guo, F. (2019). Statistical methods for naturalistic driving studies. Annual Review of Statistics and its Application6309–328. https://doi.org/10.1146/annurev-statistics-030718-105153. MR3939523Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H. and Bing, G. (2017). Learning from class-imbalanced data: Review of methods and applications. Expert Systems with Applications73220–239.Han, Q., Wang, T., Chatterjee, S., Samworth, R. J.et al. (2019). Isotonic regression in general dimensions. The Annals of Statistics47(5) 2440–2471. https://doi.org/10.1214/18-AOS1753. MR3988762Hastie, T. J. and Tibshirani, R. J. (1990). Generalized additive models43. CRC press. MR1082147He, H. and Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering21(9) 1263–1284.Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S. and Lerchner, A. (2017). Beta-VAE: learning basic visual concepts with a constrained variational framework. The International Conference on Learning Representations2(5) 6.Jin, D., Jin, Z., Zhou, J. T. and Szolovits, P. (2020). Is Bert really robust? A strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI Conference on Artificial Intelligence348018–8025.King, G. and Zeng, L. (2001). Logistic regression in rare events data. Political Analysis9(2) 137–163.Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv:1312.6114.Kingma, D. P., Welling, M.et al. (2019). An introduction to variational autoencoders. Foundations and Trendső in Machine Learning12(4) 307–392.Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I. and Welling, M. (2016). Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems29.Li, R., Zhong, W. and Zhu, L. (2012). Feature screening via distance correlation learning. Journal of the American Statistical Association107(499) 1129–1139. https://doi.org/10.1080/01621459.2012.695654. MR3010900Li, X., Li, R., Xia, Z. and Xu, C. (2020). Distributed feature screening via componentwise debiasing. Journal of Machine Learning Research21. MR4071207Lin, T. -Y., Goyal, P., Girshick, R., He, K. and Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision2980–2988.Liu, L., Racz, D., Vaillancourt, K., Michelman, J., Barnes, M., Mellem, S., Eastham, P., Green, B., Armstrong, C., Bal, R.et al. (2023). Smartphone-based hard-braking event detection at scale for road safety services. Transportation research part C: emerging technologies146103949.Lusa, L.et al. (2017). Gradient boosting for high-dimensional prediction of rare events. Computational Statistics & Data Analysis11319–37. https://doi.org/10.1016/j.csda.2016.07.016. MR3662388Massoli, F. V., Falchi, F., Kantarci, A., Akti, ., Ekenel, H. K. and Amato, G. (2021). MOCCA: Multilayer one-class classification for anomaly detection. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/tnnls.2021.3130074. MR4442308Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C. and Joulin, A. (2017). Advances in pre-training distributed word representations. arXiv:1712.09405.O’Kelly, M., Sinha, A., Namkoong, H., Tedrake, R. and Duchi, J. C. (2018). Scalable end-to-end autonomous vehicle testing via rare-event simulation. Advances in Neural Information Processing Systems319827–9838.Ren, M., Zeng, W., Yang, B. and Urtasun, R. (2018). Learning to reweight examples for robust deep learning. arXiv:1803.09050.Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International conference on machine learning1530–1538. PMLR.Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E. and Kloft, M. (2018). Deep one-class classification. In International Conference on Machine Learning4393–4402.Shen, D., Wang, G., Wang, W., Min, M. R., Su, Q., Zhang, Y., Li, C., Henao, R. and Carin, L. (2018). Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. arXiv:1805.09843.Shi, L., Qian, C. and Guo, F. (2022). Real-time driving risk assessment using deep learning with XGBoost. Accident Analysis & Prevention178106836.Snoek, J., Ovadia, Y., Fertig, E., Lakshminarayanan, B., Nowozin, S., Sculley, D., Dillon, J., Ren, J. and Nado, Z. (2019). Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems13969–13980.Sohn, K., Lee, H. and Yan, X. (2015). Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems28.Sønderby, C. K., Raiko, T., Maaløe, L., Sønderby, S. K. and Winther, O. (2016). Ladder variational autoencoders. Advances in neural information processing systems29.Sur, P. and Candès, E. J. (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences116(29) 14516–14525. https://doi.org/10.1073/pnas.1810420116. MR3984492Székely, G. J., Rizzo, M. L., Bakirov, N. K.et al. (2007). Measuring and testing dependence by correlation of distances. The Annals of Statistics35(6) 2769–2794. https://doi.org/10.1214/009053607000000505. MR2382665Tomczak, J. and Welling, M. (2018). VAE with a VampPrior. In International Conference on Artificial Intelligence and Statistics1214–1223. PMLR.Wang, H. (2020). Logistic regression for massive data with rare events. In International Conference on Machine Learning9829–9836. PMLR.Wehenkel, A. and Louppe, G. (2019). Unconstrained monotonic neural networks. Advances in Neural Information Processing Systems32.Wu, T., Liu, Z., Huang, Q., Wang, Y. and Lin, D. (2021). Adversarial robustness under long-tailed distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition8659–8668.Zhang, F., Gao, C.et al. (2020). Convergence rates of variational posterior distributions. Annals of Statistics48(4) 2180–2207. https://doi.org/10.1214/19-AOS1883. MR4134791Zhang, Q., Yu, S., Xin, J. and Chen, B. (2022). Multi-view information bottleneck without variational approximation. In ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing4318–4322. IEEE.Zhang, W. E., Sheng, Q. Z., Alhazmi, A. and Li, C. (2020). Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology11(3) 1–41.Zheng, L., Sayed, T. and Mannering, F. (2021). Modeling traffic conflicts for use in road safety analysis: A review of analytic methods and future directions. Analytic Methods in Accident Research29100142.