The New England Journal of Statistics in Data Science

The principles set forth in our editorial can be remembered easily by the acronym ATOM: Accept uncertainty and be Thoughtful, Open, and Modest. The acronym extends to ATOMIC because of the need for Institutional Change. Briefly, the principles are as follows: Meng’s challenge to statisticians is to be trustworthy, to “deliver what they promise.” That promise refers to statistical procedures doing what they say they are doing – reliably attaining the stated coverage and error rates. In our view, the promise extends further to all of Meng’s suggestions and, indeed, to the entire enterprise of statistics. For example, the promise entails clearly communicating what the statistical results reveal and openly and modestly acknowledging what remains uncertain. The promise also involves equipping nonstatisticians to do likewise through, for example, the sorts of educational changes advocated by Meng to enhance understanding and appreciation of uncertainty.

When Meng calls on us to double the variance – in the right circumstances – he is calling on us to deliver on all four of the principles in ATOM. He cites the need for our statistical “products” to do what they say they do, and his example of product expiration dates is a helpful illustration. Opening a container of spoiled food will quickly sour one on future purchase of that product. Ninety-five percent confidence intervals that fall well short of 95% coverage leave a sour trail of scientific results that do not replicate. Recognizing and addressing this requires the modesty to acknowledge the limitations of our statistics. Meng’s call to ensure “quality at every step” – to practice “quality-guaranteed statistics” – requires us to be thoughtful. That is, care is required when choosing to double the variance, just as care is required for every fork [5] we choose on the path to deriving and presenting statistical results. And quality requires transparency (openness). Reporting and justifying our strategies for dealing with variance – perhaps by doubling the variance in certain situations – is “a reminder of always being transparent about the data and process that lead to our findings.” Of course, the very reason for doubling the variance is an acceptance that uncertainty is part of the quality guarantee.

Meng’s notion of principled corner-cutting – to “dirtify Bayes,” for example – calls on us to be thoughtful about the important properties of the problem at hand. Where can we give up a little (in the “Car Talk” example, we are missing the false negative rate) and still get a reasonable answer? Meng’s three parts to principled corner-cutting, which distinguish between “good, bad and ugly studies,” encapsulate a thoughtful approach whereby we understand and acknowledge the tradeoffs in the choices that we make in any data analysis. As he also notes, however, most of us were not explicitly trained in principled corner-cutting and are not teaching it. Institutional change – reforms to how we teach statistics and what we emphasize in introductory classes – can bring us all closer to feeling comfortable with giving up a little (in the right places) in order to take an analysis further.

Quality introspection, or as Meng delightfully describes it, not selling what we refuse to buy, reminds us to be thoughtful, open, and modest. He provides ideas for reward systems to bring about the institutional change needed, especially in publications. When we analyze data and publish the results, do we stand by what we write? Here, Meng calls on us to question what we do and think carefully about the implications of our analysis choices – to not “impose on others risks we are not willing to take ourselves.” Meng provides an “introspection checklist” to encourage thoughtfulness – to help us think about the quality of our data and data analysis. This is a crucial step towards both openness and modesty; it is harder to be immodest while fully and candidly recognizing and acknowledging the shortcomings of our work. On the other hand, as Meng also rightly observes, such introspection is not currently rewarded by the incentive systems under which many (most?) of us work. He proposes changes that can be made at the institutional level. For example, universities can change how they evaluate publications for decisions about promotion, tenure, and salary increases. One of Meng’s radical suggestions is to reduce salary by a percentage corresponding to the p-value significance cutoff used in a paper for findings that are later found to be incorrect. While admittedly extreme, the idea of shifting the incentives to quality control (and, more broadly, to quality over quantity) holds some appeal. Likewise, regulatory agencies could encourage “self-quality control.”

Still, the biggest institutional change needed is changing the way that we teach statistics. Meng sees the need to “plant the seeds” early in the curriculum. Current practice at the pre-college level – and even beyond – emphasizes rules and increasing mathematical complexity, rather than an understanding of and appreciation for the variability and intrinsic uncertainty that are at the core of our field. Yet, as Meng notes, we can learn to appreciate uncertainty as the “other side of the coin” from information. He provides delightful examples of data visualizations that elementary school children create when left to their own devices and imaginations, and calls for changes to encourage and develop such thinking. If we are to effectively train the next generation to be comfortable – and, even more, fluent – in the languages of uncertainty and variation, it is important to start at the youngest ages. Children are naturally curious about the world around them. They will commonly collect data about their friends’ preferences for ice cream flavors, pets, or pizza toppings, providing an entry point in the curriculum to thinking and teaching about variability and instituting change from the very start of the educational journey.

In summary, we find Meng’s radical ideas stimulating and thoroughly ATOMIC. Meng says we must: deliver on our promises and promise no more than we can deliver; learn that some corners can be cut, albeit in principled ways; be ready to stand by what we write, acknowledging the limitations of our work; and plant seeds by teaching the languages of variation and uncertainty early in and throughout the statistics curriculum. Readers will undoubtedly find much to debate in Meng’s paper, and that is surely his intention. But whether one agrees with doubling variance, for example, the goals behind that suggestion – that our statistics deliver what they claim to deliver, that there is quality at every step, etc. – are goals that we can all get behind.

We have argued elsewhere [7] that statistical practice has evolved over many decades such that dichotomizing p-values and declaring results significant/not significant does far more harm than good and should be abandoned. We welcome further debate about that proposal, which because it has been widely mischaracterized, we emphasize again is about abandonment of dichotomized p-values, not continuous p-values. Rather than focusing our discussion of Meng’s paper on whatever differences exist between us and him, we have concentrated, instead, on the principles advanced by him and their substantial overlap with our ATOMIC principles. Before suggesting some best practices for adhering to the ATOMIC principles, we will raise a few questions about practices for adhering to Meng’s principles and the implications of those principles for practice.

On single studies, hypothetical replications, and soft elimination On corner cutting and the choice of $\boldsymbol{\alpha }$ On promises, tradeoffs, and disclaimers On false negatives, the pufferfish/selfish test, and pandemics

In addition to the radical but thoroughly ATOMIC ideas presented by Meng at the Symposium on Statistical Inference and in his subsequent paper, many ATOMIC ideas – not always as radical – were proffered by other symposium presenters and by authors of papers in the special issue of The American Statistician. As we have had the opportunity to speak to groups from many disciplines about the ATOMIC principles, six relatively easy-to-implement practices have seemed to resonate. They are not all as radical as some of Meng’s, but they are in the spirit of delivering on what we promise and adhering to the ATOMIC principles: Debate about these proposals and Meng’s is needed. While often radical but sometimes not so radical, Meng’s paper makes an important contribution to the debate by calling on the statistics community to challenge itself to use statistical tools better and communicate better what the results do – and do not – say.