As with all powerful ideas, 3-sigma is simple, yet difficult to get your head around. Although there is no short explanation, here's the shortest answer I can provide.
First of all, 3-sigma isn't the stuff of slogans and marketing literature. There are no black-belts in 3-Sigma and it is not a magic pill for certain riches or true love. At its core, 3-sigma refers to a theory of knowledge that gives rise to a methodology for prediction. Once you get it, for whatever purposes, your ability to understand the present and foresee the future, can be dramatically improved.
Three sigma, or three standard Deviations, is a statistical value that can be written using the greek letter sigma. A Standard Deviation is a measuring stick used to describe how data are dispersed around their average.
For what is called a normal distribution, which takes the shape of a nice bell curve, one Standard Deviation encompasses about 68% of all observation data. Two Standard Deviations include about 95% of all observations. And three Standard Deviations encompasses a bit more than 99% of all observations. It actually doesn't matter if the shape of a distribution is pretty, like the bell curve shown above. All that matters is that all measurements you make as an observer of the world, will vary, and understanding why measurements always vary and what that variation means, is the key to understanding how we know the world and how we make predictions about the future.
Three sigma stands out from 1, 2, 4, 5, 6... sigma because it alone represents the boundary point that Walter Shewhart determined can be used to signal the difference between events that are ordinary and predictable and those that are unusual and unpredictable.
At first blush, this idea of a statistical boundary between the ordinary and the extraordinary might seem rather silly and arbitrary. Our common sense tells us that we can easily discriminate between what is special and what is common in our daily experience without resorting to statistical reasoning. Why is this not the case?
Behind the concept of 3-sigma lies a theory of how we know the world. This theory asserts that we are genetically and culturally programmed to see assignable causes for every effect of interest to us. This way of knowing permits us to act upon the world. For every observation of interest, we seek out some button to push or some lever to actuate that will bend the world to our purposes. These buttons and levers are the theories we construct about how the world works.
Our minds tell us that for every problem we experience and every challenge we face, there is always a cause that can be acted upon. The Sun's light and gravitation are causes. My neighbor's barking dog is a cause. Temperature is a cause. The economy is a cause. God is a cause. Whistling in the wind is a cause. Things we observe and things we imagine can be designated as causes. We seek and see causes in all things that interest us. This is how we make predictions about the world and how we guide our actions in the world. We are prediction machines and button-pushers, par excellence.
But do our intuitions about assignable causes serve us well or do they misguide us in our predictions about the future?
In his book, Statistical Method from the Viewpoint of Quality Control, Shewhart shows us that most of what a system (or process) does is a product of interactions that cannot be reduced to one or more assignable causes. Shewhart's un-assignable cause is also called, “common cause“. In a sense, he is telling us that we cannot identify a cause that is in the system because everything in the system is a cause. A system, he says, is irreducible. This means that when we start pushing buttons and pulling levers for observations that are not clearly produced by assignable causes, we only make a system increasingly unpredictable. If we fail to understand the nature of variation we will more likely than not, make a royal mess of things.
Based on a theory of knowledge, Shewhart created a statistical tool called a control chart, for monitoring what a system or process is doing as a whole, over time, and will likely continue doing into the future, barring the introduction of some assignable cause (sometimes called special cause). Some people like to say that Shewhart's control charts are a way to listen to the voice of the process. It is a variation detector that tells us what variation is in the nature of the system we are observing and what variation is special. When assignable causes do start influencing the system, it will leave it's state of control and become unpredictable. He assigns 3-sigma as the signaling point for differentiating cause that can and cannot be assigned.
When an assignable cause is signaled, Shewhart tells us that we can search for that cause in order to remove it and restore the system to its former state of predictability, which is a very good thing, OR we can act to change the system as a whole in ways that make it even more predictable and more suited to our purposes, which is sometimes, an even better thing.
The author is an experienced pharmaceutical blogger.