Normal distribution
The normal distribution is an extremely important probability distribution considered in statistics. Among people whose field is not primarily probability theory or statistics (notably in physics) it is often called the Gaussian distribution. It is actually a family of distributions of the same general form, differing only in their location and scale parameters: the mean and standard deviation. The standard normal distribution is the normal distribution with a mean of zero and a standard deviation of one. Because the graph of its probability density resembles a bell, it is often called the bell curve.
The normal distribution was first introduced by de Moivre in an article in 1733 (reprinted in the second edition of his Doctrine of Chances, 1738) in the context of approximating certain binomial distributions for large n. His result was extended by Laplace in his book Analytical Theory of Probabilities (1812), and is now called the Theorem of de Moivre-Laplace.
Laplace used the normal distribution in the analysis of errors of experiments. The important method of least squares was introduced by Legendre in 1805. Gauss, who claimed to have used the method since 1794, justified it rigorously in 1809 by assuming a normal distribution of the errors.
The name "bell curve" goes back to Jouffret who used the term "bell surface" in 1872 for a bivariate normal with independent components. The name "normal distribution" was coined independently by Charles S. Peirce, Francis Galton and Wilhelm Lexis around 1875 [Stigler]. This terminology is unfortunate, since it reflects and encourages the fallacy that "everything is Gaussian". (See the discussion of "ocurrence" below).
There are various ways to specify a random variable. The most visual is the probability density function (plot at the top), which represents how likely each value of the random variable is. The cumulative density function is a conceptually cleaner way to specify the same information, but to the untrained eye its plot is much less informative (see below). Equivalent ways to specify the normal distribution are: the characteristic function (related to the cumulant generating function) and the moment generating function. These are very useful for theoretical work, but much less intuitive. See probability distribution for a discussion.
The probability density function of the normal distribution with mean μ and standard deviation σ (or variance σ2) is also known as the Gaussian function
These statements are also true for non-standard normal distributions.
The cumulative distribution function of the normal distribution is the probability that a given standard normal variable has a value less than z. Given the probability density function above, the cumulative distribution function has formula:
For instance, the probability that a standard normal variable has a value less than 0.12 is equal to 0.54776. The cumulative distribution function of the normal distribution does not have an analytic form, and has to be calculated using numerical techniques. It is so commonly used that it is often called "the" error function. A special feature of the normal distribution is that the cumulative distribution function is not needed to simulate normal random variables (see below).
The characteristic function of a gaussian random variable X ~ N(μ,σ2) is defined as the expected value of eitX and can be written as
As a consequence of the first listed property, it is possible to relate all gaussian random variables to the standard normal.
If X is a Gaussian random variable with mean μ and variance σ2, then
The standard normal distribution has been tabulated, and the other normal distributions are simple transformations of the standard one. Therefore, if one knows the mean and the standard deviation of a normal distribution, one can use this table to answer all questions about the distribution.
For computer simulations, it is often necessary to generate values that follow a Gaussian distribution. This is best done with the Box-Muller transforms. These methods require two uniformly distributed values as input which can easily be generated by the computer's pseudorandom number generator.
The Box-Muller transform is a beautiful consequence of the third listed property, and the fact that the chi-square distribution with two degrees of freedom is an exponential random variable (which can be easily simulated exactly).
Approximately normal distributions occur in many situations, as a result of the central limit theorem. Simply stated, this theorem says that adding up a large number of small independent variables results in an approximately normal distribution. Therefore, whenever there is reason to suspect the presence of a large number of small effects acting additively, it is reasonable to assume that observations will be normal. This assumption is then subject to empirical test using well-established statistical methods.
It is important to realize, however, that small effects often act as multiplicative (rather than additive) modifications. In that case, the assumption of normality is not justified, and it is the logarithm of the variable of interest that is normally distributed. The distribution of the directly observed variable is then called log-normal.
Finally, if there is a single external influence which has a large effect on the variable under consideration, the assumption of normality is not justified either. This is true even if, when the external variable is held constant, the resulting distributions are ideed normal. The full distribution will be a superposition of normal variables, which is not in general normal. This is related to the theory of errors (see below).
To summarize, here's a list of situations where approximate normality is expected. For a fuller discussion, see below.
Poisson distribution
The IQ score of an individual for example can be seen as the result of many small additive influences: many genes and many environmental factors all play a role.
The overwhelming biological evidence is that bulk growth processes of living tissue proceed by multiplicative, not additive, increments, and that therefore measures of body size should at most follow a lognormal rather than normal distribution. Despite common claims of normality, the sizes of plants and animals is approximately lognormal. The evidence and an explanation based on models of growth was first published in the classic book
The assumption that linear size of biological specimens is normal leads to a non-normal distribution of weight (since weight/volume is roughly the 3rd power of length, and gaussian distributions are only preserved by linear transformations), and conversely assuming that weight is normal leads to non-normal lengths. This is a problem, because there is no a priori reason why one of length, or body mass, and not the other, should be normally distributed. Lognormal distributions, on the other hand, are preserved by powers so the "problem" goes away if lognormality is assumed.
Repeated measurements of the same quantity are expected to yield results which are clustered around a particular value. If all major sources of errors have been taken into account, it is assumed that the remaining error must be the result of a large number of very small additive effects, and hence normal. Deviations from normality are interpreted as indications of systematic errors which have not been taken into account. Note that this is the central assumption of the mathematical theory of errors.
Because of the exponential nature of interest and inflation, financial indicators such as interest rates, stock values, or commodity prices make good examples of multiplicative behaviour. As such, they should not be expected to be normal, but lognormal.
Mandelbrot, the popularizer of fractals, has claimed that even the assumption of lognormality is flawed.
Other examples of variables that are not normally distributed include the lifetimes of humans or technical devices. Examples of distributions used in this connection are the exponential distribution (memoryless) and the Weibull distribution. In general, there is no reason that waiting times should be normal, since they are not directly related to any kind of additive influence.
Light intensity from a single source varies with time, and is usually assumed to be normally distributed. However, quantum mechanics interprets measurements of light intensity as photon counting. Ordinary light sources which produce light by thermal emission, should follow a Poisson distribution or Bose-Einstein distribution on very short time scales. On longer time scales (longer than the coherence time), the addition of independent variables generates a gaussian distribution. Laser light, which is definitely a quantum phenomenon, has an exactly gaussian light intensity.
History
Specification of the normal distribution
Distribution functions
Probability density function
(see exponential function and pi). If a random variable X follows this distribution, we write X ~ N(μ, σ2). If μ = 0 and σ = 1, we talk about the standard normal distribution, with formula
The picture at the top is the graph of the probability density function of the standard normal distribution. The distribution is symmetric about its mean value. About 68% of the area under the curve is within one standard deviation of the mean, 95.5% within two standard deviations, and 99.7% within three standard deviations (the "68 - 95.5 - 99.7 rule"). The inflection points of the curve occur at one standard deviation away from the mean.Cumulative distribution function
The following graph represents the cumulative distribution function for values of z from -4 to +4: Generating functions
Moment generating function
Characteristic function
as can be seen by completing the square in the exponent.Properties
Standardizing Gaussian random variables
is a standard normal random variable: Z~N(0,1). Conversely, if Z is a standard normal random variable,
is a Gaussian random variable with mean μ and variance σ2. Generating Gaussian random variables
Occurrence
Of relevance to biology and economics is the fact that complex systems tend to display power laws rather than normality.Instances of the central limit theorem
Test scores
Criticisms: test scores are discrete variable associated with the number of correct/incorrect answers, and as such they are related to the binomial. Moreover (see this USENET post), raw IQ test scores are customarily 'massaged' to force the distribution of IQ scores to be normal. Finally, there is no widely accepted model of intelligence, and the link to IQ scores let alone a relationship between influences on intelligence and additive variations of IQ, is subject to debate.Physical characteristics of biological specimens
Differences in size due to sexual dimorphism, or other polymorphisms like the worker/soldier/queen division in social insects, further make the joint distribution of sizes deviate from lognormality. Measurement errors
Financial variables
Lifetime
Photon counts
Further reading
External links and references
