لخّصلي

Statistical Inference
In this chapter, we begin our discussion of statistical inference. Probability theory is primarily concerned with calculating various quantities associated with a probability model.
This requires that we know what the correct probability model is. In applications, this is
often not the case, and the best we can say is that the correct probability measure to use
is in a set of possible probability measures. We refer to this collection as the statistical
model. So, in a sense, our uncertainty has increased, not only do we have the uncertainty
associated with an outcome or response as described by a probability measure, but now we
are also uncertain about what the probability measure is. Statistical inference is concerned
with making statements or inferences about characteristics of the true underlying probability measure. Of course, these inferences must be based on some kind of information,
the statistical model makes up part of it. Another important part of the information will be
given by an observed outcome or response, which we refer to as the data. Inferences then
take the form of various statements about the true underlying probability measure from
which the data were obtained.
Inference includes estimation and hypothesis testing which are discussed in this chapter
and the next one.
SECTION 3.1
Statistical model
In a statistical context, we observe the data s, but we are uncertain about P. In such a
situation, we want to construct inferences about P based on this data.
How we should go about making these statistical inferences is probably not at all obvious.
Common to virtually all approaches to statistical inference is the concept of the statistical
model for the data. This takes the form of a set {Pθ, θ ∈ Θ :} of probability measures, one
35
of which corresponds to the true unknown probability measure P that produced the data
s In other words, we are asserting that there is a random mechanism generating s and we
know that the corresponding probability measure P is one of the probability measures in
{Pθ, θ ∈ Θ :}.
The statistical model {Pθ, θ ∈ Θ :} : corresponds to the information a statistician brings
to the application about what the true probability measure is, or at least what one is willing
to assume about it. The variable θ is called the parameter of the model, and the set Θ is
called the parameter space. Typically, we use models where θ ∈ Θ indexes the probability
measures in the model, i.e., Pθ1 = Pθ2
if and only if θ1 = θ2 If the probability measures
Pθ can all be presented via probability functions or density functions fθ
From the definition of a statistical model, we see that there is a unique value θ ∈ Θ such
that Pθ is the true probability measure. We refer to this value as the true parameter value.
It is obviously equivalent to talk about making inferences about the true parameter value
rather than the true probability measure, i.e., an inference about the true value of θ is at
once an inference about the true probability distribution. So, for example, we may wish to
estimate the true value of θ.
There are two main approaches to estimation; point estimation and confidence interval
estimation. The first one gives an approximate value for the unknown parameter, while the
second gives a an interval that likely contains the value of the parameter.
SECTION 3.2
Point estimation
Point estimation method is based on the notion of estimators, this notion is defined by the
following concepts.
Definition 3.2.1. We call a statistic any function of the data in a sample (X1, ..., Xn).
which is denoted by Tn(X1, ..., Xn). A statistic does not depend on unknown parameters.
Definition 3.2.2. Let X be a random variable whose distribution depends on a parameter
θ, and let X1, X2, . . . , Xn be a size n sampling of X. A point estimator of θ is a statistic
of the form
ˆθ = T(X1, X2, . . . , Xn).
In an application, we want to know how reliable an estimator ˆθ is. or we might have to
choose between two estimators of the same parameter. This leads us to following concepts.
Definition 3.2.3. The bias in an estimator ˆθ of θ is given by E(
ˆθ) − θ whenever E(
ˆθ)
exists. When the bias in an estimator ˆθ is 0, we call ˆθ an unbiased estimator of θ, i.e., T is
unbiased whenever E(
ˆθ) = θ.
36
Unbiasedness tells us that, in a sense, the sampling distribution of the estimator is centered
on the true value θ.
Definition 3.2.4. The mean squared error (MSE) of the estimator ˆθ is given by MSE(
ˆθ) =
E(
ˆθ − θ)
2
).
Clearly, the smaller MSE(
ˆθ) is, the more concentrated the sampling distribution of ˆθ is
about the value θ. Looking at MSE(
ˆθ) as a function of gives us some idea of how reliable
T is as an estimate of the true value of θ.
The following result gives an important identity for the MSE.
Theorem 3.2.1. The mean squared error (MSE) is also expressed by
MSE(
ˆθ) = V ar(
ˆθ) + (E(
ˆθ) − θ)
2
Note that when the bias in an estimator is 0, then the MSE is just the variance and
Sd(
ˆθ) = q
V ar(
ˆθ)
is an estimate of the standard deviation of ˆθ and is referred to as the standard error of the
estimate. As a principle of good statistical practice, whenever we quote an estimate of
a quantity, we should also provide its standard error at least when we have an unbiased
estimator, as this tells us something about the accuracy of the estimate.
Definition 3.2.5. An estimator is said to be convergent (consistent) if the sequence (
ˆθn)
converges in probability to θ :
∀ > 0, P(|
ˆθn − θ| > ) −→n→∞
0.
A Strong convergent (consistent) estimator is one where convergence is almost sure.
If the variance of an estimator tends to zero then, this estimator is consistent. this condition
is sufficient not necessary.
There are several methods for determining point estimators, the method of moments, the
maximum likelihood method and other methods.
3.2.1 Method of moments
In short, the method of moments involves equating sample moments with theoretical moments. So, let’s start by making sure we recall the definitions of theoretical moments, as
well as learn the definitions of sample moments.
37
Definition 3.2.6. Moments(Review)
Let X be a random variable. Then, The k
th moment of X is:
E(X
k
)
and the k
th central moment of X is:
E[(X − E(X))k
]
Usually, we are interested in the first moment of X, µ = E(X), and the second central
moment of X, V ar(X) = E[(X − µ)
2
].
Definition 3.2.7. Sample moments
Let X be a random variable. Let X1, ..., Xn be iid realizations (samples) from X. Then,
The k
th sample moment of X is:
1
n
Xn
i=1
X
k
i
and the k
th central sample moment of X is:
1
n
Xn
i=1
(Xi − X¯)
k
where X¯ is the first sample moment.
For example, the first sample moment is just the sample mean, and the second central
sample moment is the sample variance.
Common estimators are the sample mean and sample variance which are used to estimate
the unknown population mean and variance.
Theorem 3.2.2. (Estimation of µ)
Suppose that the mean µ is unknown. The method of moments estimator of µ based on
(Xn) is the sample mean
µˆ =
1
n
Xn
i=1
Xi = X¯
µˆ is unbiased and consistent estimator.
Proof. X¯ is unbiased because :
E(X¯) = E(X¯) = E