Lakhasly

Principles of Estimation

1.Random Sample: A set of random variables X1, X2, ..., Xn is called a random sample from a population with probability density function f(x; ?) or cumulative distribution function F(x; ?) if they are independent and identically distributed.Definition of Estimation and Estimator: If we use the value obtained from an estimator function as an approximation for the parameter ?, the obtained number is called an estimate and the random variable of the obtained function is called an estimator for ?.The role of the estimator function is to combine and summarize the sample members to obtain information about the unknown parameter and find its estimate, Draper, N and Smith, H (1998) .Parameter: A parameter is a characteristic of a population that is typically a function of all members of the population.An estimator is a random variable whose probability distribution may or may not depend on the parameter ?, but the parameter ?Estimator: A function u = g(X1, ..., Xn) defined on a random sample is called an estimator.The estimator is also called a statistic.Introduction 1.The values of parameters are usually unknown.is an unknown parameter.2.Here, ?In most cases, random samples are assumed to be independent.3.itself does not appear in the function.

Principles of Estimation

1. Introduction


2. Parameter: A parameter is a characteristic of a population that is typically a function of all members of the population. The values of parameters are usually unknown.


3. Random Sample: A set of random variables X1, X2, ..., Xn is called a random sample from a population with probability density function f(x; θ) or cumulative distribution function F(x; θ) if they are independent and identically distributed. Here, θ is an unknown parameter. In most cases, random samples are assumed to be independent.


4. Estimator: A function u = g(X1, ..., Xn) defined on a random sample is called an estimator. The estimator is also called a statistic. An estimator is a random variable whose probability distribution may or may not depend on the parameter θ, but the parameter θ itself does not appear in the function.

Definition of Estimation and Estimator:
If we use the value obtained from an estimator function as an approximation for the parameter θ, the obtained number is called an estimate and the random variable of the obtained function is called an estimator for θ. The role of the estimator function is to combine and summarize the sample members to obtain information about the unknown parameter and find its estimate, Draper, N and Smith, H (1998) .
Estimates are denoted by lowercase letters such as "u" and estimators are denoted by uppercase letters such as "U".
Example:
Estimate: (x_1,…,x_n )→¯x
Estimator: (X_1,X_2,…,X_n )→X ̄
Example: Let X be a random variable with a Bernoulli distribution with parameter p, i.e., X ~ B(1, p).
Let's consider the functions:
u=X_1+X_2+X_3+X_4 = (X_1+ X_2+ X_3+ X_4)/4
where X₁, X₂, X₃, and X₄ For these Bernoulli observations, the estimator is based on the sample’s success proportion:
u = 3
In this example, p is the unknown parameter. When we use the value obtained from an estimator function as an approximation for the parameter θ, the obtained number is called an estimate. The random variable of the obtained function is called an estimator for θ.
The role of the estimator function is to combine and summarize the sample members to obtain information about the unknown parameter and find its estimate.
In this section, we intend to introduce the properties that are used to compare two estimators. Therefore, at the end of this section, we will be able to choose the best estimator among several estimators. Now, we will explain the properties of an estimator.
Methods of Finding Estimators:

1. Method of Estimating Estimators Based on Error Reduction: To determine a good estimator, we must consider that an estimator is considered good if its error is minimal.
   If U is an estimator of the unknown parameter θ, the error is defined as e = |U - θ|. The error is a random variable. To minimize it, we use its average. This can be done probabilistically; that is, we choose an estimator that has a small probability of error. Other criteria can be defined based on the expected value of the error as follows:

It is clear that the smaller this criterion, the better the estimator. Gauss (1821) considered P = 2 for the simplicity of mathematical calculations, and this led to the definition of the mean squared error, Brown, A (2001).
A) Estimation by Reducing Mean Squared Error in Two Steps:
Reducing Error Using the Concept of Unbiasedness:
If we square the random error and take its expected value, we obtain the mean squared error, denoted by MSE(U).

The smaller the value of MSE, the higher the accuracy of the estimator, Crake, K (1995).

Example: For a random sample, the MSE of the sample mean equals the variance divided by the sample size.
MSE(X ̄ )=E(X ̄-μ)^2=Var σ^2/n
The MSE shows that increasing the sample size reduces the error. It can be easily shown that the following relationship holds for the MSE:
MSE(U)=Var(U)+(E(U)-θ)^2
Prove:
MSE(U)=E(U-θ)^2=E(U-EU+EU-θ)^2
=E(U-EU)^2+E(EU-θ)^2+2E(U-EU)(EU-θ)
=Var(U)+(EU-θ)^2+∘
▭(MSE(U)=Var(U)+(E(U)-θ)^2 )
This relationship shows that the amount of error depends on two positive quantities. By reducing these two quantities, we can reduce the error.

The expression can be made zero by choosing a suitable estimator.
Therefore, we choose U such that EU = θ. This naturally provides a desirable property that we will refer to later.
Definition: An estimate U is said to be suitable for a parameter \ theta if the expected value is equal to the perfect parameter for all possible samples. If this condition does not catch, the estimate is biased. The prejudice of an estimate is the difference between the expected value and the correct parameter. Figure 2 states that different samples have different estimates around the correct value, which reflects the property of justice. This characteristic was not observed in Figure 1. Figure 2 demonstrates the property of unbiasedness. It seems that the increased accuracy in Figure 2 ultimately leads to a precise determination of the true value, which is not apparent in Figure 1.

Example: The sample mean is an unbiased estimator of the population mean μ.
E(X ̄ )=E((∑(i=1)^n▒X_i )/n)=(∑(i=1)^n▒〖EX_i 〗)/n=(∑(i=1)^n▒μ)/n=nμ/n=μ
Example: Show that S^2is an unbiased estimator of σ^2.
S^2=∑(i=1)^n▒(x_i-x ̄ )^2/(n-1)=n/(n-1) (¯(x^2 )-x ̄^2 )
⇒E(S^2 )=n/(n-1) (E(¯(x^2 ))-E(x ̄^2 ))
Ex^2=Varx+E^2 x=σ^2+μ^2
=n/(n-1) (μ^2+σ^2-σ^2/n-μ^2 )=n/(n-1)⋅(1-1/n) σ^2=σ^2

Asymptotically Unbiased Estimator:
An estimator U_n is called asymptotically unbiased for the parameter θ if the bias B_θ(U_n) tends to
〖lim〗┬(n→∞) B_θ (U_n )=∘
Example: Let X1, X2, ..., Xn be a random sample with a defined mean of μ.
U_n=X ̄_n+1/nis an asymptotically unbiased estimator. Because:
B_θ (U_n )=E(X ̄_n+1/n)-μ=μ+1/n-μ=1/n
And since 〖lim〗┬(n→∞) 1/n=∘ as n approaches infinity, U_n is an asymptotically unbiased estimator, Draper, N and Smith, H (1998) .
Zero Unbiased Estimator:
We call an estimator W a zero unbiased estimator if we have:
EW = 0
A zero unbiased estimator for a parameter θ can be obtained in many ways. However, it should be noted that the purpose of defining such estimators is to determine the best estimator for the unknown parameter, Hosmer, D, and Lemeshow, S (2002).
Example: Let X be a random variable with a normal distribution with mean θ and variance σ². The estimator U = X^3 - X is a zero unbiased estimator.
E(U)=E(X^3-X)=E(X^3 )-E(X)=∘-∘=∘

Example: Let X be a random variable with a uniform distribution on the interval (-θ, θ). The estimator U = X is a zero unbiased estimator.

Example: Let X1, X2, ..., Xn be a random sample with mean μ. The estimator is a zero unbiased estimator, Because:
E(U_n )=E(X_i-X ̄ )=EX_i-EX ̄=μ-μ=∘
Class of Zero Unbiased Estimators: Zero unbiased estimators are a class of estimators that can be generated in large numbers. It is sufficient to define one zero unbiased estimator. Let W be a zero unbiased estimator. KW is also a zero unbiased estimator, where K is a constant.
E(KW) = KE(W) = K × 0 = 0
Therefore, the class of zero unbiased estimators can be infinite.

Class of Unbiased Estimators: If U is an unbiased estimator for the unknown parameter θ and W is a zero unbiased estimator, then the class of unbiased estimators is defined as:
A = {U + KW | U is an unbiased estimator and W is a zero unbiased estimator, K is a constant}
E(U + KW) = EU + KEW = θ + K × 0 = θ
The number of members of the class A can be infinite by changing K.

Theorem: If a zero unbiased estimator exists, then the unbiased estimator is unique.
Proof: U₁ and U₂ are both unbiased estimators of the same parameter:
E(U_1 – U_2) = E(U_1) - E(U_2) = θ - θ = 0
Therefore, U_1 – U_2 is a zero unbiased estimator, which contradicts the assumption. Therefore, if an unbiased estimator exists, it is unique.
In the previous section, we showed that in some cases, there are infinitely many unbiased estimators for a parameter. The question arises as to which one to choose. This question can be resolved by showing that if one unbiased estimator exists, then it is unique.
Example: Let X be a random variable with a Poisson distribution with parameter λ. It can be shown that X is the only unbiased estimator for the parameter λ in this distribution.
E(X) = λ
Therefore, if an unbiased estimator exists in the Poisson distribution, it must be X.
Complete Family of Distributions:
As mentioned, zero unbiased estimators increase the number of unbiased estimators. In this section, we will introduce distributions for which zero unbiased estimators do not exist. We call this family of distributions a complete family.
Definition: A family is complete if a function with zero expectation for all θ is zero almost everywhere:
E( h(x)=∘→P_θ (h(x)=∘ )=1 , ∀θ ∈ A
where A is the parameter space, and E and denote the expectation and probability, respectively, Minorsky, V (1959).

Example: If f(x, p) is the density function of a Bernoulli random variable, show that the family of densities below is complete.
{f(x; p) : 0 < p < 1}
Solution: Let us assume that for any function h(x) we have:
E(h(x)) = 0 ⇒ h(0)p + h(1)(1 - p) = 0 ⇒ (h(0) - h(1))p + h(1) = 0
From this equation, we conclude that h(0) = h(1) = 0. Therefore, h(x) = 0 for all values of x. Thus, P(h(x) = 0) = 1. Therefore, the family of densities is complete.

Example: If f(x;p) is the density function of a binomial distribution, show that the family of binomial distributions is complete.
Solution: Let us assume that for any function h(x) we have E(h(x)) = 0, 0 < p < 1 and using the density binomial function, we have:
∑_(x=0)^n▒〖h(x)(█(&n@&x)) P^x (1-P)^(n-x)=∘〗

∑_(x=0)^n▒〖h(x)(█(&n@&x))(P/(1-P) )^x=∘〗

With taking into account that , we have:

∑_(x=0)^n▒〖h(x)(█(&n@&x)) θ^x=∘〗

The resulting equation is a polynomial of degree n in x and is equal to zero. Therefore,
h(x)=(█(&n@&x))=∘ . Since a polynomial of degree n in the interval [0, ∞) must be identically zero, we conclude that h(0) = h(1) = ... = h(n) = 0. Therefore, the family of binomial distributions is complete and all statistics that have this distribution are complete.

Example: If X is a random variable with a Poisson distribution with parameter λ, show that the family of Poisson distributions is complete.
Assume that h(x) is the function that E(h(x)=0, therefore:

The resulting series is zero. Therefore, the coefficients must be zero.
h(0)=h(1)=0...h(x)=0

Example: If X is a random variable with a uniform distribution on the interval ( ), is this family of distributions complete?
Solution: Let us assume that h(x) is a function such that E(h(x)) = 0. Then we have:
∫_0^θ▒〖h(x) 1/θ dx=0〗
∫_0^θ▒〖h(x)dx=0〗 h(θ)=0

Therefore, h(x) = 0 for all x. Thus, the family of uniform distributions is complete.

Under what conditions is the family of densities not complete?

If the parameter space is restricted and the support of the density function is not affected, then the completeness property of the family may not hold. To show that the completeness property of a family does not hold, it is enough to find a function whose expectation is zero.

Example: Let X be a random variable with a normal distribution with mean zero and variance σ². This family is not complete. If we define h(x) = x, then we have:
E(h(x)) = E(x) = 0
Therefore, the family of densities is not complete.

Example: Let X be a random variable with a uniform distribution on the interval ( ). Show that the family of densities is not complete.
If θ > 0, the family of densities is not complete. If we define h(x) = x, then we have:
E(X) = 0

Example: Let X be a random variable with a uniform distribution on the interval (0, θ). Show that the family of densities is not complete if θ > 1.
If we define h(x) as follows:
h(x)={█(&2x-1∘