Estimators¶
One of the main differences between topics of probability and topics in statistics is that in statistics we have some task in mind. While a probability model \(P_X(X \mid \theta)\) is an object of study when discussing probability, in statistics we usually want to do something with it.
The first example that we will consider is to estimate the true, unknown value \(\theta^*\) given some dataset \(\{x_i\}_{i=1}^N\) assuming that the data were drawn from \(X_i \sim p_X(X|\theta^*)\).
Definition
An estimator \(\hat{\theta}(x_1, \dots, x_N)\) is a function of the data (that aims to estimate the true, unknown value \(\theta^*\) assuming that the data were drawn from \(X_i \sim p_X(X|\theta^*)\).
There are several concrete estimators for different quantities, but this is an abstract definition of what is meant by an estimator. It is useful to think of the estimator as a procedure that you apply to the data, and then you can ask about the properties of a given procedure.
Terminology
These closely related terms have slightly different meanings:
The estimand refers to the parameter \(\theta\) being estimated.
The estimator refers to the function or procedure \(\hat{\theta}(x_1, \dots, x_N)\)
The specific value that an estimator takes (returns) for specific data is known as the estimate.
We already introduced two estimators when studying Transformation properties of the likelihood and posterior:
The maximum likelihood estimator: \(\hat{\theta}_\textrm{MLE} := \textrm{argmax}_\theta p(X=x \mid \theta)\)
The maximum a posteriori estimator: \(\hat{\theta}_{MAP} := \textrm{argmax}_\theta p(\theta \mid X=x)\)
Note both of these estimators are defined by procedures that you apply once you have specific data.
Notation
The estimate \(\hat{\theta}(X_1, \dots, X_N)\) depends on the random variables \(X_i\), so it is itself a random variable (unlike the parameter \(\theta\)). Often the estimate is denoted \(\hat{\theta}\) and the dependence on the data is implicit. Subscripts are often used to indicate which estimator is being used, eg. the maximum likelihood estimator \(\hat{\theta}_\textrm{MLE}\) and the maximum a posteriori estimator \(\hat{\theta}_\textrm{MAP}\).
Hint
It is often useful to consider two straw man estimators:
A constant estimator: \(\hat{\theta}_\textrm{const} = \theta_0\) for \(\theta_0 \in \Theta\)
A random estimator: \(\hat{\theta}_\textrm{random} =\) some random value for \(\theta\) independent of the data Neither of these are useful estimators, but they can be used to help clarify your thinking due to their obvious properties.