question about likelihood function/MLE

**messianic** · 08-04-2010 09:05 PM

If a random variable Y has the following probability distribution:
P (Y = 1) = p, P (Y = 2) = q, and P (Y = 3) = 1−p−q. A random sample of size
n is drawn from this distribution and the random variables are denoted Y1, Y2, · · · , Yn.

How would you derive the likelihood function for p and q? Would you just sum the probability distributions first?

How would you derive the MLE for p and q?

If someone could get me in the right direction that would be appreciated

**BGM** · 08-05-2010 12:13 AM

Ok. So now you are given

$\Pr\{Y = 1\} = p, \Pr\{Y = 2\} = q, \Pr\{Y = 3\} = 1 - p - q$

The probability mass function of $Y$ is
$f_Y(y) = \left\{ \begin{matrix} p & ~ {\rm if} ~ y = 1 \\ q & ~ {\rm if} ~ y = 2 \\ 1 - p - q & ~ {\rm if} ~ y = 3 \\ 0 & ~ {\rm otherwise} \end{matrix} \right.$

However, you can write a more condensed product form:
$f_Y(y) = \left\{ \begin{matrix} p^{\frac {(y-2)(y-3)} {2}} q^{-(y-1)(y-3)} (1 - p - q)^{\frac {(y-1)(y-2)} {2}}& ~ {\rm if} ~ y = 1, 2, 3 \\ 0 & ~ {\rm otherwise} \end{matrix} \right.$

such that the joint log-likelihood for n sample
$L(p,q;y_1, ..., y_n) = a \ln p + b \ln q + c \ln (1 - p - q)$

where $a= \frac {1} {2} \sum_{i = 1}^n (y_i - 1)(y_i - 2), b = -\sum_{i = 1}^n (y_i - 1)(y_i - 3), c = \frac {1} {2} \sum_{i = 1}^n (y_i - 2)(y_i - 3) = n - a - b$
which can be readily obtained from your sample.

Now we take the partial differentiation with respect to the parameters
and set them equal to 0:
$\left.\frac {\partial L} {\partial p} \right|_{p = \hat{p}, q = \hat{q}} = \frac {a} {\hat{p}} - \frac {c} {1 - \hat{p} - \hat{q}} = 0$

$\left.\frac {\partial L} {\partial q} \right|_{p = \hat{p}, q = \hat{q}} = \frac {b} {\hat{q}} - \frac {c} {1 - \hat{p} - \hat{q}} = 0$

Simplify them, we have a 2 by 2 linear system in $\hat{p}, \hat{q}$ :
$\left\{\begin{matrix} (a+c)\hat{p} + a\hat{q} = a \\ b\hat{p} + (b+c)\hat{q} = b \end{matrix}\right.$

Eventually, we have
$\hat{p} = \frac {a} {n}, \hat{q} = \frac {b} {n}$
Again note $a + b + c = n$

But note that a, b, c are just counts of the outcomes 1, 2, 3.
So the MLE are just the ratio of the specific counts to the total sample.

**messianic** · 08-05-2010 04:14 PM

how did you derive the condensed product form?

**Dason** · 08-05-2010 06:00 PM

Well just look what happens to it when you plug in the values y=1, y=2, y=3. You get what you want. Notice that for y = 1 we want it to be p. If we plug in y = 2 we get p^{(2-2)(2-3)/2} = p^0, if we plug in y = 3 we get p^0 again. It's just being clever about how you set up the exponents so that things cancel and give you what you want.

**messianic** · 08-05-2010 07:42 PM

if someone could verify my answer to the MLE for p and q that would be much appreciated,

p = (1-q)/2

q = (1-p)/2

**BGM** · 08-06-2010 12:05 AM

I guess the most important thing here is not the tricky product form.
It just ease my presentation.

In fact, without mentioning that we can also write down the likelihood
function, which is the most important thing you need to know:
the exponents of each outcome in those kind of "multinomial" distribution
are just the observed frequency for each outcome.

PS: Sorry not quite understand the follow up post.
Anyway it is an easy linear system and you can solve it to get
$p = q = \frac {1} {3}$