PDA

View Full Version : question about likelihood function/MLE

messianic
08-04-2010, 09:05 PM
If a random variable Y has the following probability distribution:
P (Y = 1) = p, P (Y = 2) = q, and P (Y = 3) = 1−p−q. A random sample of size
n is drawn from this distribution and the random variables are denoted Y1, Y2, · · · , Yn.

How would you derive the likelihood function for p and q? Would you just sum the probability distributions first?

How would you derive the MLE for p and q?

If someone could get me in the right direction that would be appreciated

BGM
08-05-2010, 12:13 AM
Ok. So now you are given

\Pr\{Y = 1\} = p, \Pr\{Y = 2\} = q, \Pr\{Y = 3\} = 1 - p - q

The probability mass function of Y is
f_Y(y) = \left\{ \begin{matrix} p & ~ {\rm if} ~ y = 1 \\
q & ~ {\rm if} ~ y = 2 \\
1 - p - q & ~ {\rm if} ~ y = 3 \\
0 & ~ {\rm otherwise} \end{matrix} \right.

However, you can write a more condensed product form:
f_Y(y) = \left\{ \begin{matrix}
p^{\frac {(y-2)(y-3)} {2}} q^{-(y-1)(y-3)}
(1 - p - q)^{\frac {(y-1)(y-2)} {2}}& ~ {\rm if} ~ y = 1, 2, 3 \\
0 & ~ {\rm otherwise} \end{matrix} \right.

such that the joint log-likelihood for n sample
L(p,q;y_1, ..., y_n) = a \ln p + b \ln q + c \ln (1 - p - q)

where a= \frac {1} {2} \sum_{i = 1}^n (y_i - 1)(y_i - 2),
b = -\sum_{i = 1}^n (y_i - 1)(y_i - 3),
c = \frac {1} {2} \sum_{i = 1}^n (y_i - 2)(y_i - 3)
= n - a - b

Now we take the partial differentiation with respect to the parameters
and set them equal to 0:
\left.\frac {\partial L} {\partial p} \right|_{p = \hat{p}, q = \hat{q}}
= \frac {a} {\hat{p}} - \frac {c} {1 - \hat{p} - \hat{q}} = 0

\left.\frac {\partial L} {\partial q} \right|_{p = \hat{p}, q = \hat{q}}
= \frac {b} {\hat{q}} - \frac {c} {1 - \hat{p} - \hat{q}} = 0

Simplify them, we have a 2 by 2 linear system in \hat{p}, \hat{q} :
\left\{\begin{matrix} (a+c)\hat{p} + a\hat{q} = a \\
b\hat{p} + (b+c)\hat{q} = b \end{matrix}\right.

Eventually, we have
\hat{p} = \frac {a} {n}, \hat{q} = \frac {b} {n}
Again note a + b + c = n

But note that a, b, c are just counts of the outcomes 1, 2, 3.
So the MLE are just the ratio of the specific counts to the total sample.

messianic
08-05-2010, 04:14 PM
how did you derive the condensed product form?

Dason
08-05-2010, 06:00 PM
Well just look what happens to it when you plug in the values y=1, y=2, y=3. You get what you want. Notice that for y = 1 we want it to be p. If we plug in y = 2 we get p^{(2-2)(2-3)/2} = p^0, if we plug in y = 3 we get p^0 again. It's just being clever about how you set up the exponents so that things cancel and give you what you want.

messianic
08-05-2010, 07:42 PM
if someone could verify my answer to the MLE for p and q that would be much appreciated,

p = (1-q)/2

q = (1-p)/2

BGM
08-06-2010, 12:05 AM
I guess the most important thing here is not the tricky product form.
It just ease my presentation.

In fact, without mentioning that we can also write down the likelihood
function, which is the most important thing you need to know:
the exponents of each outcome in those kind of "multinomial" distribution
are just the observed frequency for each outcome.

PS: Sorry not quite understand the follow up post.
Anyway it is an easy linear system and you can solve it to get
p = q = \frac {1} {3}

Dason
08-06-2010, 10:25 AM
if someone could verify my answer to the MLE for p and q that would be much appreciated,

p = (1-q)/2

q = (1-p)/2

Those don't really look like estimators to me. Since you don't know either p or q to begin with how would you calculate these once you gather data?