+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 31 to 38 of 38

Thread: Posterior Distribution

  1. #31
    TS Contributor
    Points: 22,410, Level: 93
    Level completed: 6%, Points required for next Level: 940

    Posts
    3,020
    Thanks
    12
    Thanked 565 Times in 537 Posts

    Re: Posterior Distribution




    But will rejection sampling work in this case? How do I know it will not work?
    It will work whenever the support of your target density is a subset of your proposal density. Since now the support of your target density is the whole real line, you also required that the tail of your proposal is thicker.

    How am I able to tell that my probablity of acceptance will be low without using any software packages?
    You need some sort of numerical integration. Since the area under the density is 1, the constant M tells you the proportion of samples accepted.

  2. The Following User Says Thank You to BGM For This Useful Post:

    MrAnon9 (12-15-2011)

  3. #32
    Points: 1,278, Level: 19
    Level completed: 78%, Points required for next Level: 22

    Posts
    135
    Thanks
    13
    Thanked 0 Times in 0 Posts

    Re: Posterior Distribution

    Could you tell me what is the best method for my question and reasoning? I am totally baffled

    Would appreciate it so much!

  4. #33
    Points: 1,278, Level: 19
    Level completed: 78%, Points required for next Level: 22

    Posts
    135
    Thanks
    13
    Thanked 0 Times in 0 Posts

    Re: Posterior Distribution

    I think I worked out that the proposal density does have thicker tails and that m = 0.00004 so the acceptance rate is 1/0.00004 ? :S
    Last edited by MrAnon9; 12-15-2011 at 05:58 PM.

  5. #34
    Points: 1,278, Level: 19
    Level completed: 78%, Points required for next Level: 22

    Posts
    135
    Thanks
    13
    Thanked 0 Times in 0 Posts

    Re: Posterior Distribution

    Hi again, sorry for all the questions but what value of M do I want? and if I don't get this type of value, does this mean I should employ MCMC ?
    Last edited by MrAnon9; 12-15-2011 at 05:00 PM.

  6. #35
    Points: 1,278, Level: 19
    Level completed: 78%, Points required for next Level: 22

    Posts
    135
    Thanks
    13
    Thanked 0 Times in 0 Posts

    Re: Posterior Distribution

    I guess one last shot at asking for help here :P

    I have f(x) < g(x)*M and I have worked out that M is 0.0004 by substituting xhat = log(5/11) into h(x) where h(x) = f(x) / g(x), and also plotted both the target and the proposal density and noticed that the proposal density was a lot "fatter" than the target density, which was thin. But I don't know what this all means, does this mean rejection sampling is working well here?

  7. #36
    TS Contributor
    Points: 22,410, Level: 93
    Level completed: 6%, Points required for next Level: 940

    Posts
    3,020
    Thanks
    12
    Thanked 565 Times in 537 Posts

    Re: Posterior Distribution

    I have not check any of your numerical solution here. But if the claim in your last post is correct, then everything is fine. Even the constant M is not optimal (smallest M that larger then the ratio), the algorithm can still work.

  8. #37
    Points: 1,278, Level: 19
    Level completed: 78%, Points required for next Level: 22

    Posts
    135
    Thanks
    13
    Thanked 0 Times in 0 Posts

    Re: Posterior Distribution

    Thanks for the reply!

    How can i work out my probablity of acceptance? and also how it rejection sampling fine here? I have plotted the two graphs and saw that the density of the proposal is a lot larger and the target is only a small part of it

    Also the acceptance rate is 1/M right? 1/0.00004 = A hugee number, what's that mean and how is that plausible?

  9. #38
    TS Contributor
    Points: 22,410, Level: 93
    Level completed: 6%, Points required for next Level: 940

    Posts
    3,020
    Thanks
    12
    Thanked 565 Times in 537 Posts

    Re: Posterior Distribution


    Since there is a few posts concern about the rejection sampling here maybe I just gives a detailed example here, using your problem.

    Now the posterior density

    f(\theta|x) = \frac {1} {c} \exp\left\{-\frac {\theta^2} {200} + x\theta \right\} (1 + e^{\theta})^{-n} \propto \exp\left\{-\frac {\theta^2} {200} + x\theta \right\} (1 + e^{\theta})^{-n}

    where c \triangleq \int_{-\infty}^{+\infty} \exp\left\{-\frac {\theta^2} {200} + x\theta \right\} (1 + e^{\theta})^{-n} d\theta is the normalizing constant.


    You do not need to know the exact value of c if you are just want the rejection sampling algorithm. However if you want to know "theoretically" the exact acceptence rate you may need to do a numerical integration to get this constant. Of course the acceptance rate can also be estimated by the proportion of acceptance in your simulation.

    Now suppose your proposal density is a normal density

    g(\theta|\mu, \sigma^2) = \frac {1} {\sqrt{2\pi\sigma^2}} \exp\left\{-\frac {(\theta-\mu)^2} {2\sigma^2}\right\} \propto \exp\left\{- \frac {\theta^2} {2\sigma^2} + \frac {\mu} {\sigma^2} \theta \right\}

    Now, dropping the all the constants out, note that

    \frac {f(\theta)} {g(\theta)} \propto \exp\left\{\left(\frac {1} {2\sigma^2} - \frac {1} {200}\right)\theta^2 + \left(x - \frac {\mu} {\sigma^2}\right)\theta\right\} (1 + e^{\theta})^{-n}

    Since we want \lim_{\theta \to \infty} \frac {f(\theta)} {g(\theta)} < +\infty, we require \frac {1} {2\sigma^2} - \frac {1} {200} \leq 0 \iff \sigma^2 \geq 100

    In addtion when \sigma^2 = 100, in this special case we also require 0 \leq x - \frac {\mu} {\sigma^2} \leq n

    With these constraints on the parameters, we differentiate the ratio with respect to \theta:

    \frac {\partial} {\partial \theta} \frac {f(\theta)} {g(\theta)} \propto \exp\left\{\left(\frac {1} {2\sigma^2} - \frac {1} {200}\right)\theta^2 + \left(x - \frac {\mu} {\sigma^2}\right)\theta\right\} (1 + e^{\theta})^{-n-1}
    \times \left[\left(\frac {1} {\sigma^2} - \frac {1} {100}\right)\theta + \left(x - \frac {\mu} {\sigma^2}\right)(1 + e^{\theta}) - ne^{\theta}\right]

    In general you can optimize the ratio by choosing the optimal parameters \mu, \sigma^2. But anyway let say you have chosen \sigma^2 = 100, \mu = 0 which satisfy the above constraint. Then the first order condition

    \frac {\partial} {\partial \theta} \frac {f(\theta)} {g(\theta)} = 0 \iff x - (n - x)e^{\theta} = 0 \iff e^{\theta} = \frac {x} {n - x}

    And you can easily verify that it indeed is the global maximum point of the desired ratio.

    Let say your rejection sampling algorithm is to accept \Theta \sim g when U < \frac {1} {M} e^{x\Theta} (1 + e^{\Theta})^{-n}

    Then such constant M is optimally chosen by

    \left(\frac {x} {n - x}\right)^x \left(1 + \frac {x} {n - x}\right)^{-n} = \frac {x^x(n-x)^{n-x}} {n^n} \approx 4.8333677\times 10^{-5} when n = 16, x = 5
    as you have calculated above .

    Note that in the process we have omit some constants - if we include them back we can calculate the acceptance rate.
    If we accept the sample when U < \frac {\sqrt{200\pi}} {M'c} e^{x\Theta} (1 + e^{\Theta})^{-n}, then the acceptance rate

    \frac {1} {M'} = \frac {c} {\sqrt{200\pi}M}

    Borrowing Dason's code you can also do the simulation and estimate the acceptance rate from R:
    Code: 
    n <- 100000
    t <- rnorm(n, 0, 10)
    u <- runif(n)
    r <- exp(5*t)/(1+exp(t))^16
    M <- 5^5*11^11/16^16
    id.accept <- (u <= r/M)
    samp <- proposals[id.accept]
    accept.rate <- mean(id.accept)
    print(accept.rate)
    P.S.: The constant c should be able to numerically integrated in R, but it seems the integrand overflowed at some numbers.

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats