+ Reply to Thread
Results 1 to 13 of 13

Thread: Monte Carlo simulation: how many runs?

  1. #1
    Points: 22, Level: 1
    Level completed: 43%, Points required for next Level: 28

    Posts
    3
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Monte Carlo simulation: how many runs?




    Hi,

    Most papers I read related to Monte Carlo (MC) simulation method say that there is no rule on the minimum number of runs required. However, in book Risk Analysis by D. Vose there is the following formula for the estimation of the true mean:

    n>\left ( \frac{\Phi ^{-1}\left ( \frac{1+\alpha }{2} \right )\sigma }{\delta } \right )^{2}

    n - number of runs
    Ф^(-1) - the inverse of the normal cumulative distribution function
    sigma - standard deviation
    alpha and delta - desired error and confidence

    (the formula is derived based on the distribution of the estimate of the true mean (asymptotically) from the Central limit theorem and the assumption that If Monte Carlo sampling is used, each generated value Xi is an iid random variable).

    The formula is derived for making random draws from a univariate distribution. My question is, would it be still valid if I have to make random draws using MC from a multivariate distribution? If not, is there any way to derive a similar formula for a multivariate case based on the same assumptions as for the univariate case?

    Thank you.

  2. #2
    TS Contributor
    Points: 14,811, Level: 78
    Level completed: 91%, Points required for next Level: 39
    Miner's Avatar
    Location
    Greater Milwaukee area
    Posts
    1,171
    Thanks
    34
    Thanked 405 Times in 363 Posts

    Re: Monte Carlo simulation: how many runs?

    It would be interesting to know when that book was written. The only reason to limit the number of MC simulations is if those simulations take an unacceptable amount of time to run. In the early days, it might take a day to run 10k simulations. Today, I can run a million simulations in 15 seconds. Why limit the number of simulations?

  3. #3
    Super Moderator
    Points: 13,151, Level: 74
    Level completed: 76%, Points required for next Level: 99
    Dragan's Avatar
    Location
    Illinois, US
    Posts
    2,014
    Thanks
    0
    Thanked 223 Times in 192 Posts

    Re: Monte Carlo simulation: how many runs?

    It really depends on what it is you're doing. Of course, Miner is correct if an individual is conducting a MC simulation that involves, for example, a t-test or One-Way ANOVA. However, if you're doing something that involves, say, empirical Bayesian estimation using a technique such Markov Chain Monte Carlo methods (e.g. Gibbs Sampling) in the context of Item Response Theory (IRT), then things can get really computationally expensive. In this case, this really would limit the number of replications in the MC study - it doesn't matter how advanced your desktop computer/software is.

  4. #4
    Devorador de queso
    Points: 95,889, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Monte Carlo simulation: how many runs?

    No matter how advanced our computing power gets we will always be asking questions that require computational power outside the reach of what we can easily do in a "short" amount of time.
    I don't have emotions and sometimes that makes me very sad.

  5. #5
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Monte Carlo simulation: how many runs?

    I ran into this issue recently in multiple imputations. The early recommendations were say to run 5 imputations. Now they recomends hundreds or thousands. The reason is because the results get better the more your run. Old recommendations were based on less powerful computers that took a long time to run each imputation. The trade off is, once you reach a minimum size, between increasing accuracy and time to increase the number run.

    Which is a reason to be cautious about software comments from a decade or more ago. My last computer bought say 3 years ago had 8 gigs of memory (still a lot from the past). The one I got this year has 388 gigs. That makes a lot of difference (has does having 4 processors rather than two I would guess)
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  6. #6
    Devorador de queso
    Points: 95,889, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Monte Carlo simulation: how many runs?

    388 gigs of ram? Are you talking about a server?
    I don't have emotions and sometimes that makes me very sad.

  7. #7
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Monte Carlo simulation: how many runs?

    No a developers computer designed to handle really large queries with massive amounts of data from relational tables very fast. Very expensive, very high end computer (they cost thousands of dollars). Because I am the only one who runs stats in a large state agency they bought it for me.

    The servers I use are entirely different....
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  8. #8
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: Monte Carlo simulation: how many runs?

    I checked with IT and I lied. I have 388 gigs of volitair memory on my hard drive. I only have 10 gigs of RAM....
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  9. #9
    Points: 22, Level: 1
    Level completed: 43%, Points required for next Level: 28

    Posts
    3
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Monte Carlo simulation: how many runs?

    one run for the model I'm working with takes several hours on one of the most powerful computers in our university. So if it's 100 runs, Monte Carlo is feasible, if it's 1000 Runs, then it is not. That's why it is important for me to understand how to determine the minimum number of runs required.

  10. #10
    Devorador de queso
    Points: 95,889, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,937
    Thanks
    307
    Thanked 2,630 Times in 2,246 Posts

    Re: Monte Carlo simulation: how many runs?

    So you're saying the statistic you're interested in is multivariate?
    I don't have emotions and sometimes that makes me very sad.

  11. #11
    Points: 22, Level: 1
    Level completed: 43%, Points required for next Level: 28

    Posts
    3
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: Monte Carlo simulation: how many runs?

    yes, it is multivariate (around 8-10 components)

    So I wonder if the formula I presented can be used to justify the number of runs for the multivariate case.
    Or if there is any rule of thumb to determine the minimum number of draws required depending on the dimension of the multivariate distribution.

    There are papers published with 100 runs for similar problems but this number of runs is never justified by anything but the computational time.

  12. #12
    TS Contributor
    Points: 5,246, Level: 46
    Level completed: 48%, Points required for next Level: 104
    maartenbuis's Avatar
    Location
    Konstanz
    Posts
    372
    Thanks
    3
    Thanked 146 Times in 123 Posts

    Re: Monte Carlo simulation: how many runs?

    What is the purpose of the Monte Carlo simulation? For example, are you trying to check or estimate a p-value or a confidence interval or are you trying to get an idea about the consistency of the point-estimate? What is your model? What estimates from that model are you interested in?

  13. #13
    TS Contributor
    Points: 22,410, Level: 93
    Level completed: 6%, Points required for next Level: 940

    Posts
    3,020
    Thanks
    12
    Thanked 565 Times in 537 Posts

    Re: Monte Carlo simulation: how many runs?


    So if you look back the derivation,

    \Pr\{|\bar{X}_n - \mu| < \delta\} > \alpha where \mu is the true mean

    \Rightarrow \Pr\left\{\sqrt{n}\frac {|\bar{X}_n - \mu|} {\sigma} < \frac {\delta\sqrt{n}} {\sigma} \right\} > \alpha

    \Rightarrow \Phi\left(\frac {\delta\sqrt{n}} {\sigma}\right) - \Phi\left(-\frac {\delta\sqrt{n}} {\sigma}\right) > \alpha (approxmiated by CLT)

    \Rightarrow 2\Phi\left(\frac {\delta\sqrt{n}} {\sigma}\right) - 1  > \alpha

    \Rightarrow \Phi\left(\frac {\delta\sqrt{n}} {\sigma}\right) > \frac {1 + \alpha} {2}

    \Rightarrow \frac {\delta \sqrt{n}} {\sigma} > \Phi ^{-1}\left(\frac{1 + \alpha} {2}\right)

    \Rightarrow n > \left ( \frac{\Phi ^{-1}\left ( \frac{1+\alpha }{2} \right )\sigma }{\delta } \right )^{2}


    So if you want to extend it to a multivariate case where \bar{X}_n will be a p-dimensional random vector instead, then you will need to seek a vector norm to measure the distance between the sample mean and the population mean. One possible distance will be Mahalanobis distance, and we can make use of the result from the multivariate CLT:

    n(\bar{X}_n - \mu)\Sigma^{-1}(\bar{X}_n - \mu) \stackrel {d} {\to} \chi^2(p)

    provided that the random sample itself satisfy the multivariate CLT assumption.

  14. The Following 2 Users Say Thank You to BGM For This Useful Post:

    AlinaN (04-08-2015), GretaGarbo (04-07-2015)

+ Reply to Thread

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats