Bayesian vs. Frequentist

[whomp]

Hi, I am a college student who has taken a course in frequentist probility learned a little bit of Bayesian probability. I don't know enough to compare the two on a philosophical basis, but I used both of them to solve a problem and then compared it to a computer simulation of the problem.

The problem was, if a coin has an unknown probability p of being heads (and 1-p of being tails) on each flip, and after flipping the coin the 14 times the coin comes up heads 10 times, what is the chance that the next two flips are both heads?

The question and its solutions are discussed at this blog: http://www.behind-the-enemy-lines.co...entist-or.html

The answer I came up using billions of trials on my computer is 18 times closer to the answer a Bayesian approach gives than the answer a frequentist approach gives! Does this mean that Bayesian methods are better? What are your guys thoughts in general about the two approaches?

I am very curious to know, because I will be picking one to learn this summer for my work in statistics.

[Dason]

The answer I came up using billions of trials on my computer is 18 times closer to the answer a Bayesian approach gives than the answer a frequentist approach gives! Does this mean that Bayesian methods are better? What are your guys thoughts in general about the two approaches?

How exactly did you do your trials?

Somethings to think about: Does the actual parameter value that you use in your simulation make a difference on which method is preferred? Can you think of a way to analyze the difference between the Bayesian estimate and the frequentist method without necessarily having to do simulations? Which prior did you use? Does this make a difference?

As for your question of whether or not Bayesian methods are better - it highly depends on what you mean by "better". The questions I ask in the previous paragraph hopefully shed some light on that and make you think about it a little more.

You say you don't want to compare them on a philosophical basis and that's alright (hopefully I gave you some ideas on how to compare them on a more quantitative basis) but personally the philosophical basis is the reason I prefer Bayesian methods.

[whomp]

How exactly did you do your trials?

For each set of trials, I picked a p value between 0 and 1. Then, for each trial, I "flipped a coin" 14 times and saw if it came up heads 10 times. If it did, I then tested if the next two flips were heads. The total I got at the end (roughly .486) is the number of times it gets two heads in a row after getting 10 out of 14 divided by the total number of times it gets 10 out of 14. I ran 20,000 sets of trials, and each set of trials consisted of 20,000 trials. The code for the program (it is written in C) is here: http://pastebin.com/mc2F5tgz

Somethings to think about: Does the actual parameter value that you use in your simulation make a difference on which method is preferred? Can you think of a way to analyze the difference between the Bayesian estimate and the frequentist method without necessarily having to do simulations? Which prior did you use? Does this make a difference?

I really have no idea, I just wrote a program to simulate flipping a coin and then compared the results to the results within the blog article I linked in my first post. The blog article says the prior distribution is uniform.

As for your question of whether or not Bayesian methods are better - it highly depends on what you mean by "better". The questions I ask in the previous paragraph hopefully shed some light on that and make you think about it a little more.

You say you don't want to compare them on a philosophical basis and that's alright (hopefully I gave you some ideas on how to compare them on a more quantitative basis) but personally the philosophical basis is the reason I prefer Bayesian methods.

That's fine, but I don't have the knowledge to evaluate them on a philosophical basis, and in the interests of time I can only learn one. Since there are clear quantitative differences between the two, I want to choose the best, and it seems like the Bayesian method won out here. But that's only with the tiny bit of info I have to go on.

[whomp]

----Bump----

[Dason]

I don't want you thinking I forgot about this thread. I just want to actually have a decent response. I'll let it be known that I don't really like the way you did the simulation if your goal was to see how the Bayesian approach compares to the frequentist approach.

[Dason]

For each set of trials, I picked a p value between 0 and 1. Then, for each trial, I "flipped a coin" 14 times and saw if it came up heads 10 times. If it did, I then tested if the next two flips were heads. The total I got at the end (roughly .486) is the number of times it gets two heads in a row after getting 10 out of 14 divided by the total number of times it gets 10 out of 14. I ran 20,000 sets of trials, and each set of trials consisted of 20,000 trials. The code for the program (it is written in C) is here: http://pastebin.com/mc2F5tgz

The problem I see with this is that your simulation method is sampling from the posterior distribution of "p". This directly gives an advantage to the Bayesian method in your comparison. The goal you've set out to accomplish is tailored to make the Bayesian approach look better. If your goal is to compare the two methods of estimation then this isn't a very good way to do it. I mean clearly since Bayesian methods take advantage

I had a much longer response written but it got eaten after I accidentally navigated away from the page.

I think a better way to compare the frequentist method of estimating p and the Bayesian method would be to choose a suitable loss function and then compare the risk functions. For example if we use squared error loss then it's easy to show that for a true "p" near either 0 or 1 that the frequentist approach does a better job - but for p near 0.5 the Bayesian approach (with a uniform prior) does better. So you really can't say that one method is better than the other because it depends on what the true parameter value is. The numbers that you used in your sampling (10 out of 14) ends up being in that area where the Bayesian approach does better.

[Autobot]

The biggest difference between the two is for the Bayesian approach you need to know some information before hand. How are you choosing your prior(s)? In Geophysics and other fields they will choose priors that are most convenient, i.e. conjugate priors but are not necessarily the best choices. The best part of Bayesian approach is you always get an answer, the worst part is you always get an answer...

[Dason]

How or when do you not get an answer when doing a frequentist analysis?

[Autobot]

I was thinking of an inversion theory standpoint
when K(f)+e = y where K:X->Y and e~N(0,sigma)
there are certain restrictions on the operator K when doing the frequentist approach, can't think of them off the top of my head, I'll try to remember to post later. Whereas bayesian approach you treat f as random and apply a distribution and from there you find the posterior.