What is the difference between classical and bayesian understanding of probability?

noetsi

Fortran must die
#1
I am reading a book on monte carlo analysis and the author stresses in passing that the definition of probability used in an example in the book is the classical aka frequentist one, and that Bayesian understanding of probability is different. They say the difference is too complex to explain in the book.

The comment goes "Thus, we can define probability in terms of the relative frequency with which an event will occur if we repeated our "experiment" [a coin flip] a great many times. This is a classic "frequentist" interpretation of probability that lies at the heart of traditional or "classical" statistics." [they then to go on to point out that Bayesian's have a different interpretation of probability, but no details are provided].

I was hoping for a brief explanation of how the two differ.
 

CowboyBear

Super Moderator
#2
Re: What is the difference between classical and bayesian understanding of probabilit

Welp there is a metric ****-ton of articles about the difference, so it'd be worth reading up, but a very simple definition would be that a frequentist interprets probability just how it's stated in your quote:

the relative frequency with which an event will occur if we repeated our "experiment" [a coin flip] a great many times.
This means that you can only usefully talk about the probability of events happening if you can at least imagine in principle conducting repeated "trials" (experiments, coin flips, whatever) in which sometimes the event takes place and sometimes it doesn't. To a frequentist, it's meaningless to ask what the probability it is that some hypothesis is true; the hypothesis is either true or it isn't (it isn't true in some trials and false in others). This is part of the reason why the probabilities we talk about in frequentism (e.g., p values) refer to how probable it is that a particular thing will be observed in an experiment, not how probable hypotheses themselves are. We can imagine conducting repeated experiments in which sometimes a thing happens and sometimes it doesn't. (But the hypothesis will simply remain either true or false).

In the Bayesian interpretation of probability, a probability is a state of certainty or belief with respect to some proposition. That probability could refer to the rate at which an event (e.g., a coin flip coming up heads) occurs over repeated trials, but it doesn't have to: A Bayesian can also refer to the probability that a particular hypothesis is true.

And Bayes theorem shows us how to combine existing knowledge/beliefs/information along with the data observed to make statements about probabilities.

(Note: That's a super simplistic version!)
 

noetsi

Fortran must die
#3
Re: What is the difference between classical and bayesian understanding of probabilit

As a non-statistician I guess the frequentist position makes more sense to me. The sky is, to be simplistic, either blue or its not. That is hypothesis are either true or they are not since to me the hypothesis is about what is true in the population not the sample (at a given time anyhow obviously reality changes in some cases).
 

CowboyBear

Super Moderator
#4
Re: What is the difference between classical and bayesian understanding of probabilit

As a non-statistician I guess the frequentist position makes more sense to me. The sky is, to be simplistic, either blue or its not. That is hypothesis are either true or they are not since to me the hypothesis is about what is true in the population not the sample (at a given time anyhow obviously reality changes in some cases).
Just to be clear: The Bayesian position isn't that a particular hypothesis is sometimes true and sometimes false. The Bayesian position is that we can make probability statements about things that are either true or false. So a Bayesian can say something like "there is a 95% probability that my hypothesis is true", whereas a frequentist can't.

Anyway the best thing to do if you're interested in the difference between the two perspectives is to read more. Some sources to start with:

Kruschke, J. K. (2010). What to believe: Bayesian methods for data analysis. Trends in Cognitive Sciences, 14(7), 293–300. https://doi.org/10.1016/j.tics.2010.05.001
Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.
Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290. https://doi.org/10.1177/1745691611406920
 
#5
Re: What is the difference between classical and bayesian understanding of probabilit

I have a follow up question. Because conditional probabilities and Bayes' Theorem piqued my interest as an undergraduate.

Are these two definitions of probability always used together? In other words, is it possible for someone to think entirely in terms of Bayesian statistics and "ignore" the frequentist perspective? I don't imagine so. But I'm curious.

I really enjoy what I've seen of Bayesian statistics.
 

Dason

Ambassador to the humans
#6
Re: What is the difference between classical and bayesian understanding of probabilit

One of the reasons a lot of people like the bayesian perspective is entirely because they prefer the bayesian interpretation of probability.

Want to talk about the probability your favorite team will win the super bowl? Well try thinking about what a frequentist defines probability as and think if it makes sense to talk about the probability that the Packers (obviously this is everybody's favorite team) wins the super bowl next year. With a bayesian interpretation it's very natural to talk about probability of one time events like this. If you have this chat with a frequentist there is a non-zero probability (hah) that you'll hear something about multiple universes or hypothetical timelines and it starts to sound *really complicated* just to talk about how you can define probability for an event like that.
 

Dason

Ambassador to the humans
#7
Re: What is the difference between classical and bayesian understanding of probabilit

I have a follow up question. Because conditional probabilities and Bayes' Theorem piqued my interest as an undergraduate.

Are these two definitions of probability always used together? In other words, is it possible for someone to think entirely in terms of Bayesian statistics and "ignore" the frequentist perspective? I don't imagine so. But I'm curious.

I really enjoy what I've seen of Bayesian statistics.
There are a few issues here. First note that if you're just using bayes theorem to do a probability calculation that doesn't mean you're doing bayesian statistics.

Also it's entirely possible to be a frequentist and use bayesian methods. You might just be interested in different estimators that have nice properties and want to discuss the frequentist properties of such estimators. It's also possible to be a bayesian and and want to explore the frequentist properties of your estimators. They aren't entirely mutually exclusive schools of thought.
 
#8
Re: What is the difference between classical and bayesian understanding of probabilit

Okay, I have a limited knowledge under my belt. So, I naturally attributed Bayes' Theorem to that school of thought. As a side note, I'm excited to pursue a master's in the fall. Applications of statistics are endless!
 

noetsi

Fortran must die
#9
Re: What is the difference between classical and bayesian understanding of probabilit

While this may not be theoretically the case I commonly read analysis that mix bayseian and classical statistics. That is they use Bayesian approaches sometimes and frequentist statistics other times.
 

CowboyBear

Super Moderator
#10
Re: What is the difference between classical and bayesian understanding of probabilit

If you have this chat with a frequentist there is a non-zero probability (hah) that you'll hear something about multiple universes or hypothetical timelines and it starts to sound *really complicated* just to talk about how you can define probability for an event like that.
God**** frequentists with their multiverses and superpopulations...:p
 

spunky

Super Moderator
#11
Re: What is the difference between classical and bayesian understanding of probabilit

If you have this chat with a frequentist there is a non-zero probability (hah) that you'll hear something about multiple universes or hypothetical timelines and it starts to sound *really complicated* just to talk about how you can define probability for an event like that.
Are you half-kidding or is this for real? As in, does the frequentist interpretation of probability implicitly assumes the need for multiple universes/timelines? Is it something you heard or is it something that someone has tried to justify somewhere like in a journal article or book or something?
 

CowboyBear

Super Moderator
#12
Re: What is the difference between classical and bayesian understanding of probabilit

Are you half-kidding or is this for real? As in, does the frequentist interpretation of probability implicitly assumes the need for multiple universes/timelines? Is it something you heard or is it something that someone has tried to justify somewhere like in a journal article or book or something?
Dunno about Dason, but I'm pretty sure I've seen this kind of thing. I mean, after all the winner of next year's superbowl isn't a random variable: You can't say it's 65% likely to be the Packers in the frequentist definition, because there aren't going to be multiple 2018 superbowls in which sometimes the Packers win and sometimes they don't. So to make a frequentist probability statement you have to make come up with some imaginary way in which there are multiple "trials". (Not that the frequentist interpretation implicitly requires multiverses - it just kinda requires them if we want to stretch frequentist probability into making statements about things that it's not designed to make statements about).
 

rogojel

TS Contributor
#13
Re: What is the difference between classical and bayesian understanding of probabilit

The simplest way to explain the randomness in quantum physics is the multiple universe interpretation of Everett - so frequentists might in the end be right after all.

regards
 

gianmarco

TS Contributor
#14
Re: What is the difference between classical and bayesian understanding of probabilit

As a non-statistician I guess the frequentist position makes more sense to me.
I am the least qualified person who can talk about theoretical aspects of statistics in general, and of Bayesian approaches in particular. I wish to just stress that Bayesian "standpoint" has its relevance in those fields where you want to assess, for instance, how likely is your prior knowledge on the basis of what you observe, i.e. data.

For instance, in archaeological radiocarbon dating, Bayesian statistic provides scientists with a formal way to update one's belief in the light of the data (s)he has currently in his/her hand. For example, we got radiocarbon data from context A, B, and C, and we have a prior belief/hypothesis (deriving from other independent observations) as to the fact that A is older than B, and B is older than C. Bayesian analysis allows us to arrive at the posterior probability as to the soundness of our prior hypothesis. In other words, given our prior hypothesis AND given the data, we can "update" our prior belief in the light of what data are telling us (so getting a posterior probability).

I found a book whose first sections are good for making non-stat people (like me) grasp some basic principle of Bayesian inference:
https://www.amazon.com/Bayes-Rule-Tutorial-Introduction-Bayesian/dp/0956372848



gm
 

hlsmith

Omega Contributor
#15
Re: What is the difference between classical and bayesian understanding of probabilit

Right on Rogojel it is all about M-verses. String theory is so 90's for buzz terms. I am also interested in the idea of M-verses. Opens the doors to many awesome movies as well. And I keep wondering if our universe is ever expanding cant we stave off our own freezing demise with some type of human generated nuclear power. I digress, but was thinking the same thing about M-verses.


I don't get all gummed up by the idea that given the event in a super-population is true, that realization has a certain probability. I think the issue is when thinking about it as a Superbowl win, instead of average change in weight or a more approach example.
 

spunky

Super Moderator
#16
Re: What is the difference between classical and bayesian understanding of probabilit

I did some googling around and, for better or worse, CBear and Dason are right. Within the frequentist paradigm of the interpretation of probability you need to do some serious mental gymnastics to be able to do account for simple probabilistic statements like the chance that X or Y team (I refuse to use the P-word :p) will win A or B tournaments.

I used to be a big advocate of the Bayesian paradigm when I was still a master's student but I am somewhat ambivalent of it now. It's not because of Bayesian statistics itself but mostly the fact that, in my field (education/social sciences), we're just so bad at data analysis that I feel if we don't clean our act first it's better to remain within the frequentist paradigm a little longer.

Every time I'm talking about this issue with my profs colleagues that Herman Rubin claim comes up in my mind:

A good Bayesian does better than a non-Bayesian, but a bad Bayesian gets clobbered.

Which immediately makes me think Will these people be good Bayesians or bad Bayesians? Do I really want to find out?
 

rogojel

TS Contributor
#17
Re: What is the difference between classical and bayesian understanding of probabilit

Right on Rogojel it is all about M-verses. String theory is so 90's for buzz terms. I am also interested in the idea of M-verses. Opens the doors to many awesome movies as well. And I keep wondering if our universe is ever expanding cant we stave off our own freezing demise with some type of human generated nuclear power. I digress, but was thinking the same thing about M-verses.


I don't get all gummed up by the idea that given the event in a super-population is true, that realization has a certain probability. I think the issue is when thinking about it as a Superbowl win, instead of average change in weight or a more approach example.
Not to be prim about a technical issue but the Everett interpretation is way older then the M-verses. It just says that each time the wave function collapses - basically at every interaction of a particle with the environment - the universe splits in as many parallel universes as we have outcomes of the event. Our consciousness just stays in one of these universes - in all other there will be a similar consciousness that registered a different outcome. Oldie but goldie from the fifties.
 

hlsmith

Omega Contributor
#18
Re: What is the difference between classical and bayesian understanding of probabilit

I was referring to:


Scenarios where the big bang was created, along with our universe via the collision of two membranes.
 
#20
Re: What is the difference between classical and bayesian understanding of probabilit

I'm not sure if it's been mentioned yet, but part of the way that Bayesian's view probability comes from the difference in how they view population parameters (if I'm not mistaken). Their 95% credible interval is the prime example of this. The interpretation fits the natural question that many have; "What's the probability that I'm right?" Their interpretation says there exists a 95% chance that the true parameter value is within the 95% credible interval. This statement is allowed because Bayesians have a notion that the data are fixed but the parameter is a random variable (as opposed to Frequentists viewing the data as random and the parameters as fixed). This also helps illustrate why the Frequentist says the true parameter value either is or is not in the 95% confidence interval, there is no in between because the parameter is not a random variable. This thinking can be extended to other kinds of tests, too.