Comparing Fs

#1
Hi ya'll

How do you compare Fs (or Ts)?

I know how to compare r-values and r2 values - but what if I have two Fs? How do I know if one is sig higher than the other?

Here's what I have
(the names have been changed to protect the innocent)

Test 1
Apples: M = 3.472, SD = .075;
Oranges: M = 3.415, SD = .0820;
F [1, 842] = 113.408, p < .001, partial eta squared = .119.

Test 2
Apples: M = 3.482, SD = .129
Oranges: M = 3.414, SD = .137;
F [1, 842] = 54.891, p < .001, partial eta squared = .061

So is test 1's F significantly higher than test 2's?

:wave:
 
#2
I'm not sure it makes sense to test if one F is higher than the other.

What would the null and alternate hypotheses be? What are you ultimately getting at?

It would make more sense to me as "Does one model fit better than the other" or even combining these two comparisons into a 2x2 anova and testing if one difference in means is greater than the other difference in means. That's just an interaction.

Karen
 
#3
Re: "Does one model fit better than the other"

I thought ... that's what I was asking ;-)

I know one explains more - but is it significantly more?
 
#4
Sorry, now I get it. So much easier to consult in conversation.

You use an F test to compare the two models. I assume they are ANOVAs, right?

Whups. Was just going to say I answered this on my blog recently, but that was for nested models. I assume these models are not nested? Nested means one model contains all of the parameters of the other model, but yours sound like same model on different data. Correct?

fyi- if anyone needs that test, it's in the comments at http://www.analysisfactor.com/statchat/?p=100#comment-24.

Hmmm, comparing two non-nested models. I think you stumped me. I have this article, but I'm not sure it answers your question. http://www.analysisfactor.com/statchat/?p=100#comment-24

Okay, Give me more context. What is the reason for this comparison? Would an interaction work?

Karen

And a note from my daughter (She's 4):
cabqfypijhnvaaassssssssssssss
:wav means e:
 
#5
Ummm ... OK .... err ...

First, I didn't get the message from your daughter ...?
This is kinda important to me as I have a first on the way ...

what was the question again?

Oh right ...

nah ... still not getting it ...

Let's try it this way.

Let's talk correlations.

I have a GOLD standard variable (X)
I have two other variables (A and B)
Which, out of A and B correlates highest with X?
A & X = .76 and B & X = .72
Now - is .76 REALLY higher than .72?
Well, I know the N so I can calcualte the z-diff and say, A&X and B&X are NOT significantly different.

Now - same case - but this time it's a t-test, or F-test, or ANOVA.
Gold standard is a fixed factor of Xs, Ys, and Zs.
I have two DVs (A and B)

FF&A = F1
FF&B = F2

which F is higher?

Is there a z-diff (if you will) for this?

Make sense?

any sense?

little bit?
 
#6
you said you are testing correlation, right?
"Which, out of A and B correlates highest with X?"
what if you alter the DV and predictor's roles?
i mean, you have A and B as DV in two different cases and X as the same predictor, right?
what if you take X as DV and A as predictor in first case, then in another model, B as the predictor with same X as DV.
and then apply Chow test.: http://en.wikipedia.org/wiki/Chow_test

if you need precise information, refer to these books:
1. Gujarati, D. N. "Basic Econometrics" McGraw-Hill, Inc.
2. Gujarati, D. N. "Essentials of econometrics"McGraw Hill.
3. Maddala G. S. "Introduction to econometrics"Macmillan Publishing Company.

since you are opting for correlations, it is definitely not a matter with DV and predictor.
 
#7
Bourne ... no, sorry

I am NOT doing correlations - I am comparing Fs in an ANOVA.

I was showing that it is not difficult to compare r-values but I don't know how to compare F-values.
 
#8
Ummm ... OK .... err ...

First, I didn't get the message from your daughter ...?
This is kinda important to me as I have a first on the way ...
Well, first things first. Congratulations! And don't worry, they don't try to type on your computer while you're using it for at least a few years. Well, maybe they'll just bang on it a bit....

But back to your question....

what was the question again?

Oh right ...

nah ... still not getting it ...

Let's try it this way.

Let's talk correlations.

I have a GOLD standard variable (X)
I have two other variables (A and B)
Which, out of A and B correlates highest with X?
A & X = .76 and B & X = .72
Now - is .76 REALLY higher than .72?
Well, I know the N so I can calcualte the z-diff and say, A&X and B&X are NOT significantly different.

Now - same case - but this time it's a t-test, or F-test, or ANOVA.
Gold standard is a fixed factor of Xs, Ys, and Zs.
I have two DVs (A and B)

FF&A = F1
FF&B = F2

which F is higher?

Is there a z-diff (if you will) for this?

Make sense?

any sense?

little bit?
I get it.

First, Bourne has a point. I don't know the Chow test well, but I believe it is used for testing two regression lines. ANOVA really is just a specific type of regression, so it should work. I can't remember the specifics though. I believe it is common in economics.

But that is probably not what I would do, if I understand the point of your comparison correctly. I would combine them into a two-way anova and do an interaction.

But wait. Maybe I don't get it. I just reread your description. Are you saying that all the IVs are identical but the two tests have two different DVs?

So test one compared apples and oranges on weight and test 2 compared the same apples and oranges on juice content?

Or is test 1 comparing Florida apples to oranges and test 2 is comparing California apples to oranges?

An interaction (and a Chow test, I believe) are appropriate in the latter scenario, but not the first.

Karen
 
#9
Karen

Re: "But wait. Maybe I don't get it. I just reread your description. Are you saying that all the IVs are identical but the two tests have two different DVs?"

YES ;-)
ahhh ... welcome to my world .... hence, no interaction possible.

Not sure ... about the chow test because actually I have three groups (Apples/oranges/pears) and so either SS doesn't make much sense to me OR, the other way of doing a chow means I'd have 1,2,3 as a co-variate (where 1,2,3 are apples, oranges, and pears) and I can't do that ...

I'll have to look more into CHOW but I'm pretty sure I can't work it here.
 
#11
Okay, this is where I say you need to start over and explain exactly what you are trying to get at and exactly what your data are.

It might work to use one DV as a predictor. If your two DVs are similar, measured on the same scale, you could just have a binary predictor that says DV1/DV2. Depends what they are.

If you don't want to post publicly what your variables are, you can email me.

Karen
 
#12
no - happy to explain ;-)

Students are asked to self explain a sentence. They SEE a sentence. They SELF EXPLAIN it. They use one of three strategies to self explain (SE) it. These strategies are A, B, and C. They are my grouping variables.

I have two computational models (X and Y); they provide a value between 1 and 6. This value is based on the sentence and the self-explanation (it is a guesstimate value).

So, the ANOVA has a,b,c as fixed factors and X and Y as DVs.
Both X and Y work! Neatly dividing the three categories (or explaining them anyway).
So, now I have two F-values.
One F-value is higher.
Is it significantly higher though?
I mean, is 87 REALLY more than 73?

The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results.

that's it really ... ;-)
 
#13
Well, now I see why you were trying to simplify it with an example. :) I am still not getting this stuff entirely, but perhaps it doesn't matter.

Anyway, if X and Y are on the same scale, i.e. if the 1 and the 6 and everything in between are equivalent, then put it all in one model. Create a new IV called "Model" which indicates if it's X or Y. Just call the DV "response" or something.

I didn't get whether A, B, and C are the 3 levels of a single variable or 3 different variables, but let's assume they're different, since you said they are fixed factors. Interact "Model" with A, B, and C and any interactions between them. (If indeed they were a single variable, this would be very simple. It will get a little messier if you do have 3 variables with interactions. If that is the case, it might be neater to just do the Chow test.)

Or maybe I'm still not understanding the design well enough.

But, I wanted to comment on a few things.

First, 87 and 73 are both enormous F values. Enormous.

You said: "The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results."

Remember than statistically significant really just means you should reliably find the same size effect in other samples. It's not telling you if your results are "significantly better" in a scientifically significant way. Just that you're likely to get the same (possibly tiny) effect in any sample you took. If you're making expensive changes, then perhaps you need both statistical and scientific significance--meaning big effects.

Karen
 
#15
... waiting for plane ...

> Anyway, if X and Y are on the same scale, i.e. if the 1 and the 6 and everything in between are equivalent, then put it all in one model.

They're not quite QUITE on the same scale but I can Z them to cover that.

> Create a new IV called "Model" which indicates if it's X or Y. Just call the DV "response" or something.

I smell a CHOW coming ;-)

> I didn't get whether A, B, and C are the 3 levels of a single variable or 3 different variables, but let's assume they're different, since you said they are fixed factors.

They're different levels. Males, females, and ummm, apples.

> Interact "Model" with A, B, and C and any interactions between them. (If indeed they were a single variable, this would be very simple. It will get a little messier if you do have 3 variables with interactions. If that is the case, it might be neater to just do the Chow test.)

Yes - but the three levels means that the co-variate would have 1,2,3 and that 1,2,3 would be categories - so, that doesn't work.

BUT WHAT I AM THINKING IS THAT MAYBE IT COULD BE A HLM.

So, the 1,2,3 is the random variable nested within the model - and the DV is the combined output variable. That would be a neat test of different F values. I think I like it!

> But, I wanted to comment on a few things.
First, 87 and 73 are both enormous F values. Enormous.

Fs are relative - I have a large corpus - frequently my Fs are over 100.

> You said: "The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results."
Remember then statistically significant really just means you should reliably find the same size effect in other samples. It's not telling you if your results are "significantly better" in a scientifically significant way. Just that you're likely to get the same (possibly tiny) effect in any sample you took. If you're making expensive changes, then perhaps you need both statistical and scientific significance--meaning big effects.

Yes, I know ... I know ... but I feel you're a "REAL" stats person whereas I just abuse statistics to make my various fields feel comfortable in their claim that they are scientific.

(ooh honesty ... ouch ... ouch ... ouch ...)
 
#16
huff!! i am lost!! i do not know whether this will help you or increase your confusion:
you have a variable with three categories 1,2,3 ok?
well, i know in spss, we can split file (data>>split file). after that, every analysis we perform, will result in three outputs, separate for 1,2,3.
so, this will result in 3 Fs i guess. and you can perform this for the second DV.
so you will have 2 sets of 3 Fs.
and then perform Chow!!

please do not be angry on me!! if this looks a ****, just flush it out!!
 
#18
you have A,B and C as three levels of a single variable, right?: "They're different levels. Males, females, and ummm, apples."

so, while performing HLM, you have just one variable as a fixed factor. but in that variable you have three categories. so my opinion is to split the data based on this variable.
 
#19
No HLMs. Back...away...slowly.

Okay, this is why I keep harping on what is it you're trying to test? The analysis has to match the research question and the study design.

If 1,2,3 (male, female, apples)--I'll call it Gender--is what you're trying to compare X and Y on, then you need it to be a fixed factor. There is no reason to not have a 3 level fixed factor in either an ANOVA or a regression. In a regression, you would just dummy code it.

Then you can have "Model" as a two-level fixed factor. And an interaction. That is what GLMs (anovas and regressions) were born to do. No problem.

And, Bourne, I understand what you're suggesting about splitting the file, but not why. If I understand the point of this model (and I'm not making any strong claims here about understanding it), he needs to see if Gender affects X and Y the same. He won't be able to see how Gender affects anything if he looks at each level separately.

But he could just do two regression models regressing X on Gender and Y on Gender, then do a Chow test. But my impression is that a Chow test is best suited for when there are many predictors (independent variables). Since he has only one, he might as well just do an interaction. If there were many predictors, doing interactions becomes unwieldy, thus the need for the Chow.

Have fun in Japan!

Karen
 
#20
and good morning from Japan!
... and a good morning from my luggage still in the States ... anyhoo ...

> No HLMs.
got it ... easy to ... not do.

> Okay, this is why I keep harping on what is it you're trying to test? The analysis has to match the research question and the study design.

Research question is ... does X explain/predict "gender" better than Y?
Now, I know my DVs and IVs are a bit back to front in that regard but it's an applied world.

> If 1,2,3 (male, female, apples)--I'll call it Gender--is what you're trying to compare X and Y on, then you need it to be a fixed factor.

yeap - it IS a fixed factor.

> There is no reason to not have a 3 level fixed factor in either an ANOVA or a regression. In a regression, you would just dummy code it.

right with you ...

> Then you can have "Model" as a two-level fixed factor.

Going to try that ... not ... quite ... sure ... what it'll tell me though? Or how ... but, I will run it and see.

Thanks!
Phil