+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 24

Thread: Comparing Fs

  1. #1
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post

    Comparing Fs




    Hi ya'll

    How do you compare Fs (or Ts)?

    I know how to compare r-values and r2 values - but what if I have two Fs? How do I know if one is sig higher than the other?

    Here's what I have
    (the names have been changed to protect the innocent)

    Test 1
    Apples: M = 3.472, SD = .075;
    Oranges: M = 3.415, SD = .0820;
    F [1, 842] = 113.408, p < .001, partial eta squared = .119.

    Test 2
    Apples: M = 3.482, SD = .129
    Oranges: M = 3.414, SD = .137;
    F [1, 842] = 54.891, p < .001, partial eta squared = .061

    So is test 1's F significantly higher than test 2's?


  2. #2
    Points: 4,760, Level: 44
    Level completed: 5%, Points required for next Level: 190

    Location
    Ithaca, NY
    Posts
    266
    Thanks
    0
    Thanked 3 Times in 3 Posts
    I'm not sure it makes sense to test if one F is higher than the other.

    What would the null and alternate hypotheses be? What are you ultimately getting at?

    It would make more sense to me as "Does one model fit better than the other" or even combining these two comparisons into a 2x2 anova and testing if one difference in means is greater than the other difference in means. That's just an interaction.

    Karen
    The Analysis Factor
    http://TheAnalysisFactor.com

  3. #3
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    Re: "Does one model fit better than the other"

    I thought ... that's what I was asking ;-)

    I know one explains more - but is it significantly more?

  4. #4
    Points: 4,760, Level: 44
    Level completed: 5%, Points required for next Level: 190

    Location
    Ithaca, NY
    Posts
    266
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Sorry, now I get it. So much easier to consult in conversation.

    You use an F test to compare the two models. I assume they are ANOVAs, right?

    Whups. Was just going to say I answered this on my blog recently, but that was for nested models. I assume these models are not nested? Nested means one model contains all of the parameters of the other model, but yours sound like same model on different data. Correct?

    fyi- if anyone needs that test, it's in the comments at http://www.analysisfactor.com/statch...100#comment-24.

    Hmmm, comparing two non-nested models. I think you stumped me. I have this article, but I'm not sure it answers your question. http://www.analysisfactor.com/statch...100#comment-24

    Okay, Give me more context. What is the reason for this comparison? Would an interaction work?

    Karen

    And a note from my daughter (She's 4):
    cabqfypijhnvaaassssssssssssss
    :wav means e:
    The Analysis Factor
    http://TheAnalysisFactor.com

  5. #5
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    Ummm ... OK .... err ...

    First, I didn't get the message from your daughter ...?
    This is kinda important to me as I have a first on the way ...

    what was the question again?

    Oh right ...

    nah ... still not getting it ...

    Let's try it this way.

    Let's talk correlations.

    I have a GOLD standard variable (X)
    I have two other variables (A and B)
    Which, out of A and B correlates highest with X?
    A & X = .76 and B & X = .72
    Now - is .76 REALLY higher than .72?
    Well, I know the N so I can calcualte the z-diff and say, A&X and B&X are NOT significantly different.

    Now - same case - but this time it's a t-test, or F-test, or ANOVA.
    Gold standard is a fixed factor of Xs, Ys, and Zs.
    I have two DVs (A and B)

    FF&A = F1
    FF&B = F2

    which F is higher?

    Is there a z-diff (if you will) for this?

    Make sense?

    any sense?

    little bit?

  6. #6
    Points: 2,798, Level: 32
    Level completed: 32%, Points required for next Level: 102

    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    you said you are testing correlation, right?
    "Which, out of A and B correlates highest with X?"
    what if you alter the DV and predictor's roles?
    i mean, you have A and B as DV in two different cases and X as the same predictor, right?
    what if you take X as DV and A as predictor in first case, then in another model, B as the predictor with same X as DV.
    and then apply Chow test.: http://en.wikipedia.org/wiki/Chow_test

    if you need precise information, refer to these books:
    1. Gujarati, D. N. "Basic Econometrics" McGraw-Hill, Inc.
    2. Gujarati, D. N. "Essentials of econometrics"McGraw Hill.
    3. Maddala G. S. "Introduction to econometrics"Macmillan Publishing Company.

    since you are opting for correlations, it is definitely not a matter with DV and predictor.

  7. #7
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    Bourne ... no, sorry

    I am NOT doing correlations - I am comparing Fs in an ANOVA.

    I was showing that it is not difficult to compare r-values but I don't know how to compare F-values.

  8. #8
    Points: 4,760, Level: 44
    Level completed: 5%, Points required for next Level: 190

    Location
    Ithaca, NY
    Posts
    266
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Philyuko View Post
    Ummm ... OK .... err ...

    First, I didn't get the message from your daughter ...?
    This is kinda important to me as I have a first on the way ...
    Well, first things first. Congratulations! And don't worry, they don't try to type on your computer while you're using it for at least a few years. Well, maybe they'll just bang on it a bit....

    But back to your question....

    Quote Originally Posted by Philyuko View Post
    what was the question again?

    Oh right ...

    nah ... still not getting it ...

    Let's try it this way.

    Let's talk correlations.

    I have a GOLD standard variable (X)
    I have two other variables (A and B)
    Which, out of A and B correlates highest with X?
    A & X = .76 and B & X = .72
    Now - is .76 REALLY higher than .72?
    Well, I know the N so I can calcualte the z-diff and say, A&X and B&X are NOT significantly different.

    Now - same case - but this time it's a t-test, or F-test, or ANOVA.
    Gold standard is a fixed factor of Xs, Ys, and Zs.
    I have two DVs (A and B)

    FF&A = F1
    FF&B = F2

    which F is higher?

    Is there a z-diff (if you will) for this?

    Make sense?

    any sense?

    little bit?
    I get it.

    First, Bourne has a point. I don't know the Chow test well, but I believe it is used for testing two regression lines. ANOVA really is just a specific type of regression, so it should work. I can't remember the specifics though. I believe it is common in economics.

    But that is probably not what I would do, if I understand the point of your comparison correctly. I would combine them into a two-way anova and do an interaction.

    But wait. Maybe I don't get it. I just reread your description. Are you saying that all the IVs are identical but the two tests have two different DVs?

    So test one compared apples and oranges on weight and test 2 compared the same apples and oranges on juice content?

    Or is test 1 comparing Florida apples to oranges and test 2 is comparing California apples to oranges?

    An interaction (and a Chow test, I believe) are appropriate in the latter scenario, but not the first.

    Karen
    The Analysis Factor
    http://TheAnalysisFactor.com

  9. #9
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    Karen

    Re: "But wait. Maybe I don't get it. I just reread your description. Are you saying that all the IVs are identical but the two tests have two different DVs?"

    YES ;-)
    ahhh ... welcome to my world .... hence, no interaction possible.

    Not sure ... about the chow test because actually I have three groups (Apples/oranges/pears) and so either SS doesn't make much sense to me OR, the other way of doing a chow means I'd have 1,2,3 as a co-variate (where 1,2,3 are apples, oranges, and pears) and I can't do that ...

    I'll have to look more into CHOW but I'm pretty sure I can't work it here.

  10. #10
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    wait ... would it make sense to just use one of the DVs as a covariate?

  11. #11
    Points: 4,760, Level: 44
    Level completed: 5%, Points required for next Level: 190

    Location
    Ithaca, NY
    Posts
    266
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Okay, this is where I say you need to start over and explain exactly what you are trying to get at and exactly what your data are.

    It might work to use one DV as a predictor. If your two DVs are similar, measured on the same scale, you could just have a binary predictor that says DV1/DV2. Depends what they are.

    If you don't want to post publicly what your variables are, you can email me.

    Karen
    The Analysis Factor
    http://TheAnalysisFactor.com

  12. #12
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    no - happy to explain ;-)

    Students are asked to self explain a sentence. They SEE a sentence. They SELF EXPLAIN it. They use one of three strategies to self explain (SE) it. These strategies are A, B, and C. They are my grouping variables.

    I have two computational models (X and Y); they provide a value between 1 and 6. This value is based on the sentence and the self-explanation (it is a guesstimate value).

    So, the ANOVA has a,b,c as fixed factors and X and Y as DVs.
    Both X and Y work! Neatly dividing the three categories (or explaining them anyway).
    So, now I have two F-values.
    One F-value is higher.
    Is it significantly higher though?
    I mean, is 87 REALLY more than 73?

    The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results.

    that's it really ... ;-)

  13. #13
    Points: 4,760, Level: 44
    Level completed: 5%, Points required for next Level: 190

    Location
    Ithaca, NY
    Posts
    266
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Well, now I see why you were trying to simplify it with an example. I am still not getting this stuff entirely, but perhaps it doesn't matter.

    Anyway, if X and Y are on the same scale, i.e. if the 1 and the 6 and everything in between are equivalent, then put it all in one model. Create a new IV called "Model" which indicates if it's X or Y. Just call the DV "response" or something.

    I didn't get whether A, B, and C are the 3 levels of a single variable or 3 different variables, but let's assume they're different, since you said they are fixed factors. Interact "Model" with A, B, and C and any interactions between them. (If indeed they were a single variable, this would be very simple. It will get a little messier if you do have 3 variables with interactions. If that is the case, it might be neater to just do the Chow test.)

    Or maybe I'm still not understanding the design well enough.

    But, I wanted to comment on a few things.

    First, 87 and 73 are both enormous F values. Enormous.

    You said: "The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results."

    Remember than statistically significant really just means you should reliably find the same size effect in other samples. It's not telling you if your results are "significantly better" in a scientifically significant way. Just that you're likely to get the same (possibly tiny) effect in any sample you took. If you're making expensive changes, then perhaps you need both statistical and scientific significance--meaning big effects.

    Karen
    The Analysis Factor
    http://TheAnalysisFactor.com

  14. #14
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post
    Karen

    I'll be back ... ;-)

    must be off to Japan first, though ... stay tuned ;-)

  15. #15
    Points: 3,504, Level: 37
    Level completed: 3%, Points required for next Level: 146

    Posts
    139
    Thanks
    1
    Thanked 1 Time in 1 Post

    ... waiting for plane ...

    > Anyway, if X and Y are on the same scale, i.e. if the 1 and the 6 and everything in between are equivalent, then put it all in one model.

    They're not quite QUITE on the same scale but I can Z them to cover that.

    > Create a new IV called "Model" which indicates if it's X or Y. Just call the DV "response" or something.

    I smell a CHOW coming ;-)

    > I didn't get whether A, B, and C are the 3 levels of a single variable or 3 different variables, but let's assume they're different, since you said they are fixed factors.

    They're different levels. Males, females, and ummm, apples.

    > Interact "Model" with A, B, and C and any interactions between them. (If indeed they were a single variable, this would be very simple. It will get a little messier if you do have 3 variables with interactions. If that is the case, it might be neater to just do the Chow test.)

    Yes - but the three levels means that the co-variate would have 1,2,3 and that 1,2,3 would be categories - so, that doesn't work.

    BUT WHAT I AM THINKING IS THAT MAYBE IT COULD BE A HLM.

    So, the 1,2,3 is the random variable nested within the model - and the DV is the combined output variable. That would be a neat test of different F values. I think I like it!

    > But, I wanted to comment on a few things.
    First, 87 and 73 are both enormous F values. Enormous.

    Fs are relative - I have a large corpus - frequently my Fs are over 100.

    > You said: "The question is important because if you're going to change a computational system then that's expensive - and you'd want it to be producing significantly better results."
    Remember then statistically significant really just means you should reliably find the same size effect in other samples. It's not telling you if your results are "significantly better" in a scientifically significant way. Just that you're likely to get the same (possibly tiny) effect in any sample you took. If you're making expensive changes, then perhaps you need both statistical and scientific significance--meaning big effects.

    Yes, I know ... I know ... but I feel you're a "REAL" stats person whereas I just abuse statistics to make my various fields feel comfortable in their claim that they are scientific.

    (ooh honesty ... ouch ... ouch ... ouch ...)

+ Reply to Thread
Page 1 of 2 1 2 LastLast

           




Similar Threads

  1. Comparing 2 Anovas
    By ndogg in forum Psychology Statistics
    Replies: 0
    Last Post: 01-20-2011, 09:02 AM
  2. Comparing two distributions
    By SuzanneMonette in forum Statistics
    Replies: 4
    Last Post: 10-29-2010, 07:42 AM
  3. Comparing two conditions, n=1
    By MethoxyRoxy in forum Statistics
    Replies: 3
    Last Post: 11-04-2008, 10:25 AM
  4. Comparing mean of 3 samples
    By Andrew! in forum Statistics
    Replies: 5
    Last Post: 09-16-2008, 03:13 AM
  5. Comparing data
    By jdm in forum Psychology Statistics
    Replies: 1
    Last Post: 08-27-2008, 01:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats