Chi test and regression

[Oli]

Hello everybody

I have to test whether there is a significant difference between two groups. I want to indicate that among the two groups are no differences. First I made a chi-test and the resulting p value was about 0.4 which would be "good". But the results of a regression did not confirm that. I got a p value of 0.035 for the variable(inwhich group the people are)
The r2 is just about 0.05 for this variable and about 0.16 for the whole model.
Im really confused... Does anybody have an idea what Im doing wrong? Why do I get different results?

Appreciate any help!
Oliver

[Englund]

Chi square tests are usually quite weak in power, which could be why you dont detect a significant difference with the chi square test.

[Karabiner]

I have to test whether there is a significant difference between two groups.

Could you tell a bit more? Background, objective of the study? At least sample sizes, and description of the dependent variable?

Kind regards

K.

[Oli]

In my study I would like to show that a different type of incentive(cash vs lottery) does not have an influence on the amount of correct answers. One group has 105 participants the other group has 97.
the chi test that I have done looked like this: chisq.test(correctanswers, incentivetype) in r. Incentive type was 0 for Lottery and 1 for cash.
the regression Ive made looked like this:lm(formula=correctanswers~incentivetype+age+gender+...)
Did I make a mistake?

[Englund]

I'm not that in to R. But it looks like you haven't really tested the same thing with the chi square test and the regression. When doing the regression you're testing whether incentivetype has an effect on number of correct answers while controlling for the other variables, such as age, gender and so on. However, the regression would be, according to me, a better test in this case since the chi square test doesn't take the other IV´s into account.

[Oli]

And how do I interpret the low r squared? Its only about 5% for the incentivetype variable and just about 17% for the whole regression. I thought this means that only 5% can be explained by the variable incentivetype which would not be that much.

[Englund]

Yes, that is a good question. Something can be statistically significant, whilst not practically significant. For example, lets say that we find that 45 percent of women like Coca Cola and 44 percent of men do, and lets say that we had a really big sample which caused the results to be significant. Despite the statistically significant results, we'd draw the conclusion that the results are not of practical importance.

It's a difficult case you've got there. I guess you'll have to argue logically which test/tests you should use. Have you done a t-test to compare the means between the groups?

[Oli]

Thanks for your inputs!

I thought the t-test is only applicable if there is a normal distribution of the sample given which isnt the case with my data. Or can I do an other test like the t-test?
Do I get this right:
The Chi- test results should not be false but they might be not very meaningful.

[hlsmith]

If you have not done the following, it may be of interest. Perform your chi-square then run a simple logistic regression with only the same dependent and independent variable. Then run different combinations of dependent plus two independent variables in the chi-square test using Breslow-Day test, then repeat same procedures with logistic regression. You may also get a better understanding of your data if you calculate either the RR or OR when performing these tasks. In the end you may understand how the controlling or stratifying of the data changes the significance and the different associations, perhaps also placing these values in a table to better compare.

[Oli]

Thanks I will get some information about how to do such a test.
I have done a ks test as well to see whether the two samples(cash vs. lottery) have the same distribution. Ive got a large p-value which indicates that there are no differences. Is this something that I could use or isnt it appropriate for my study?

[hlsmith]

I am not completely sure the objectives and overall purpose of your project, so I cannot answer your question. Though, I think this previous conversation (see below) could be beneficial.

http://www.talkstats.com/showthread....v-smirnov-test

[Oli]

I have made several regressions with different variables. For instance:
correctanswers~incentivetype+gender
correctanswers~incentivetype+age+gender
correctsanswers~incentivetype+gender+I(incentivetype*gender)
etc.
But I dont really know how I should do it. The r2 never exceeded 0.2. And I find it a bit strange that in some cases the p-value for incentivetype is lower than 0.05 and in some cases higher. What do I have to consider while doing a regression? When is it appropriate to use something like that: I(incentivetype*gender)

[hlsmith]

If "I(incentivetype*gender)" is your interaction term, you should always examine for effect modification between IV. It is up to you whether you keep it in the model, this decision if typically based on its significance (p-value). In general if you are using a effect or p-value cut-off for your IV, you may also use this for the interaction term if it seems feasible. But much is situation based.

[Englund]

I thought the t-test is only applicable if there is a normal distribution of the sample given which isnt the case with my data.

Sorry for a very late response. Yes, that is true. But you can transform your data by taking the natural logaritm or the square root of it (if it is only positive values). Often it is possible to transform the data so that it gets approximately normally distributed

[Dason]

Sorry for a very late response. Yes, that is true. But you can transform your data by taking the natural logaritm or the square root of it (if it is only positive values). Often it is possible to transform the data so that it gets approximately normally distributed

For small samples we require the data in each group to be normally distributed. For large sample sizes it really doesn't matter. What we really want is the sampling distribution of the sample mean to be approximately normally distributed. With a large enough sample size then this happens due to the central limit theorem.

I'm not a fan of transforming the data just so you get a normal distribution.