Can I reject the null hypothesis?

[angusthewestie]

Hi,

I'm just finishing off the analysis of an experiment on empathising and systemising (Baron-Cohen). My problem is that in the final hypothesis, I do not know if I should accept it or not, and if I do, what can I say about it? The hypothesis was:

"There will be a significant positive relationship between self-report and performance on measures of systemising".

In fact, although the relationship turns out to be statistically significant, it is very weak, and much weaker than expected:


Pearson correlation .268

R² .072
Sig. p less than .01
N 136

So, although I have some sort of relationship, 7% really isn't much, so I was wondering how I should proceed? (I ran another correlation that had a moderate effect of 36%, so I know how to deal with that one!).

Any help very gratefully received.

Thank you!

[AtlasFrysmith]

Your p-value of .01 implies that the variable is significant at 99%. That's "good." The low R^2 doesn't have any bearing on the validity of that result per se, it just means that there's a lot of the variance that isn't explained by your included variables.

Now, this could be because there's just a lot of random error. But it could also be because there are other variables that matter, but that you haven't included. To the extent these variables may be correlated with your variable of interest, your results could be subject to omitted variable bias.

Your burden is the following: Either you have to defend theoretically the proposition that there are no other explanatory variables and it's all random error, or you have to show that any such omitted variables aren't correlated with the variable of interest. A third option would be to include them in the regression and redo the estimation.

Showing or proving a lack of correlation may not be as challenging as you might think--if for example your variable of interest is a "treatment" and you have a randomly assigned control group, any missing variables would be uncorrelated with the choice of group and hence with the treatment.

Whether your results are "significant' in the non-statistical sense is up to you to decide--but a low R^2, again, does not inherently challenge the conclusion that your variable of interest affects the dependent variable.

[angusthewestie]

Thanks for the helpful reply!

I realise now that my figures are statistically significant.

The low R^2 is likely to be due to one of the tests I'm comparing not being a very accurate test ... in other words there are other factors affecting this outcome, and if the test had been better, I would have had a stronger correlation.

So ... I suppose the final question has to be whether I accept the hypothesis, with reservations mentioned in the discussion, or reject it for being too weak to be or any relevance. I seem to bestuck between the proverbial rock and a hard place on this one! I don't think I'm expected to run a regression, as this hasn't been covered in my course, although if you could tell me which one to do, I'll get out the stats books and have a try!

Thanks again