+ Reply to Thread
Results 1 to 9 of 9

Thread: Variable change from significant to not significant

  1. #1
    Points: 533, Level: 10
    Level completed: 66%, Points required for next Level: 17

    Posts
    50
    Thanks
    20
    Thanked 0 Times in 0 Posts

    Variable change from significant to not significant




    I run a multiple regression with 4 main drivers and some control variables.

    Then I had to run another regression but with six main drivers (of which 4 the same as the previous regression) and the same control variables.

    Because I had to inlcude 2 more variables, I had to reduce my sample by 4.

    When I compare the two outputs, one variable that was significant in the first regression has become not significant. Even when I omit the other 2 variables in the 2nd regression the variable that was significant in the 1st regression stays not significant.
    Is it because in the 2nd regression my sample is slightly smaller?

  2. #2
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Variable change from significant to not significant

    You could tell us how large your sample size was, and what was actually happening. "Significant" is not very informative (could mean p=0.049 or p=0.00000000001), same with "not significant" (p=0.05? p=0.98?). You could describe the regression weights, standard errors, and precise p values for the variables in question.

    With kind regards

    Karabiner
    »Jetzt kann mich der Führer mal am Arsch lecken.« (Ernst Kuzorra, 1941)

  3. The Following User Says Thank You to Karabiner For This Useful Post:

    Jazz3 (04-10-2017)

  4. #3
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Variable change from significant to not significant

    also what happened to the beta coefficients?
    Stop cowardice, ban guns!

  5. The Following User Says Thank You to hlsmith For This Useful Post:

    Jazz3 (04-10-2017)

  6. #4
    Points: 533, Level: 10
    Level completed: 66%, Points required for next Level: 17

    Posts
    50
    Thanks
    20
    Thanked 0 Times in 0 Posts

    Re: Variable change from significant to not significant

    In the 1st regression, I have 4 main drivers and a total N of 348, I run the regression on robust because hettest was significant

    Var /Coefficient/ robust std. error/ t / P value
    X1/ .902 .1516 5.95 0.000
    X2/ -.521 .1023 -5.10 0.000
    X3/ .221 .1015 2.19 0.030
    X4/ .037 .0823 0.45 0.650

    In the 2nd regression I have six main drivers (of which 4 are from the previous regression) and a total N of 324, also run the regression on robust. X3 is the variable that becomes not significant.

    Variable / coefficient / robust std. error/ t / P value
    X1 .847 .1382 6.13 0.000
    X2 -.5214 .0973 -5.36 0.000
    X3 .0333 .1060 0.31 0.753
    X4 -.081 .0763 -1.07 0.287
    X5 .0088 .0858 0.10 0.918
    X6 -.360 .1107 -3.25 0.001

    The only difference in the sample between the 1st and the 2nd regression is that the sample is reduced by 4 (= dependent variable) so that I could include the other 2 variables in the 2nd regression. Because its over a time period of six years, I guess the total sample reduces then by 24

  7. #5
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Variable change from significant to not significant

    Looks very much like in the 2nd model, X6 explains the variance previously
    explained by X2.
    Couldn't quite follow you with regard to number of missing cases (4? 24?).
    What suprises me:
    Even when I omit the other 2 variables in the 2nd regression the variable that was significant in the 1st regression stays not significant.
    So what were the coefficients in that analysis (the same analysis as the first, but
    with a reduced sample?).

    Because its over a time period of six years
    Does that mean that you followed-up the same subjects (n=58?) over
    6 time periods? If yes, then your analysis is incorrrect if you treat subjects
    measured repeatedly as if they were independent cases.

    With kind regards

    Krabiner
    »Jetzt kann mich der Führer mal am Arsch lecken.« (Ernst Kuzorra, 1941)

  8. #6
    Omega Contributor
    Points: 38,396, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,001
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Variable change from significant to not significant

    Pay heed to Karabiners questions on your data structure.


    Just for giggles, why don't you use model 1 on model 2's subsample and see what X3 looks like with and without the other new variables. You should also look at the collinearity between variables and since X3 decreases so much is there potential confounding or mediation between variables. You should draw out how you think the covariates influence each other and the dependent variable.
    Stop cowardice, ban guns!

  9. #7
    Points: 533, Level: 10
    Level completed: 66%, Points required for next Level: 17

    Posts
    50
    Thanks
    20
    Thanked 0 Times in 0 Posts

    Re: Variable change from significant to not significant

    Quote Originally Posted by Karabiner View Post
    Looks very much like in the 2nd model, X6 explains the variance previously
    explained by X2.
    Couldn't quite follow you with regard to number of missing cases (4? 24?).
    What suprises me:

    So what were the coefficients in that analysis (the same analysis as the first, but
    with a reduced sample?).


    Does that mean that you followed-up the same subjects (n=58?) over
    6 time periods? If yes, then your analysis is incorrrect if you treat subjects
    measured repeatedly as if they were independent cases.

    With kind regards

    Krabiner

    In the 2nd regression with the reduced sample

    coeff std. error t p value
    X1| .9149 .1452 6.30 0.000
    X2| -.4557 .0970 -4.70 0.000
    X3| .1360 .1022 1.33 0.184
    X4| -.0579 .0807 -0.72 0.474

    thus not the same as in the first regression


    What I mean with the sample is that, I have 58 categories with their market shares over a six year period.
    So my data is listed as followed:
    column 1 column2 column3
    year 1 category a share category a year 1
    year 1 category ab share category ab year 1
    year 1 category b share category b year 1
    etc etc etc
    year 2 category a
    year 2 category ab
    etc.
    year 3 category a
    year 3 category ab

    Column 3 is my dependent variable

    The 2nd regression has 4 categories less (Because I had to include the other two variables, and since for the 4 categories the variables weren't available I had to exclude them from the analysis) and thereby thus a total of 24 less observations than the first regression


    I checked for collinearity and didn't find anything above .6

  10. #8
    TS Contributor
    Points: 17,779, Level: 84
    Level completed: 86%, Points required for next Level: 71
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,542
    Thanks
    56
    Thanked 640 Times in 602 Posts

    Re: Variable change from significant to not significant

    You have small change of the coefficient for variable X3 between model 1 and model 3. Seemingly, the missingness of 24 values caused this difference, which in turn changed the p-value.
    »Jetzt kann mich der Führer mal am Arsch lecken.« (Ernst Kuzorra, 1941)

  11. The Following User Says Thank You to Karabiner For This Useful Post:

    Jazz3 (04-23-2017)

  12. #9
    Points: 533, Level: 10
    Level completed: 66%, Points required for next Level: 17

    Posts
    50
    Thanks
    20
    Thanked 0 Times in 0 Posts

    Re: Variable change from significant to not significant


    Quote Originally Posted by Karabiner View Post
    You have small change of the coefficient for variable X3 between model 1 and model 3. Seemingly, the missingness of 24 values caused this difference, which in turn changed the p-value.
    I run the original model (where X3 was significant) on the sample of the 2nd model (where X3 is not significant) and X3 is significant.

    I take this back:
    "Even when I omit the other 2 variables in the 2nd regression the variable that was significant in the 1st regression stays not significant."

    I overlooked: when I omit the 2 variables in the 2nd regression, X3 is significant.
    Thus I included X5 and X3 was still significant, when I added X6, X3 became insignificant. Thus I checked for correlation

    But between X3 and X6 the correlation is "only" -0.5 (X3 and X1 correlate on -0.7, but that's not the issue here?)

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats