Interpretation of my results (Significance varies a lot)

#1
Hi,

I have a problem with the interpretation of my results from a multiple linear regression. I want to evaluate the impact of a policy change (X) on an economic indicator (Y) and have included various covariates in the regression (6). The policy change occurred in 2015 and I have data from 2012 to 2018 available. I first run the regression with the full data set and it turned out highly significant. Afterwards, I played around a bit with the data range and got some weird results. If the data range is 2014 to 2016, it was also highly significant. However, if I used the data from 2013 to 2017, the results were insignificant (with p-values up to 0.6). Therefore, I decided to plot the significance (Y-axis) and the corresponding days from the policy implementation (X-axis). 1586221143220.png
As you can see in the plot, the results turn significant for at around 350 days away from the policy implementation date (data: 2014-2016) and after around 1100 days away from the policy implementation date (data 2012-2018). In between they are insignificant.

How am I supposed to interpret these results? How should I know which period correctly estimates the effect? Could there be a problem with my covariates? Interestingly, the estimation coefficient also becomes negative for the time period where the results are not significant.

Thanks a lot for your help, I am really lost with these results!
 

Karabiner

TS Contributor
#2
p-values are not a measure of effect size, therefore it is not clear whether your analysis makes much sense.
For example, a p value 0.01 (2-sided) corresponds to a z value 2.58, a largely different-looking p value 0.001
corrsponds to z=3.29, and 0.0001 corrsponds to 3.89. Not so impressive at it first seemed.
And the difference between significant and not significant is not itself statistically significant
If you had reason to assume that effects vary with time, then time*predictor interactions could have
been included into the model.

With kind regards

Karabiner
 
Last edited:
#3
Thanks a lot for your fast reply!

I am aware that the interpretation should be based on the estimate and not on the p-value.

I quickly plotted the development of the estimate (Y-axis) and the corresponding days from the policy implementation (X-axis).

1586248310785.png
My hypothesis was that the policy will have a positive effect on the economic indicator Other research indicates that the impact is expected to be around 15% to 25% which would mean the policy was of importance. The estimator is in this range when the results are significant. So, overall the results would be clear if there would not be this drop.

How should I deal with this drop?

Could I conclude the following:
Based on my results, the new policy had a significant impact of economic meaning on the economic indicator. However, the result is not very robust to data variations and therefore further research is needed.

Thanks a lot!
 

Karabiner

TS Contributor
#4
I don't know what your research area is, or the research questions, the variables or data or even sample size,
so I'm afraid I cannot tell you how you should interpret your results.

With kind regards

Karabiner
 
Last edited: