Statistical Significance

What is it that makes a change statistically significant? Is this relationship solely based on whether the allowance is larger than the value being tested for or the mean?

For example 2%+/-4% with Alpha being 5%, the allowance is 4%.
The answer states that it is not significant because the 4% allowance is greater than the 2%.

Does the alpha of 5% have any impact on statistical significance?