We set some value, called 
, as our maximum tolerance for type I error rate. That is, we accept that our work could reject true null hypotheses 
of the time the null hypothesis is true. In the common situation of 
, we accept that to be 
. In fact, 
is so common that it typically is implied when no 
is specified, and we consider p-values of 
or smaller to be “small” p-values.
Then we run the test and calculate a p-value. If 
, we reject the null hypothesis in favor of the alternative hypothesis.
I got into a discussion with someone who arguably deals with statistics much more than I do, but I nevertheless think they are wrong on the subject. He was explaining p-values and alpha-values and how one should reject the null hypothesis if the p-value was less than the alpha value and accept it otherwise. Given everything he explained earlier about what the p-value and alpha values mean, it seems to me that this interpretation is wrong and that rather you should reject (as "statistical likely" of course) the null hypothesis (i.e. accepting the test hypothesis as statistically likely) if the p-value is less than alpha, but not reject the null hypothesis otherwise. The difference is subtle, but if switched the test hypothesis and null hypothesis, then the p-value would become (1-original p-value) while alpha would remain unchanged. By using the same logic, we wouldn't be rejecting the original test hypothesis / accepting the original test hypothesis as statistically likely unless the original p-value were above (1-alpha). So in mind, there's really 3 areas to think about:
p-value <= alpha: Accept test hypothesis as statistically likely.
alpha < p-value < 1-alpha: Not enough statistical evidence to strongly suggest one hypothesis over another
1-alpha <= p-value: Accept the null hypothesis as statistically likely.
This person said I was simply wrong and we had to either reject or accept the null hypothesis -- that we had to make a binary choice. I argued that in that case we should either "reject the null hypothesis" or "not reject the null hypothesis" instead of "reject the null hypothesis" or "accept the null hypothesis". He still claimed I was wrong.
Anyway, it's been bugging me and that's why I'm here. Who's right? Thank you for helping me with my understanding.
Videos
When do you reject the null hypothesis?
The significance level is the probability of rejecting the null hypothesis when it is true. Commonly used significance levels are 0.01, 0.05, and 0.10.
Remember, rejecting the null hypothesis doesn't prove the alternative hypothesis; it just suggests that the alternative hypothesis may be plausible given the observed data.
The p -value is conditional upon the null hypothesis being true but is unrelated to the truth or falsity of the alternative hypothesis.
Are all p-values below 0.05 considered statistically significant?
Statistical significance depends on factors like the study design, sample size, and the magnitude of the observed effect.
A p-value below 0.05 means there is evidence against the null hypothesis, suggesting a real effect. However, it's essential to consider the context and other factors when interpreting results.
Researchers also look at effect size and confidence intervals to determine the practical significance and reliability of findings.
Can a non-significant p-value indicate that there is no effect or difference in the data?
There could still be a real effect or difference, but it might be smaller or more variable than the study was able to detect.
Other factors like sample size, study design, and measurement precision can influence the p-value. It's important to consider the entire body of evidence and not rely solely on p-values when interpreting research findings.
We set some value, called , as our maximum tolerance for type I error rate. That is, we accept that our work could reject true null hypotheses of the time the null hypothesis is true. In the common situation of , we accept that to be . In fact, is so common that it typically is implied when no is specified, and we consider p-values of or smaller to be “small” p-values.
Then we run the test and calculate a p-value. If , we reject the null hypothesis in favor of the alternative hypothesis.
The outcome of a hypothesis test is reported in two ways:
- The p-value is p where p is a given small number.
- The null hypothesis is rejected at the α significance level; usually α = 0.05.
If the p-value p is smaller than α, then the null hypothesis is rejected at the α level. And if the null hypothesis is rejected, we know the corresponding p-value is < α. However, we don't know the exact p-value. It might be 0.049, it might be 0.000001.
The first statement is preferred because it presents more information (the strength of evidence against the null hypothesis). Note that we don't say that p-value is significant or not; it's enough to report the p-value since it's obvious that the null hypothesis will be rejected at all significance levels α > p.
This surely will not top the list of possible "cool undergraduate-level tips", but simply recalling the definition of a p-value might be helpful (quoted from Wikipedia):
The probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.
So the smaller the probability, the smaller significance level at which we are willing to reject.
The standard mnemonic for remembering how to make a conclusion in a hypothesis test is:
If p is low, the null must go!
As to why this is the case, the best explanation of a classical hypothesis test is that it is the inductive anologue of a proof by contradiction. In a proof by contradiction we begin with a null hypothesis, show that this leads logically to a contradiction, and therefore reject the initial premise that the null is true. In a classical hypothesis test, we begin with a null hypothesis, show that this leads to a highly implausible result in favour of the alternative (so not quite a deductive contradiction, but close), and therefore reject the initial premise that the null is true. The p-value in this test is the probability of a result at least as conducive to the alternative hypothesis, assuming the null is true (see formal explanation here). If this is low then it means that something very implausible happened (under the assumption that the null is true) which gives the "contradiction" in the "inductive proof by contradiction".
I have been thinking about how we do not accept a null hypothesis if we reject it, and I am not sure if i do not understand it well enough, what I think is that we do not accept the null hypothesis because when we fail to reject the null hypothesis we are only saying that the alternative hypothesis is incorrect but that does not make it impossible to another alternative hypothesis to appear and this one be correct. Please let me know if this is correct
In case that the last paragraph is correct then I do not know why we say that we do not accept the null hypothesis if this is based in how we think things are, would it not be more appropiate to say that the null hypothesis is correct when we compare it to the the alternative that we just reject, because we do not know which alternative hypothesis might make us reject the null
Thank you