Given a null distribution, how can I calculate a p-value for my test statistic?
9 views (last 30 days)
For example, let's say I have two groups and want to see if their means are significantly different. However, I want to do so in a shuffling/permutation framework.
Accordingly, I shuffle the group labels across data points, calculate the difference between means, and do so 5000 times to create a null distribution.
I have my original unshuffled mean difference, and see that it is in the top 2.5 percentile of the null distribution. I can thus conclude that the difference is significant at the two-tailed level.
However, in this context, how can I compute the exact p-value of my original mean difference value with the null distribution? I am lost when it comes to finding the right function to do so.
Jeff Miller on 3 Feb 2022
The one-tailed p value is just the tail probability of your original unshuffled difference relative to the null distribution that you created by shuffling. In your example where the unshuffled mean is at the edge of the top 2.5% of the null distribution, p=.025.
For two-tailed testing, the p would be double this tail probability (e.g., 2*.025=.05)