The "Multiple Testing Toolbox" includes a compilation of eight different algorithms to deal with the Multiple Comparisons Problem, focusing on both the correction of the Family-Wise Error Rate (FWER), and the False Discovery Rate (FDR) as well.
All the included methods provide the adjusted p-values, the adjusted significance level, the number of rejected null hypotheses and extra relevant information. An example of use of each one is provided in the header.
The algorithms are enumerated below:
- fwer_bonf: Bonferroni correction of the FWER.
- fwer_sidak: Šidák correction of the FWER.
- fwer_holmbonf: Holm-Bonferroni correction of the FWER (also known as sequential Bonferroni).
- fdr_BH: Benjamini-Hochberg correction of the FDR.
- fdr_BY: Benjamini-Yekutieli correction of the FDR.
- fdr_storey: Storey correction of the positive FDR, which returns the computed q-values.
- mt_fisher: Combined Probability of Fisher correction.
Check the header of each algorithm for further information.
Please, cite this project in your publications if you are going to use these scripts for your research work.
Víctor Martínez-Cagigal (2021). Multiple Testing Toolbox (https://www.mathworks.com/matlabcentral/fileexchange/70604-multiple-testing-toolbox), MATLAB Central File Exchange. Retrieved .
@Adam Zabicki Yes, you are right. The original algorithm stops checking values if the previous one already did not reject the null hypothesis. Although the corrected p-values were right, the binary vector that indicated what p-values reject the null hypothesis should have considered that :) Thus, I just updated the algorithm to take it into account. Thanks!
nice idea to collect various multiple testing correction methods in one toolbox. Exactly what i was looking for :)
but, as far as i see it, there is a bug in the "fwer_holmbonf" function. to my knowledge, the first and sorted p-Value which is not passing the corrected significance-threshold is some kind of abortion criterion, and no further p-value should become significant.
"Select the lowest p-value and compare it to α/k. If the p-value is lower, then reject that hypothesis and perform the same selection with the remaining k−1 hypotheses and a threshold of α/(k−1). Repeat the process until the selected p-value is not smaller, at which time all remaining hypotheses should be accepted."
For example, let
p = [0.043 , 0.048 , 0.041 , 0.001 , 0.046];
here, just the 4th pValue should be significant. But the function ( [~,~,h] = sami.stat.fwer_holmbonf(p,0.05); ) returns
h = [0 1 0 1 0]
I would expect
h = [0 0 0 1 0].
Which is return by https://de.mathworks.com/matlabcentral/fileexchange/69817-bonferroni_holm
What are your thoughts? Or is the other algorithm buggy?
Very complete and works easily
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!