File Exchange

image thumbnail

Multiple Testing Toolbox

version 1.1.0 (254 KB) by Víctor Martínez-Cagigal
This toolbox includes eight different approaches to deal with the Multiple Comparisons Problem.


Updated 26 Apr 2021

View Version History

View License

The "Multiple Testing Toolbox" includes a compilation of eight different algorithms to deal with the Multiple Comparisons Problem, focusing on both the correction of the Family-Wise Error Rate (FWER), and the False Discovery Rate (FDR) as well.

All the included methods provide the adjusted p-values, the adjusted significance level, the number of rejected null hypotheses and extra relevant information. An example of use of each one is provided in the header.

The algorithms are enumerated below:
- fwer_bonf: Bonferroni correction of the FWER.
- fwer_sidak: Šidák correction of the FWER.
- fwer_holmbonf: Holm-Bonferroni correction of the FWER (also known as sequential Bonferroni).
- fdr_BH: Benjamini-Hochberg correction of the FDR.
- fdr_BY: Benjamini-Yekutieli correction of the FDR.
- fdr_storey: Storey correction of the positive FDR, which returns the computed q-values.
- mt_fisher: Combined Probability of Fisher correction.

Check the header of each algorithm for further information.

Please, cite this project in your publications if you are going to use these scripts for your research work.

Cite As

Víctor Martínez-Cagigal (2021). Multiple Testing Toolbox (, MATLAB Central File Exchange. Retrieved .

Comments and Ratings (4)

@Adam Zabicki Yes, you are right. The original algorithm stops checking values if the previous one already did not reject the null hypothesis. Although the corrected p-values were right, the binary vector that indicated what p-values reject the null hypothesis should have considered that :) Thus, I just updated the algorithm to take it into account. Thanks!


Adam Zabicki

dear victor,
nice idea to collect various multiple testing correction methods in one toolbox. Exactly what i was looking for :)

but, as far as i see it, there is a bug in the "fwer_holmbonf" function. to my knowledge, the first and sorted p-Value which is not passing the corrected significance-threshold is some kind of abortion criterion, and no further p-value should become significant.

From "":
"Select the lowest p-value and compare it to α/k. If the p-value is lower, then reject that hypothesis and perform the same selection with the remaining k−1 hypotheses and a threshold of α/(k−1). Repeat the process until the selected p-value is not smaller, at which time all remaining hypotheses should be accepted."

For example, let
p = [0.043 , 0.048 , 0.041 , 0.001 , 0.046];
here, just the 4th pValue should be significant. But the function ( [~,~,h] = sami.stat.fwer_holmbonf(p,0.05); ) returns
h = [0 1 0 1 0]

I would expect
h = [0 0 0 1 0].

Which is return by

What are your thoughts? Or is the other algorithm buggy?


frederic chambat

Very complete and works easily

MATLAB Release Compatibility
Created with R2018b
Compatible with any release
Platform Compatibility
Windows macOS Linux

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!