Cfxtrjtrk

I'm having difficulties finding a clear rule for when a series of test should be considered a multiple comparison and when we should apply p-value corrections (like Bonferroni).

I understand corrections must be applied every time multiple hypothesis are tested using the same dataset. A classical example is a Post-hoc Tukey test on data from an ANOVA.

However, what is the proper definition of a "dataset"? Whenever two tests share the a sample, are they the same dataset? Do they need to share all samples? The tests must share the same hypothesis?

I found many questions related to mine in this forum and online, but all of them seem to handle examples. If their particular case is or is not a multiple comparison and whether it needs correction, but none seem to come with a objective definition of "dataset".

The justification for control of multiple tests has to do with the family of tests. The family of tests can be mutually independent, which is often the case when they are drawn from different datasets; if so, Bonferroni is a good way to control for FWER. But in general, the concept of a dataset doesn't even enter the picture when discussing multiplicity.

It's assumed (incorrectly) that data in different datasets must, by design, be independent whereas two tests calculated with the same dataset must be dependent (also not necessarily correct). To justify and discuss the type of testing correction to use, one should consider the "family of tests". If the tests are dependent or correlated (that is to say that the $p$-value of one test actually depends on the $p$-value from another test), Bonferroni will be conservative. (NB: some rather dicey statistical practices can make Bonferroni anti-conservative, but that really boils down to non-transparency. For instance: test main hypothesis A. If main hypothesis non-significant, test hypotheses A and B and control with Bonferroni. here you allowed yourself to test B only because A was negative, this makes tests A and B negatively correlated even if the data contributing to these tests are independent.)

When the tests are independent, Bonferroni as you know is non-conservative in controlling the FWER. There is some grey area with respect to what constitutes a family of tests. This can be illustrated by considering subgroup analyses, here a global test may or may not have been significant, then the sample population is divvied up into K distinct groups. These groups are likely independent because they are arbitrary combinations of independent data from the parent dataset. You can view them as K distinct datasets, or 1 divided dataset, it doesn't matter. The point is that you conduct K tests. If you report the global hypothesis: at least one group showed heterogeneity of effect from the other groups, then you don't have to control for multiple comparisons. If, on the other hand, you report specific subgroup findings, you have to control for the K number of tests it took you to sniff that finding out. This is the XKCD Jelly Bean comic in a nutshell.

This is a much harder question than one would think and I doubt there are clear answers. The answer is relatively clear when we talk about clinical trials for regulatory purposes (whatever the regulatory authority says). I have the impression that this is an area of pragmatic traditions that have evolved in a kind of ad-hoc and not necessarily philosophically consistent manner within each field of science. There are simply some standard conventions that are typically (but not always) followed in certain fields. However, even with a field where type I error rate control per study has a lot of tradition such as medicine, there is still a debate on this topic.