Abstract
Design Analysis was introduced by Gelman & Carlin (2014) as an extension of Power Analysis. Traditional power analysis has a narrow focus on statistical significance. Design analysis, instead, evaluates together with power levels also other inferential risks (i.e., Type M error and Type S error), to assess estimates uncertainty under hypothetical replications of a study. Given an hypothetical value of effect size and study characteristics (i.e., sample size, statistical test directionality, significance level), Type M error (Magnitude, also known as Exaggeration Ratio) indicates the factor by which a statistically significant effect is on average exaggerated. Type S error (Sign), instead, indicates the probability of finding a statistically significant result in the opposite direction to the hypothetical effect. Although Type M error and Type S error depend directly on power level, they underline valuable information regarding estimates uncertainty that would otherwise be overlooked. This enhances researchers awareness about the inferential risks related to their studies and helps them in the interpretation of their results. However, design analysis is rarely applied in real research settings also for the lack of dedicated software. To know more about design analysis consider Gelman & Carlin (2014) and Lu et al. (2018). While, for an introduction to design analysis with examples in psychology see Altoè et al. (2020) and Bertoldo et al. (2020).
Claudio Zandonella Callegher
Post-Doc Researcher
My research interests include the formalization of psychological thoeries, Bayesian methods in Behavioral Sciences, and everything related to programming in R!