2 Citations for Writeup
This chapter is UNDER CONSTRUCTION, so check back often.
2.1 MLM: Multilevel Modeling, aka Mixed Effects Regression
2.1.1 General MLM
Fitzmaurice, G. M., Laird, N. M., Ware, J. H. (2004). Applied Longitudinal Analysis. Hoboken, NJ: John Wiley & Sons.
Hox, J. J., Moerbeek, M., & Van de Schoot, R. (2017). Multilevel analysis: Techniques and applications. Routledge.
Gelman, A., & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (Vol. 1). Sage.
Singer, J. D., Willett, J. B., & Willett, J. B. (2003). Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press.
Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Los Angeles; London, SAGE.
2.1.2 Pairwise Differenes Effect Size (Standardized Mean Difference, SMD)
There are a number of ways to calculate standardized mean difference (SMD) effect sizes and there really is not a universally accepted “right” way. Using residual SD and the square root of the sum of variance components from a mixed model for standardization are NOT traditional Cohen’s d statistics, so they should be referred to as standardized effect sizes.
The important thing is to detail which method you are using and make sure to label it appropriately.
Probably the effect size calculation with the most academic support is taking the square root of the sum of all variance components in the model. This method was first discussed by Westfall, Judd, and Kenny (2014) and later included in a statistical tutorial on Effect Sizes and Power Analysis in Mixed Effects Models by Marc Brysbaert and Michael Stevens (2018).
We can add interpretations to these SMD effects. There have been different propositions regarding the qualitative interpretation of effect sizes’. The original interpretation given by Cohen suggests:
- Small: < 0.5
- Moderate: 0.5 - 0.7
- Large: > 0.8
More recently Hopkins suggested a revised scale with threshold values at:
- 0.2 small
- 0.6 moderate
- 1.2 large
- 2.0 very large
- 4.0 nearly perfect
Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6646942/ Links to an external site.
Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior research methods, 49(4), 1494-1502. https://link.springer.com/article/10.3758/s13428-016-0809-y#Sec3 Links to an external site.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fxge0000014
2.1.2.1 Example Write-up for Methods-Analysis section:
Follow-up pairwise comparisons utilizing Kenward-Rodger degrees of freedom (Luke, 2017) were made without/with?? controlling for multiple comparisons. Corresponding effect sizes give Cohen’s d-like standardized mean differences (SMD) utilizing the standard deviation based on the sum of the MLM variance components (Brysbaert & Stevens, 2018).
2.3 GEE: Generalized Estimating Equations
Fitzmaurice, G. M., Laird, N. M., Ware, J. H. (2004). Applied Longitudinal Analysis. Hoboken, NJ: John Wiley & Sons.
Hardin, J. W., Hilbe, J. M. (2003). Generalized Estimating Equations. Boca Raton: Chapman & Hall/CRC.
Liang, K.-Y., Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73:13–22.
McCullagh, P., Nelder, J. A. (1989). Generalized Linear Models. 2nd ed. London: Chapman & Hall.
Nelder, J. A., Wedderburn, R. W. M. (1972). Generalized linear models. J. Roy. Statist. Soc.,Ser. A 135:370–384.
Burger, H. G., Dudley, E. C., Cui, J., Dennerstein, L., Hopper, J. L. (2000). A prospective longitudinal study of serum testosterone levels during and after the menopause transition. J. Clin. Endocrinol. Metabolism 85:2832–2838.
2.3.1 QIC: Comparing GEE Models
The QIC involves using the quasi-likelihood constructed under the working independence model and the naive and robust covariance estimates of estimated regression coefficients.
- QIC-U is asymptotic, not appropriate for selected an optimal correlation structure
Cui, J., & Qian, G. (2007). Selection of working correlation structure and best model in GEE analyses of longitudinal data. Communications in statistics—Simulation and computation®, 36(5), 987-996. Download PDF
We propose to choose the best correlation structure first and then choose the best subsets of covariates. The best correlation structure was selected based on the full model including all covariates. However, if we use only a subset of covariates to select the best correlation structure, we may end up with a different correlation structure. This is a dilemma as to choose the best correlation structure first or the best subset of covariates first. It seems that the full model contains most information to predict the outcome variable and is reasonable to use it to select the best correlation structure first (Hardin and Hilbe, 2003).
Burnham, K. P. and D. R. Anderson. 2002. Model selection and multimodel inference: a practical information-theoretic approach. Second edition. Springer Science and Business Media, Inc., New York. Download PDF
Pan, W. 2001. Akaike’s information criterion in generalized estimating equations. Biometrics 57:120-125. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.0006-341X.2001.00120.x Download PDF
Wedderburn, R. W. M. (1974). Quasi-likelihood functions, generalized liner models, and the Gauss-newton method. Biometrika 61:439–447.