3 Formula Warehouse
This is the home for notation and formulas used thorugh this eBook. Most important equations will be located here.
3.1 Data Notation
Sample Sizes:
- \(n_j\) = number of pupils in class \(j\)
- \(N\) = number of classes
Indicators:
- \(i \in (1, 2, \dots, n_j)\) = index for pupil number
- \(j \in (1, 2, \dots, N)\) = index for class number
| Level | Type of Variable | Symbol | pupil \(i\) in class \(j\) |
|---|---|---|---|
| 1 | Outcome (Dependent) | \(Y\) | \(Y_{ij}\) |
| 1 | Predictor (Independent) | \(X_1\) | \(X_{1ij}\) |
| 1 | Predictor (Independent) | \(X_2\) | \(X_{2ij}\) |
| 2 | Predictor (Independent) | \(Z\) | \(Z_j\) |
3.2 Single-level Regression Analysis
3.2.1 The Only Equation
Since we are don’t have or are ignoring clustering, there is only one level.
Single-Level Regression Equation \[ \overbrace{Y_{ij}}^{Outcome} = \underbrace{\beta_{0}}_{\text{Fixed}\atop\text{intercept}} + \underbrace{\beta_{1}}_{\text{Fixed}\atop\text{slope } X_1} \overbrace{X_{1ij}}^{\text{Predictor 1}} + \underbrace{\beta_{2}}_{\text{Fixed}\atop\text{slope } X_2} \overbrace{X_{2ij}}^{\text{Predictor 2}} + \underbrace{e_{ij}}_{\text{Random}\atop\text{residuals}} \tag{Hox 2.1} \]
3.3 Multi-level Regression Analysis
Continue taking into account fixed slopes for two Level 1 variables, \(X_1\) and \(X_2\).
3.3.1 Level 1 Regression Equation*
\[ \overbrace{Y_{ij}}^{\text{Level 1}\atop\text{Outcome}} = \underbrace{\beta_{0j}}_{\text{Level 2}\atop\text{intercepts}} + \underbrace{\beta_{1j}}_{\text{Level 2}\atop\text{slopes}} \overbrace{X_{1ij}}^{\text{Level 1}\atop\text{Predictor 1}} + \underbrace{\beta_{2j}}_{\text{Level 2}\atop\text{slopes}} \overbrace{X_{2ij}}^{\text{Level 1}\atop\text{Predictor 2}} + \underbrace{e_{ij}}_{\text{Random}\atop\text{residuals}} \tag{Hox 2.1} \]
Now we take clustering into account and include random intercepts (\(\beta_{0j}\)) and slopes (\(\beta_{1j}, \beta_{2j}\)), as well as including a single Level 2 variable, \(Z\) that interacts with both Level 1 variables.
3.3.2 Level 2 Regression Equations
3.3.2.1 Random Intercepts:
\[ \overbrace{\beta_{0j}}^{\text{Level 2}\atop\text{intercepts}} = \underbrace{\gamma_{00}}_{\text{Fixed}\atop\text{intercept}} + \underbrace{\gamma_{01}}_{\text{Fixed}\atop\text{slope } Z} \overbrace{Z_{j}}^{\text{Level 2}\atop\text{Predictor 3}} + \underbrace{u_{0j}}_{\text{Intercept}\atop\text{residual}} \tag{Hox 2.3} \]
3.3.2.2 Random Slopes
- For the first predictor, \(X_1\):
\[ \overbrace{\beta_{1j}}^{\text{Level 2}\atop\text{slopes}} = \underbrace{\gamma_{10}}_{\text{Fixed}\atop\text{Slope } X_1} + \underbrace{\gamma_{11}}_{\text{Fixed}\atop X_1 \times Z} \overbrace{Z_{j}}^{\text{Level 2}\atop\text{Predictor 3}} + \underbrace{u_{1j}}_{\text{Slope } X_1\atop\text{residual}} \tag{Hox 2.4a} \]
- For the second predictor, \(X_2\):
\[ \overbrace{\beta_{2j}}^{\text{Level 2}\atop\text{slopes}} = \underbrace{\gamma_{20}}_{\text{Fixed}\atop\text{Slope } X_2} + \underbrace{\gamma_{21}}_{\text{Fixed}\atop X_2 \times Z} \overbrace{Z_{j}}^{\text{Level 2}\atop\text{Predictor 3}} + \underbrace{u_{2j}}_{\text{Slope } X_2\atop\text{residual}} \tag{Hox 2.4a} \]
3.3.2.3 Merging the Equations
Starting with Level 1 equation (2.1) and allow the \(\beta\)’s to be varry for each class and plug in the level 2 equations (2.3 and 2.4) into the level 1 equation (2.1) to make the combined equation. \[ Y_{ij} = \overbrace{(\gamma_{00} + \gamma_{01} Z_{j} + u_{0j})}^{\beta_{0j}} + \overbrace{(\gamma_{10} + \gamma_{11} Z_{j} + u_{1j})}^{\beta_{1j}} X_{1ij} + \overbrace{(\gamma_{20} + \gamma_{21} Z_{j} + u_{2j})}^{\beta_{2j}} X_{2ij} + e_{ij} \]
Use the distributive property of multiplication to get rid of the parentheses. \[ Y_{ij} = \overbrace{\gamma_{00} + \gamma_{01} Z_{j} + u_{0j}}^{\beta_{0j}} + \overbrace{\gamma_{10} X_{1ij} + \gamma_{11} Z_{j} X_{1ij} + u_{1j} X_{1ij}}^{\beta_{1j} \times X_{1ij}} + \overbrace{\gamma_{20} X_{2ij} + \gamma_{21} Z_{j} X_{2ij} + u_{2j} X_{2ij}}^{\beta_{2j} \times X_{2ij}} + e_{ij} \]
3.3.3 Combinded, Multilevel Regression Equation
Collect ‘like-terms’ (i.e. get the \(\gamma\)’s together and the \(u\)’s together)
Combinded, Multilevel Regression Equation - Generic
\[ Y_{ij} = \overbrace{\gamma_{00} + \gamma_{10} X_{1ij} + \gamma_{20} X_{2ij} + \gamma_{01} Z_{j} + \gamma_{11} Z_{j} X_{1ij} + \gamma_{21} Z_{j} X_{2ij}}^{\text{Fixed part}\atop\text{Deterministic}} + \\ \underbrace{u_{0j} + u_{1j} X_{1ij} + u_{2j} X_{2ij} + e_{ij} }_{\text{Random part}\atop\text{Stochastic}} \tag{Hox 2.5} \]
3.3.4 Parameters
| Type | Parameter of Interest | Estimates This |
|---|---|---|
| Fixed | Intercept | \(\gamma_{00}\) |
| Fixed | Main Effect of \(X_1\) | \(\gamma_{10}\) |
| Fixed | Main Effect of \(X_2\) | \(\gamma_{20}\) |
| Fixed | Main Effect of \(Z\) | \(\gamma_{01}\) |
| Fixed | Cross-Level interaction between \(X_1\) and \(Z\) | \(\gamma_{11}\) |
| Fixed | Cross-Level interaction between \(X_2\) and \(Z\) | \(\gamma_{21}\) |
| Random | Variance in random intercepts, \(var[u_{0j}]\) | \(\sigma^2_{u0}\) |
| Random | Variance in random slope of \(X_1\), \(var[u_{1j}]\) | \(\sigma^2_{u1}\) |
| Random | Variance in random slope of \(X_2\), \(var[u_{2j}]\) | \(\sigma^2_{u2}\) |
| Random | Covariance between random intercepts and random slope of \(X_1\), \(cov[u_{0j}, u_{1j}]\) | \(\sigma^2_{u01}\) |
| Random | Covariance between random intercepts and random slope of \(X_2\), \(cov[u_{0j}, u_{2j}]\) | \(\sigma^2_{u02}\) |
| Random | Covariance between random slopes of \(X_1\) and \(X_2\), \(cov[u_{1j}, u_{2j}]\) | \(\sigma^2_{u12}\) |
| Random | Residual Variance \(var[e_{ij}]\) | \(\sigma^2_{e}\) |
The \(u_{1j}\) and \(u_{2j}\) terms allow for heteroscedasticity by fitting different error terms for different values of \(X_1\) and \(X_2\). The HOV assumption is that AFTER accounting for this, the remaining residuals are HOV.
3.4 Intraclass Correlation (ICC)
3.4.1 Two Level Models
Combined, Multilevel Model Equation - Null Model, 2 levels \[ \overbrace{Y_{ij}}^{Outcome} = \underbrace{\gamma_{00}}_{\text{Fixed}\atop\text{intercept}} + \underbrace{u_{0j}}_{\text{Random}\atop\text{intercepts}} + \underbrace{e_{ij}}_{\text{Random}\atop\text{residuals}} \tag{Hox 2.8} \]
Although the Null model above does not explain any variance in the dependent variable, since there are no independent variables, it does decompose (i.e. divide up) the variance into two pieces. We can compute the amount of total variance in the outcome that is attribute to the clustering of Level 1 untis (micro-units) into clusters of Level 2 units (macro-units) verses the total variance.
Intraclass Correlation (ICC) Formula, 2 level model \[ \overbrace{\rho}^{\text{ICC}} = \frac{\overbrace{\sigma^2_{u0}}^{\text{Random Intercept}\atop\text{Variance}}} {\underbrace{\sigma^2_{u0}+\sigma^2_{e}}_{\text{Total}\atop\text{Variance}}} \tag{Hox 2.9} \]
3.4.2 Three Level Models
Indicators:
- \(i\) = index for units in the lowest level (Level 1)
- \(j\) = index for units in the middle level (Level 2)
- \(k\) = index for units in the highest level (Level 3)
Combined, Multilevel Model Equation - Null Model, 3 levels \[ \overbrace{Y_{ijk}}^{Outcome} = \underbrace{\gamma_{000}}_{\text{Fixed}\atop\text{intercept}} + \underbrace{v_{0k }}_{\text{Random Intercepts}\atop\text{Level 3}} + \underbrace{u_{0jk}}_{\text{Random Intercepts}\atop\text{Level 2}} + \underbrace{e_{ijk}}_{\text{Random}\atop\text{residuals}} \tag{Hox 2.15} \]
If you are interested in teh decomposition of variance across all levels, use the Davis and Scott method:
Intraclass Correlation (ICC) Formula, 3 level model - Davis and Scott Method \[ \overbrace{\rho_{mid}}^{\text{ICC}\atop\text{at level 2}} = \frac{\overbrace{\sigma^2_{u0}}^{\text{Random Intercept}\atop\text{Variance Level 2}}} {\underbrace{\sigma^2_{v0}+\sigma^2_{u0}+\sigma^2_{e}}_{\text{Total}\atop\text{Variance}}} \tag{Hox 2.16} \] \[ \overbrace{\rho_{top}}^{\text{ICC}\atop\text{ at level 3}} = \frac{\overbrace{\sigma^2_{u0}}^{\text{Random Intercept}\atop\text{Variance Level 3}}} {\underbrace{\sigma^2_{v0}+\sigma^2_{u0}+\sigma^2_{e}}_{\text{Total}\atop\text{Variance}}} \tag{Hox 2.17} \]
If you would like to estimate the expected (population) correlation between two randomly chosen elements of the same group:
Intraclass Correlation (ICC) Formula, 3 level model - Siddiqui Method \[ \overbrace{\rho_{mid}}^{\text{ICC}\atop\text{at level 2}} = \frac{\overbrace{\sigma^2_{v0}+\sigma^2_{u0}}^{\text{Random variance}\atop\text{at levels 2 & 3}}} {\underbrace{\sigma^2_{v0}+\sigma^2_{u0}+\sigma^2_{e}}_{\text{Total}\atop\text{Variance}}} \tag{Hox 2.18} \] \[ \overbrace{\rho_{top}}^{\text{ICC}\atop\text{ at level 3}} = \frac{\overbrace{\sigma^2_{u0}}^{\text{Random variance}\atop\text{at only level 3}}} {\underbrace{\sigma^2_{v0}+\sigma^2_{u0}+\sigma^2_{e}}_{\text{Total}\atop\text{Variance}}} \tag{Hox 2.19} \]
3.5 Proporion of Variance Explianed
See pages 61-63 of Hox, Moerbeek, and Van de Schoot (2017)
http://journals.sagepub.com/doi/10.1177/1094428114541701
Analogous to multiple \(R^2\) - done seperately by level
- \(BL\) = Baseline model (Null)
- \(MC\) = Model to Compare to
3.5.1 Level 1 Variance Explained
MODELS SHOULD NOT INCLUDE ANY RANOM EFFECTS, OTHER THAN RANDOM INTERCEPTS.
Different approaches differ in values and meaning.
3.5.1.1 Snijders and Bosker
- Explained variance is a proportion of the total variance, because in principle first-level variables can explain all variation, including the variation at the second level.
- Correction removes the spurious increase in \(R^2\) when random slopes are added to a model
Snijders and Bosker Formula - Level 1
Random Intercepts Models Only, address potential negative \(R^2\) issue \[ R^2_1 = 1 - \frac{\sigma^2_{e-MC} + \sigma^2_{u0-MC}} {\sigma^2_{e-BL} + \sigma^2_{u0-BL}} \]
3.5.1.2 Raudenbush and Bryk
- Explained variance is a proportion of first-level variance only
- A good option when the multilevel sampling process is is close to two-stage simple random sampling
Raudenbush and Bryk Approximate Formula - Level 1 approximate \[ approx \;R^2_1 = \frac{\sigma^2_{e-BL} - \sigma^2_{e-MC}} {\sigma^2_{e-BL} } \tag{Hox 4.8} \]
3.5.2 Level 2 Variance Explined
3.5.2.1 Snijders and Bosker
Snijders and Bosker Formula Extended - Level 2 \[ R^2_2 = 1 - \frac{\frac{\sigma^2_{e-MC}}{B} + \sigma^2_{u0-MC}} {\frac{\sigma^2_{e-BL}}{B} + \sigma^2_{u0-BL}} \]
\(B\) is the average size of the Level 2 units (schools). Technically, you should use the harmonic mean, but unless the clusters differ greatly in size, it doesn’t make a huge difference.
3.6 Using \(\LaTeX\) for Equation Typesetting
R markdown is a user friendly, simplified language that allows for more complex formating utilizing standard \(\LaTeX\) code. A great resource for learning how to many common tasks in \(\LaTeX\) is the Sharewebsite.
Specific mathematical equation documentation may be found on the Mathematical Expressions subpage.
There are also many websites that offer Point-n-click interfaces to build \(\LaTeX\) equations, including: Host Math, Code Cogs, LaTeX 4 Technics, and Sci-Weavers