The barriers for academic researchers including multilevel models in their research have historically been due to lack of knowledge, lack of data, and lack of computing power. While multilevel methods are often discussed in terms theory, many students aren’t exposed to the realities of conducting these analyses. This is often because their program’s curriculum doesn’t leave room for it or because their methods instructors don’t know how conduct such analysis. This is unfortunate because incorporating these types of methods is getting easier every year.
Until recently, the lack of data on geographic indicators and lack of computing power restricted researchers from including these models. Since comparatively little geographic data was available in a digital/analyzable format, incorporating geographic indicators such as neighborhood, county, city, or state often required manual data entry. This process often included creating database variables based on digital scans of historical documents (if such data is available). For many researchers, such a task was of questionable value, since there wasn’t a lot of certainty of what the multi-level analyses would add to a particular project.
Now that the data are becoming available, and most researchers have access to the computing power to conduct such analyses on standard office machines, now is a good time to give multi-level models serious consideration. Doing so allows investigators to ask different questions. Multi-level modeling allows for the estimation of the influence of various indicators at different levels of analyses simultaneously. For example:
A researcher investigating the effect of state-level health policies on infant and maternal health in the United States might use multi-level modeling to model the individual risk of a mother giving birth to an infant with compromised birth outcomes, such as neural tube defects (NTDs). The investigator might be interested in knowing what this mother’s risk of giving birth to an infant with neural tube defects (NTDs) in states with public prenatal multivitamin distribution programs versus those without such programs. In the absence of data regarding a mother’s consumption of prenatal vitamins (and the association her risk of having a NTD affected birth), this might be an interesting place to start with analyses. Additionally, this investigator might be interested in an infinite number of characteristics of state health policies and their influence on infant health. If the investigator had a geographic indicator such as zip code, census tract, census block, county, city, or state then it would possible to look at a variety of area-level variables and their influence on individual-level outcomes.
It is not all rainbows and butterflies in regard to multi-level models. Having conducted a substantial part of my dissertation with these models, I feel compelled to offer a caveat emptor for researchers considering a potential multi-level project. These models can get out of hand pretty quickly. It is quite easy to merge geographic data into a working dataset and quickly find it extraordinarily difficult to interpret anything. I think there are a couple of important things to keep in mind, before embarking on such an endeavor.
The first is the number of geographic units you might have in relation to a particular outcome variable. In the example above, while NTDs are the most common birth effects, they occur in only 1 of 1000 births. There were 8.49 million births in the US in 2010. This means that we could expect that there would be about 8490 NTD affected births in the 2010 birth cohort. Assuming that each contributed 1/50th of that number[i], we are looking at potentially 169 births per state. If we begin comparing state-level programs to each other, we may quickly run into the problem of small n’s (declining power for statistical analyses).
The next thing to keep in mind is what the findings mean for your interpretation of your causal model. I find the “wheels tend to come off” pretty quickly when trying to interpret state characteristics that are continuous variables (“For every one percentage point increase in the state’s minority population as a proportion of the total population, black mothers have a 2 percent decrease in the risk of having an NTD effected birth, compared to white mothers.”) One way to get around this is to construct binomial variables of these measures. It’s not perfect, but it helps the investigator with the interpretation. We might be able to say, “Black mothers giving birth in “poor” states (those with more than 50% of the population living below the poverty line) were 10 percent less likely than whites to have an NTD affected birth in 2010.”
Additionally, the researcher who is interested in adding multi-level models to his or her research should consider comparing his or her findings to non-multi-level results on a regular basis, throughout the course of a project. The reason for this is that just because multi-level data are utilized does NOT mean that the models are improving the ability for these models to explain or predict differences or phenomena. It is possible that including multi-level data does not produce better fitting models. At the end of the analysis, a researcher incorporating these models must ask, “Does using multi-level models help me explain this phenomena any better than using OLS regression/logistic regression/multinomial regression, etc.?”
Another option to try different higher-level variables. The researcher should be prepared for the possibility that his or her treasured geographic variable might not add substantively to models of interest. He or she should consider other related variables. If the researcher was interested in percent of an area that is poor, he or she might be interested in looking at the reasons why that variable might matter. Then he or she could include other variables that capture those reasons (poor communities have a lower tax base, fewer community resources, etc.) Of course, there is the final recourse, which is to abandon the model, the data, and the theory. This is not failure, it is simply the recognition that investigating a specific problem isn’t aided with multi-level models, higher-level data, or theory including meso or macro level phenomena.
I am happy to help anyone interested in learning about these methods. My experience has been mostly in terms using Stata’s GLAMM command for my dissertation, and I am looking forward to learning more about them. Check out the two volume set by Sophia Rabe-Hesketh and Anders Skrondal “Multilevel and Longitudinal Modeling Using Stata” (3rd edition). Stata’s Youtube Channel has a couple of video introductions to Multilevel Linear Models in Stata.
[i] Not a valid assumption. The author acknowledges that the contribution of each state to the number of NTD affected births is proportional to its population of reproductive aged women. For the purposes of this exercise, the contribution of each state is thought to be equal to 1/50th of the total number of NTD affected births.