How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches
I am indebted to my reviewers and the editor Rick Wilson, whose criticisms and suggestions improved the article considerably. Equal thanks to Jeff Gill, Thomas Gschwend, Adam Ramey, Tobias Heinrich, Thomas Pluemper, Vera Troeger, Tom Scotto, Ray Duch, and Michel Becher for their excellent comments and advice. Earlier versions of this article were presented at the 2011 annual meeting of the Midwest Political Science Association and the first conference of the European Political Science Association in Dublin, 2011. I thank participants and discussants for helpful comments and suggestions. Replication data and supplemental material are available at the author's website and the AJPS dataverse at http://hdl.handle.net/1902.1/18628.
Abstract
Researchers in comparative research increasingly use multilevel models to test effects of country-level factors on individual behavior and preferences. However, the asymptotic justification of widely employed estimation strategies presumes large samples and applications in comparative politics routinely involve only a small number of countries. Thus, researchers and reviewers often wonder if these models are applicable at all. In other words, how many countries do we need for multilevel modeling? I present results from a large-scale Monte Carlo experiment comparing the performance of multilevel models when few countries are available. I find that maximum likelihood estimates and confidence intervals can be severely biased, especially in models including cross-level interactions. In contrast, the Bayesian approach proves to be far more robust and yields considerably more conservative tests.
Supporting Information
Disclaimer: Supplementary materials have been peer-reviewed but not copyedited.
| Filename | Description |
|---|---|
| ajps12001-sup-0001-Supmate.pdf250.7 KB | Table S1: Multilevel model for EU support. Bayesian and ML estimates. Figure S1: Performance of point and interval estimates of individual level covariate effect Figure S2: Performance of point and interval estimates of country level covariate effect Figure S3: Performance of point and interval estimates of country level covariate effect Figure S4: Performance of point and interval estimates of individual level covariate random coefficient Figure S5: Performance of point and interval estimates of country*individual level interaction effect Figure S6: Effect of intraclass correlation on performance of point estimates of country level effects Figure S7: Effect of different variance component priors on performance of point and interval estimates of |
in hierarchical linear and probit models (type III/IV). Displayed are relative bias (in %) of estimates and 95% confidence interval non-coverage as a function of the number of countries used, and ML and Bayes estimation.
in hierarchical linear and probit models (type III/IV). Displayed are relative bias (in %) of estimates and 95% interval non-coverage as a function of the number of countries used, and ML and Bayes estimation.
in hierarchical linear and probit models (type V/VI). Displayed are relative bias (in %) of estimates and 95% interval non-coverage as a function of the number of countries used, and ML and Bayes estimation.
in hierarchical linear and probit models (type V/VI). Displayed are relative bias (in %) and 95% interval non-coverage as a function of the number of countries used, and ML and Bayes estimation.
in hierarchical linear and probit models (type V/VI). Displayed are relative bias (in %) and 95% interval non-coverage as a function of the number of countries used, and ML and Bayes estimation.
in a hierarchical probit model (type VI). Displayed is relative bias (in %) of the estimate as a function of three levels of intraclass correlations, the number of countries used, and ML and Bayes estimation.
in a hierarchical linear model (type III). Displayed is 95% interval noncoverage as a function of the number of countries used, and inverse gamma and Gelman standard deviation uniform priors.Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Achen, Christopher H. 2005. “Two-Step Hierarchical Estimation: Beyond Regression Analysis.” Political Analysis 13(4): 447–56.
- Afshartous, David. 1995. “Determination of Sample Size for Multilevel Model Design.” Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
- Agresti, Alan, James G. Booth, James P. Hobert, and Brian Caffo. 2000. “Random-Effects Modeling of Categorical Response Data.” Sociological Methodology 30(1): 27–80.
- Agresti, Alan, and Ranjini Natarajan. 2001. “Modeling Clustered Ordered Categorical Data: A Survey.” International Statistical Review 69(3): 345–71.
- Aitkin, Murray, and Nicholas Longford. 1986. “Statistical Modelling Issues in School Effectiveness Studies.” Journal of the Royal Statistical Society A 149(1): 1–43.
- Albert, Jim H., and Siddharta Chib. 1993. “Bayesian Analysis of Binary and Polychotomous Response Data.” Journal of the American Statistical Association 88(422): 669–79.
- Andersen, Robert, and Tina Fetner. 2008. “Economic Inequality and Intolerance: Attitudes toward Homosexuality in 35 Democracies.” American Journal of Political Science 52(4): 942–58.
- Anderson, Christopher J., and Matthew M. Singer. 2008. “The Sensitive Left and the Impervious Right: Multilevel Models and the Politics of Inequality, Ideology, and Legitimacy in Europe.” Comparative Political Studies 41(4/5): 564–99.
- Angrist, Joshua D., and Jorn-Steffen Pischke. 2008. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton, NJ: Princeton University Press.
10.2307/j.ctvcm4j72 Google Scholar
- Arzheimer, Kai. 2009. “Contextual Factors and the Extreme Right Vote in Western Europe, 1980–2002.” American Journal of Political Science 53(2): 259–75.
- Browne, William C., and David Draper. 2006. “A Comparison of Bayesian and Likelihood-Based Methods for Fitting Multilevel Models.” Bayesian Analysis 1(3): 473–514.
- Browne, William J., and David Draper. 2000. “Implementation and Performance Issues in the Bayesian and Likelihood Fitting of Multilevel Models.” Computational Statistics 15(3): 391–420.
- Cowles, Mary Kathryn, and Bradley P. Carlin. 1996. “Markov Chain Monte Carlo Convergence Diagnostics: A Comparative Review.” Journal of the American Statistical Association 91(434): 883–904.
- Denisova, Irina, Markus Eller, Timothy Frye, and Ekaterina Zhuravskaya. 2009. “Who Wants to Revise Privatization? The Complementarity of Market Skills and Institutions.” American Political Science Review 103(2): 284–304.
- Draper, David. 2008. “ Bayesian Multilevel Analysis and MCMC.” In Handbook of Multilevel Analysis, ed. Jan de Leeuw and Erik Meijer. New York: Springer, 77–139.
10.1007/978-0-387-73186-5_2 Google Scholar
- Eichenberg, Richard, and Russell Dalton. 1993. “Europeans and the European Community: The Dynamics of Public Support for European Integration.” International Organization 47(4): 507–34.
- Eurostat. 2004. External and Intra-European Union Trade—Statistical Yearbook. Luxembourg: Office for Official Publications of the European Communities.
- Fahrmeir, Ludwig, and Gerhard Tutz. 1997. Multivariate Statistical Modelling Based on Generalized Linear Models. 2nd ed. New York: Springer.
- Franchino, Fabio, and Bjorn Hoyland. 2009. “Legislative Involvement in Parliamentary Systems: Opportunities, Conflict, and Institutional Constraints.” American Political Science Review 103(4): 607–21.
- Franzese, Robert J., Jr. 2005. “Empirical Strategies for Various Manifestations of Multilevel Data.” Political Analysis 13(4): 430–46.
- Gelfand, Alan E., and Adrian F. M. Smith. 1990. “Sampling-Based Approaches to Calculating Marginal Densities.” Journal of the American Statistical Association 85(410): 398–409.
- Gelman, Andrew. 2006. “Prior Distributions for Variance Parameters in Hierarchical Models.” Bayesian Analysis 1(3): 515–34.
- Gelman, Andrew, and Donald Rubin. 1992. “Inference from Iterative Simulation Using Multiple Sequences.” Statistical Science 7(4): 457–511.
10.1214/ss/1177011136 Google Scholar
- Gelman, Andrew, and Jennifer Hill. 2007. Data Analysis Using Regression and Multilevel /Hierarchical Models. Cambridge, MA: Cambridge University Press.
- Gelman, Andrew, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis. Boca Raton, FL: Chapman & Hall.
- Gilardi, Fabrizio. 2010. “Who Learns from What in Policy Diffusion Processes?” American Journal of Political Science 54(3): 650–66.
- Gill, Jeff. 1999. “The Insignificance of Null Hypothesis Significance Testing.” Political Research Quarterly 52(3): 647–74.
- Gill, Jeff. 2008. Bayesian Methods: A Social and Behavioral Sciences Approach. Boca Raton, FL: Chapman & Hall.
- Goldstein, Harvey. 2010. Multilevel Statistical Models. Chicester, UK: Wiley.
10.1002/9780470973394 Google Scholar
- Goldstein, Harvey, Jon Rasbash, Min Yang, Geoffrey Woodhouse, Huiqi Pan, Desmond Nuttall, and Sally Thomas. 1993. “A Multilevel Analysis of School Examination Results.” Oxford Review of Education 19(4): 425–33.
- Hooghe, Liesbet, and Gary Marks. 2004. “Does Identity or Economic Rationality Drive Public Opinion on European Integration?” PS: Political Science & Politics 37(3): 415–20.
- Hooghe, Marc, Tim Reeskens, Dietlind Stolle, and Ann Trappers. 2009. “Ethnic Diversity and Generalized Trust in Europe. A Cross-National Multilevel Study.” Comparative Political Studies 42(2): 198–223.
- Iversen, Torben, and Frances Rosenbluth. 2006. “Bayesian Inference for Comparative Research.” American Political Science Review 88(2): 412–23.
- Jackman, Simon D. 2009. Bayesian Analysis for the Social Sciences. New York: Wiley.
10.1002/9780470686621 Google Scholar
- Jackman, Simon, and Bruce Western. 1994. “The Political Economy of Gender: Explaining Cross-National Variation in the Gender Division of Labor and the Gender Voting Gap.” American Journal of Political Science 50(1): 1–19.
- Jaynes, E. T. 1976. “ Confidence Intervals vs. Bayesian Intervals.” In Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science. Dordrecht, the Netherlands: D. Reidel, 175–257.
10.1007/978-94-010-1436-6_6 Google Scholar
- Jaynes, E. T. 2003. Probability Theory: The Logic of Science. Cambridge: Cambridge University Press.
10.1017/CBO9780511790423 Google Scholar
- Jiang, Jiming. 2007. Linear and Generalized Linear Models: Mixed Models and Their Applications. New York: Springer.
- Jusko, Karen Long, and W. Philips Shively. 2005. “Applying a Two-Step Strategy to the Analysis of Cross-National Public Opinion Data.” Political Analysis 13(4): 327–44.
- Kedar, Orit, and W. Philips Shively. 2005. “Introduction to the Special Issue.” Political Analysis 13(4): 1–4.
- King, Gary. 1998. Unifying Political Methodology: The Likelihood Theory of Statistical Inference. Ann Arbor: The University of Michigan Press.
10.3998/mpub.23784 Google Scholar
- Kreft, Ita. 1996. “Are Multilevel Techniques Necessary? An Overview Including Simulation Studies.” California State University, Los Angeles. Unpublished manuscript.
- Kreft, Ita G. G., and Jan de Leeuw. 1998. Introducing Multilevel Modeling. Thousand Oaks, CA: Sage.
10.4135/9781849209366 Google Scholar
- Lax, Jeffrey R., and Justin H. Phillips. 2009. “Gay Rights in the States: Public Opinion and Policy Responsiveness.” American Political Science Review 103(3): 367–86.
- Maas, Cora J. M., and Joop J. Hox. 2004a. “The Influence of Violations of Assumptions on Multilevel Parameter Estimates and Their Standard Errors.” Computational Statistics and Data Analysis 46(3): 427–40.
- Maas, Cora J. M., and Joop J. Hox. 2004b. “Robustness Issues in Multilevel Regression Analysis.” Statistica Neerlandica 58(2): 127–37.
- Maas, Cora J. M., and Joop J. Hox. 2005. “Sufficient Sample Sizes for Multilevel Modeling.” Methodology 1(3): 85–91.
10.1027/1614-2241.1.3.85 Google Scholar
- Mahler, Vincent A., Bruce J. Taylor, and Jennifer R. Wozniak. 2000. “Economics and Public Support for the European Union: An Analysis at the National, Regional, and Individual Levels.” Polity 32(3): 429–53.
- McCulloch, Charles E., and Shayle R. Searle. 2001. Generalized, Linear, and Mixed Models. New York: Wiley.
- McLachlan, Geoffrey J., and Thriyambakam Krishnan. 2008. The EM Algorithm and Extensions. New York: Wiley.
10.1002/9780470191613 Google Scholar
- Moineddin, Rahim, Flora I. Matheson, and Richard H. Glazier. 2007. “A Simulation Study of Sample Size for Multilevel Logistic Regression Models.” BMC Medical Research Methodology 7: 1–34.
- Natarajan, Ranjini, and Robert E. Kass. 2000. “Reference Bayesian Methods for Generalized Linear Mixed Models.” Journal of the American Statistical Association 95(449): 227–37.
- Normand, Sharon-Lise T., and Kelly H. Zou. 2002. “Sample Size Considerations in Observational Health Care Quality Studies.” Statistics in Medicine 21(3): 331–45.
- O'Hagan, Anthony, and Jonathan Forster. 2009. Kendall's Advanced Theory of Statistics: Bayesian Inference. 2nd ed. Chichester, UK: Wiley.
- O'Rourke, Kevin H., and Richard Sinnott. 2006. “The Determinants of Individual Attitudes Towards Immigration.” European Journal of Political Economy 22(4): 838–61.
10.1016/j.ejpoleco.2005.10.005 Google Scholar
- Pinheiro, José C., and Douglas M. Bates. 1995. “Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model.” Journal of Computational and Graphical Statistics 4(1): 12–35.
10.2307/1390625 Google Scholar
- Rabe-Hesketh, Sophia, and Anders Skrondal. 2008. Multilevel and Longitudinal Modeling Using Stata. College Station, TX: Stata Press.
- Rabe-Hesketh, Sophia, Anders Skrondal, and Andrew Pickles. 2002. “Reliable Estimation of Generalized Linear Mixed Models Using Adaptive Quadrature.” The Stata Journal 2(1): 1–21.
- Robert, Christan P. 2007. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. New York: Springer.
- Rodríguez, Germán, and Noreen Goldman. 1995. “An Assessment of Estimation Procedures for Multilevel Models with Binary Responses.” Journal of the Royal Statistical Society A 158(1): 73–89.
- Skrondal, Anders, and Sophia Rabe-Hesketh. 2004. Generalized Latent Variable Modeling: Multilevel, Longitudinal and Structural Equation Models. Boca Raton, FL: Chapman & Hall.
10.1201/9780203489437 Google Scholar
- Snijders, Tom, and Roel Bosker. 1999. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Thousand Oaks, CA: Sage.
- Snijders, Tom, and Roel Bosker. 2012. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. 2nd ed. Thousand Oaks, CA: Sage.
- Spiegelhalter, D. J., A. Thomas, N. Best, and W. R. Gilks. 1997. BUGS: Bayesian Inference Using Gibbs Sampling: Manual. Cambridge, MA: Medical Research Council Biostatistics Unit.
- Steenbergen, Marco R., and Bradford S. Jones. 2002. “Modeling Multilevel Data Structures.” American Journal of Political Science 46(1): 218–37.
- van der Meer, Tom W. G., Jan W. van Deth, and Peer L. H. Scheepers. 2009. “The Politicized Participant: Ideology and Political Action in 20 Democracies.” Comparative Political Studies 42(11): 1426–57.
- Verbeek, Marno. 2004. A Guide to Modern Econometrics. Chichester, UK: Wiley.
- Voeten, Erik. 2008. “The Impartiality of International Judges: Evidence from the European Court of Human Rights.” American Political Science Review 102(4): 417–33.
- Weldon, Steven A. 2006. “The Institutional Context of Tolerance for Ethnic Minorities: A Comparative, Multilevel Analysis of Western Europe.” American Journal of Political Science 50(2): 331–49.
