Abstract
Objectives. We conducted a group randomized trial of 2 South African school-based smoking prevention programs and examined possible sources and implications of why our actual intraclass correlation coefficients (ICCs) were significantly higher than the ICC of 0.02 used to compute initial sample size requirements.
Methods. Thirty-six South African high schools were randomly assigned to 1 of 3 experimental groups. On 3 occasions, students completed questionnaires on tobacco and drug use attitudes and behaviors. We used mixed-effects models to partition individual and school-level variance components, with and without covariate adjustment.
Results. For 30-day smoking, unadjusted ICCs ranged from 0.12 to 0.17 across the 3 time points. For lifetime smoking, ICCs ranged from 0.18 to 0.22; for other drug use variables, 0.02 to 0.10; and for psychosocial variables, 0.09 to 0.23. Covariate adjustment substantially reduced most ICCs.
Conclusions. The unadjusted ICCs we observed for smoking behaviors were considerably higher than those previously reported. This effectively reduced our sample size by a factor of 17. Future studies that anticipate significant cluster-level racial homogeneity may consider using higher-value ICCs in sample-size calculations to ensure adequate statistical power.
It is common when one is conducting public health interventions to randomize and then intervene with intact social groups, such as schools, churches, or worksites, rather than individuals. Appropriate analysis of such group- or cluster-randomized trials must account for the statistical similarity of participants within these larger units.1,2 Individuals within clusters may enter the study with greater similarity than individuals randomly selected from the general population and they may also respond to an intervention in a dependent manner. Failure to account for either source of similarity violates the basic premise of participant nonindependence assumed in traditional statistical approaches and can result in inflated type I error.1,2 Such within-group similarity is typically captured by the intraclass correlation coefficient (ICC).
The following metaphor may help conceptualize how an ICC can affect sample size. A zero ICC occurs when 2 completely unrelated individuals are randomly selected from the general population. In this case, each person contributes a full independent observation. If, however, siblings from the same household were selected, because of their shared environment and genetics, depending on the degree of similarity, this would result in slightly less than 2 independent observations. If nonidentical twins were selected, they would contribute an even lower degree of independent observation; identical twins, even less. In the extreme case, if Siamese twins were recruited, they would begin to approach the contribution of a single individual. In most cluster-randomized studies, the degree of nonindependence within a group is low, but its impact on overall statistical power can still be great because of the variance inflation formulae used to adjust for the ICC.
If the ICC is known before conducting a trial, sample size requirements can be adjusted upward to account for the reduction in true sample size and statistical power, decreasing the chances of type II error. As noted by Murray et al.,3 however, it is often difficult to obtain a reliable ICC estimate that is entirely applicable to one's proposed study. Archival ICC estimates are often derived from samples, measures, contexts, or study designs that differ from the one proposed, limiting their utility. Whereas overestimation of the ICC can lead to an overpowered study and, therefore, inefficient use of resources, underestimation of the ICC can lead to an underpowered study (i.e., type II error).
We recently conducted a group randomized trial of 2 school-based smoking prevention programs designed for South African youths from 2004 to 2008. In computing our a priori sample-size requirements, we used ICC estimates drawn from published school-based smoking prevention studies, all of which were conducted in the United States. Prior ICCs were predominantly in the range of 0.02 to 0.04,4 and our sample size was computed accordingly. However, when the study was completed and we examined the actual ICC it turned out to be significantly higher than we had anticipated. As a result, many of our outcomes that would have been statistically significant with the ICC that was originally projected turned out to be null. We felt it would be useful to publish and explain these ICCs to assist other investigators.
All public high schools in 2 of South Africa's 9 provinces, KwaZulu-Natal and the Western Cape, were enumerated. Because of the substantially higher smoking rates among “coloured” youths,5 schools with predominantly “coloured” students were oversampled. “Coloured” in South Africa denotes mixed race of Black and at least 1 other ethnic/racial ancestry. These 2 provinces were selected because the Western Cape has a much higher proportion of “coloured” residents, whereas KwaZulu-Natal has a predominantly Black African population, the largest ethnic group in South Africa.
We used data supplied by South Africa's National Department of Education to categorize schools in these provinces by ethnic composition, size, and socioeconomic status (SES). Schools were considered large if there were more than 200 8th grade students enrolled and otherwise they were classified as small.
Schools were then randomly selected within each ethnicity, size, and SES stratum. The target sample was 36 schools, or 12 per experimental group. A total of 39 schools were approached, of which 3 refused. The 36 public schools recruited from the 2 provinces (18 per province) were then randomly assigned to 1 of 3 experimental groups. Group 1 (comparison) schools (n = 12) received the usual tobacco and substance use education, which involves little specific smoking prevention programming. Group 2 schools (n = 12) received the South African version of the “Keep Left” harm minimization curriculum beginning in 8th grade and continuing through 9th grade. Group 3 schools (n = 12) received the South African version of the “Life Skills Training” curriculum beginning in 8th grade and continuing through 9th grade.
The 2 curricula, each of which comprised 8 units per year for 2 years, were designed to be taught by life orientation teachers. Life orientation is a separate mandatory topic in South African schools similar to health education in the United States that includes student outcomes for health behaviors and social skills development. Additional details about the study and the interventions used can be found elsewhere.6
The participating schools had 175 eligible 8th grade classes. Classrooms were sampled randomly from the pool of 175 until approximately 150 students per school were assessed. To reach the 150 student quota per school, 123 (70%) classrooms were required. The 123 classes represented 59 unique life orientation teachers.
The primary outcome for the intervention was past month use of cigarettes, defined as any use in the past 30 days. This, as well as secondary outcomes of lifetime cigarette use, frequent cigarette use (more than 20 days per month), past-month marijuana use, past-month binge drinking (defined as consumption of 5 or more drinks within a few hours), and an index of past-month illicit drug use were assessed with a self-report questionnaire adapted from prior studies conducted in South Africa and elsewhere.5,7,8 Each of these indicators was recoded into a binary variable with 0 = nonuse and 1 = use. Because of the low rates of hard or illicit drug use, we created an aggregate index indicating past month use of either cocaine or crack, Mandrax (methaqualone), or “tik” (methamphetamine). Use of any 1 of these substances was coded as “1” for the illicit drug use index.
We assessed perceived harm of ever and regular use of tobacco, marijuana, and alcohol by using a 3-point scale with the following response categories and coding: 1 = no harm, 2 = slight harm, and 3 = great harm. The α in our sample for the ever-use items was 0.73, and α for the regular-use items was 0.88. Perceived refusal skills for 5 substances (cigarettes, alcohol, marijuana, cocaine, and inhalable drugs) was assessed by querying “Would you be able to say no if someone tried to get you to use [insert substance name]?” Responses ranged from 1 = definitely would to 5 = definitely would not (α = 0.97). We assessed attitude toward smoking with a 10-item measure adapted from a previous South African survey that tapped positive expectancies of smoking. Sample items include: smoking helps you cope with stress, smoking helps you enjoy a party, and smoking helps people feel more relaxed. Items were answered on a 4-point scale ranging from 1 = strongly agree to 4 = strongly disagree (α = 0.88).
Scannable questionnaires were administered in the classroom by trained research assistants. Student names were not included in the questionnaire. Each student was assigned a confidential identifying number, which was prewritten on his or her questionnaire. Teachers were asked to vacate the room during the questionnaire administration.
Students completed questionnaires on 3 occasions: (1) baseline in the beginning of 8th grade, (2) posttest 1 at the end of 8th grade, and (3) posttest 2 at the end of 9th grade. For the 2 posttest assessments, only individuals who were in the school at the beginning of 8th grade and who had completed the baseline evaluation were asked to complete questionnaires.
For a mixed-effect analysis of variance in group randomized trials, the total variance for the dependent variable y is
 |
reflects the a component of variance attributable to the groups (which are nested in condition), and 
is the variance component attributable to random variation of the members within those groups.
The ICC is calculated as
 |
 |
is the variance component attributable to the groups adjusting for other covariates, and 
is the variance attributable to individuals adjusting for covariates. Covariates in our model included treatment condition, age, race, gender, urbanicity, pocket money, and family income.
We estimated variance components by using SAS version 9.1 PROC MIXED or the SAS GLIMMIX macro (SAS Institute Inc, Cary, NC), depending on the variable distribution. For continuous variables with an assumed Gaussian distribution (e.g., attitude and efficacy measures) MIXED, which implements the general linear mixed model, was used. For binary variables (i.e., all substance use variables), we used GLIMMIX specifying a logit link and binomial error. The GLIMMIX macro iteratively calls the MIXED procedure until it converges on a solution for the fixed effects and random effects. MIXED and GLIMMIX provide estimates of the variance attributable to school and to residual error; estimates from GLIMMIX were transformed back to their original scale via the inverse link function.1(p239,240) Although other approaches to computing ICCs for binary variables have been proposed,9 by using the above method our results will be more comparable with prior studies. The ICC estimates are provided both with (model 2) and without (model 1) covariate adjustment. We compared the improvement of model 2 versus model 1 as a percentage change in
 |
A total of 5266 8th grade students completed the baseline survey. According to the school rosters provided, there were 5685 eligible 8th grade students, which equates to a response rate of 93%. Of these, 4684 (89%) completed at least 1 of the 2 posttest assessments. The sample was 51% male, with a mean age of 14.1 years. Approximately 58% of the sample was Black, 28% were “coloured,” and 14% were White or other. Thirty-day smoking rates at baseline, 1-year follow-up, and 2-year follow-up were 18%, 20%, and 21%, respectively. Rates by race at baseline were 10%, 35%, and 12% for Black, “coloured,” and White or other youths, respectively.
As shown in Table 1, for 30-day smoking, the primary outcome for the study, unadjusted ICCs ranged from 0.12 to 0.17 across the 3 time points. For lifetime smoking, ICCs were in the range of 0.18 to 0.22, whereas for heavy smoking the ICC was consistently around 0.09. After adjustment for covariates, the ICCs for the 3 smoking variables dropped on average 60% (range = −43% to −84%).
TABLE 1— Intraclass Correlation Coefficients (ICCs) of Behavioral and Cognitive Measures of Adolescent Smoking and Drug Use: KwaZulu-Natal and Western Cape, South Africa 2004–2008.
TABLE 1— Intraclass Correlation Coefficients (ICCs) of Behavioral and Cognitive Measures of Adolescent Smoking and Drug Use: KwaZulu-Natal and Western Cape, South Africa 2004–2008.
| Variables | Model 1, Raw ICCa | Model 2, Adjusted ICCb | Model 2 Versus Model 1, % ICC Change |
| Smoking | |||
| Lifetime smoking | |||
| Baseline | 0.198 | 0.094 | −52.5 |
| Posttest 1 | 0.179 | 0.065 | −63.8 |
| Posttest 2 | 0.217 | 0.064 | −70.4 |
| Past 30-day smoking | |||
| Baseline | 0.123 | 0.070 | −43.2 |
| Posttest 1 | 0.118 | 0.051 | −56.9 |
| Posttest 2 | 0.168 | 0.028 | −83.5 |
| Frequent smokingc | |||
| Baseline | 0.089 | 0.046 | −48.8 |
| Posttest 1 | 0.087 | 0.035 | −60.0 |
| Posttest 2 | 0.095 | 0.036 | −62.4 |
| Other drug use | |||
| 30-day marijuana use | |||
| Baseline | 0.049 | 0.053 | 9.5 |
| Posttest 1 | 0.045 | 0.035 | −22.7 |
| Posttest 2 | 0.095 | 0.045 | −52.5 |
| 30-day binge drinking | |||
| Baseline | 0.024 | 0.018 | −25.3 |
| Posttest 1 | 0.022 | 0.012 | −45.0 |
| Posttest 2 | 0.035 | 0.022 | −36.5 |
| 30-day illicit drug use | |||
| Baseline | 0.041 | 0.045 | 11.0 |
| Posttest 1 | 0.029 | 0.027 | −6.9 |
| Posttest 2 | 0.047 | 0.040 | −15.3 |
| Cognitive variables | |||
| Drug refusal efficacy | |||
| Baseline | 0.128 | 0.111 | −13.0 |
| Posttest 1 | 0.113 | 0.098 | −13.3 |
| Posttest 2 | 0.178 | 0.172 | −2.9 |
| Perceived harm of ever used | |||
| Baseline | 0.181 | 0.167 | −7.8 |
| Posttest 1 | 0.171 | 0.088 | −48.3 |
| Posttest 2 | 0.201 | 0.093 | −53.7 |
| Perceived harm of regular used | |||
| Baseline | 0.230 | 0.202 | −12.1 |
| Posttest 1 | 0.090 | 0.078 | −14.1 |
| Posttest 2 | 0.121 | 0.091 | −24.9 |
| Smoking beliefs | |||
| Baseline | 0.060 | 0.056 | −6.5 |
| Posttest 1 | 0.058 | 0.040 | −29.7 |
| Posttest 2 | 0.075 | 0.050 | −33.6 |
Notes. Baseline was the beginning of 8th grade; posttest 1 was at the end of 8th grade; and posttest 2 was at the end of 9th grade.
aModel 1 includes school only.
bModel 2 is adjusted for treatment condition, age, gender, race, urbanicity, pocket money, and family income.
cAt least 20 days in the past month.
dUse of tobacco, marijuana, and alcohol, each assessed separately then combined into aggregate scale.
For 30-day marijuana use, ICC values were in the range of 0.05 to 0.095. For binge drinking, ICC values were between 0.02 and 0.04, and for past month illicit drug use, values ranged from 0.03 to 0.05. After adjustment for covariates, the average ICC for other drug use variables declined on average 20% (range = 11 to −53%).
For perceived harm of ever and regular substance use, ICC values ranged from 0.17 to 0.20 and 0.09 to 0.23, respectively, across the 3 time points. For drug resistance efficacy, values ranged from 0.11 to 0.l8. For the substance use belief scale, values ranged from 0.06 to 0.07. After adjustment for covariates, the average ICC of the cognitive variables declined by 22%.
The unadjusted ICCs observed in this project for smoking behaviors were considerably higher than those previously reported in US school-based studies. Typically, school-level ICCs for smoking, alcohol, diet, and physical activity behaviors have consistently ranged from 0.01 to 0.04,3,10–16 although in 1 study, for a subsample of 11th and 12th graders, an ICC of around 0.10 was reported for weekly smoking.10 The covariate-adjusted ICCs for smoking behaviors in our study, though substantially reduced, were also considerably higher than those typically reported in US school-based studies.
Both the adjusted and unadjusted ICCs observed in this project for drug use behaviors other than smoking were slightly higher but generally close to the values reported previously in US school-based studies.14 The ICCs for smoking variables have been reported to be higher than those of other behaviors.10
The unadjusted and adjusted ICCs for cognitive variables were somewhat higher than those previously reported.11,14 The ICCs for cognitive variables within our study were much higher than the ICCs observed for our behavioral variables. This pattern is consistent with the existing literature because ICCs tend to be highest for knowledge, attitudes, and beliefs; lower for behaviors; and lowest for physiological measures.4,11
To determine whether the ICCs we observed might have been anomalous to our sample, we computed ICCs from another South African school-based survey, the 2002 South African National Youth Risk Behavior Survey (YRBS),8,17 a nationally representative study of South African youths in grades 8 through 11. Self-administered questionnaires covering a broad range of risk behaviors were obtained from 10 699 students in 188 schools. Additional details of the study can be found elsewhere.8,17 From these data, we computed school-level ICCs for several behaviors that were also assessed in our study. Similar survey items were used across the 2 studies. For 30-day smoking, the ICC in the YRBS was 0.08; for 30-day alcohol, 0.09; for lifetime cocaine or heroin use, 0.09; and for 30-day marijuana, 0.03. With the exception of marijuana, the ICCs were again higher than prior studies, suggesting that our findings may be robust with regard to South African high-school substance use behaviors. However, the YRBS sampling framework entailed approximately 13% of eligible classrooms as compared with 70% in the current study. Sampling fewer classrooms per school tends to yield higher ICCs, so the higher ICCs in YRBS are perhaps less unexpected than those observed for our current study.
The cause of the higher ICCs in our study merits explication. A likely contributing factor is the relative ethnic homogeneity of South African schools; 24 of our schools comprised at least 70% of students from 1 racial group, and race in South Africa is strongly associated with substance use.5,7,18,19 This is in part because of unique cultural attitudes among Black Africans about smoking and drug use, and socioeconomic disparities that exist in South African society. Race was included as a covariate in the multivariable model, which led to a significant reduction in the ICC. Prior ICC estimates were derived from US studies. Although most US studies comprised predominantly White populations, in 2 US studies African Americans tended to have equal or even lower ICCs than Whites.11,13
Because our ICCs were generally as high at baseline as they were at 1-year and 2-year follow up, it appears the school-level homogeneity driving the ICCs was present at randomization rather than an artifact of postrandomization effects such as variable teacher implementation, differential drop out, or common intervention response. Plus, intervention group was included as a covariate. Thus, it does not appear that the intervention generated the ICCs.
The impact of these ICCs in our study was severe. When we assumed approximately 100 students per school and used the standard design effect formula,20 our ICC effectively reduced our sample size by a factor of 17 for 30-day smoking outcomes and even moreso for psychosocial outcomes. In other words, for significance testing the ICC-adjusted sample size was essentially one seventeenth of or smaller than the sample size actually measured. The study could then be considered a case of possible type II error. We had assumed in our sample size calculations an ICC of around 0.02.
Alternatively, the study was powered to detect a between-group difference in 30-day smoking of 6% to 7%, which we did not attain. Nonetheless, if clustering is ignored and the data are analyzed on the individual level, some of our behavioral effects and almost all psychosocial effects would have been significantly different for either or both of the intervention groups versus the control group.
Our findings suggest that when one is projecting sample size needs for cluster randomized studies, either observational or intervention, empirical ICCs derived from the actual target population may be required. Extrapolation of ICCs from other studies, particularly in which the social context may differ, can lead to substantially inaccurate sample size estimates. Future studies that anticipate significant cluster-level racial homogeneity for outcomes that are race-dependent may consider using higher value ICCs in sample size calculations to ensure adequate statistical power. This may also be the case for other sociodemographic variables. And as recommended by Murray et al.12,13,14,21 and substantiated by our findings, inclusion of covariates that might drive the ICC in outcome analyses can minimize the ICC's impact on power.
Acknowledgments
Sole funding for the project was provided by the National Institutes of Health Fogarty International Center to K. Resnicow (grant TW005977).
Human Participant Protection
Active written consent was obtained from parents. This study was approved by the institutional review board or ethics committee of the University of Michigan and the Medical Research Council of South Africa
