Validity and Reproducibility of a Semiquantitative Multiple-Choice Food Frequency Questionnaire in Iranian Adults

Diet plays an important role in the development and control of chronic diseases.^1-3 Valid and reliable dietary assessment methods developed for each population are needed to find out the dietary determinants of health.² Several methods such as 24-hour recall, dietary record (DR), and food frequency questionnaire (FFQ) have been used to assess dietary intake in different studies.⁴ All of these methods have some individual limitations in estimating the dietary food intake; therefore, they are selected by researchers to be used based on the aim and the design of each study.^5,6 The semi-quantitative food frequency questionnaire (SQ-FFQ) is a widely used method to assess long-term dietary intake in population-based studies⁷; because it is inexpensive and easy to complete in large populations.⁴ The questionnaires are mostly composed of a food list and a frequency response section. Some SQ-FFQs also ask about the usual portion size of each consumed food item.

Food frequency questionnaires are prone to bias and misclassification because they rely on memory, and miss some food items that are high in the targeted nutrients, frequently consumed in the population, and are differently used by individuals living in the target population.⁴ Therefore, FFQs should be validated using the other dietary assessment methods which are not prone to the same sources of bias.⁴ Biological markers are likely to be able to improve the estimations of dietary intake assessment because random errors are independent of other dietary assessment tools.^4,8 Nevertheless, the majority of biomarkers are expensive and it is not desirable to replace the other methods of dietary assessment as part of a large epidemiological study.^8,9 Furthermore, the reproducibility of the FFQs should be assessed because they are designed to provide data on long-term intakes.⁴ Any changes in the design of FFQs might lead to a change in the performance of the questionnaire.¹⁰ In addition, these questionnaires are culture-specific which means that the dietary culture and foods consumed are highly variable between populations even in the same country. Therefore, the validity and reproducibility of a questionnaire are needed to be measured for any population.^4,6

Several FFQs are developed in Iran.^11-14 For instance, Tehran Lipid and Glucose Study (TLGS)¹³ and Golestan cohort study¹² have used open-ended FFQs which were designed to be used in Tehran and Golestan provinces, respectively. Furthermore, the FFQ used in the TLGS asked about the frequency and portion size of the intake of 168 food items.¹³ The participants’ food consumption frequency over the previous year was asked on a daily (eg, bread), weekly (eg, rice, meat), or monthly (eg, fish) basis. Likewise, a 150-item FFQ was designed for the Golestan cohort study and the frequency of consumption was recorded as times per day, week, month, year, and never. For 51 food items, pictures of different portion sizes were used to increase the precisions of estimations.¹² Both FFQs were validated using 24-hour recalls as a reference method to validate the dietary intakes and showed good validity and reproducibility.^12,13

To the best of our knowledge, no study has tried to develop and validate a multiple-choice food-based FFQ in Iran using weighed dietary records (WDRs). People residing in Yazd province, central Iran, are living in arid and semi-arid areas that have different dietary habits and food items. Therefore, we designed a semi-quantitative food-based FFQ with 178 food items to assess the habitual food and nutrient intake of adults living in Yazd to be used in large-scale epidemiologic studies. The current FFQ used a multiple-choice approach; therefore it is easy to perform for participants and interviewers to fill the questionnaire.⁴ The present study aimed to assess the validity and reproducibility of the semi-quantitative multiple-choice food frequency questionnaire using WDRs biochemical markers in a sample of adults living in Yazd city, who were attended a clinical trial.

A total of 180 subjects (89%) aged 48.9 ± 8.4 years after excluding 22 participants who did not administer 3 FFQs were included in the current analysis (50.2% female and 49.8% male). The general characteristics of the study participants are provided in Table 1.

The mean daily nutrient intake based on nine 3-day WDRs and FFQs are shown in Table 2. Compared to WDRs, the mean daily intakes calculated from FFQs tended to overestimate the energy and nutrients intake except for the iron intake. The FFQs were not significantly different in estimating the dietary nutrients intake except for the dietary protein, fat, cholesterol, niacin, folate, beta-carotene, vitamin E, zinc, copper, selenium, and manganese. The sex-stratified mean daily nutrient intakes estimated by using the three FFQs and WDRs are also described in Supplementary Tables 1 and 2. The analysis based on sex revealed that Food frequency questionnaires provided higher estimations for the intakes of dietary nutrients compared to WDRs in both sexes except for iron.

Table 3 presents the reproducibility of 3 FFQs (FFQ1 vs FFQ2, FFQ1 vs FFQ3, and FFQ2 vs FFQ3) which is calculated by Pearson correlation and ICC. The median Pearson correlation value was 0.31, 0.44, and 0.38 for FFQ1 versus FFQ2, FFQ1 versus FFQ3, and FFQ2 versus FFQ3, respectively. Moreover, the ICC ranged from 0.43 to 0.73 and was mostly above 0.50 (median: 0.56). The highest and lowest ICC was shown to be for vitamin D and thiamin, respectively (P < .001). The sex-stratified analyses are shown in Supplementary Tables 3 and 4. For men, the median: Pearson correlation value was 0.31, 0.33, 0.40 for FFQ1 versus FFQ2, FFQ1 versus FFQ3, and FFQ2 versus FFQ3, respectively. Also, the ICC ranged from 0.36 for thiamin to 0.76 for vitamin D and were mostly above 0.50 (median: 0.59). For women, the median Pearson correlation value was 0.32 for FFQ1 versus FFQ2, FFQ1 versus FFQ3, and FFQ2 versus FFQ3. Furthermore, the ICC vary from 0.34 for pyridoxine to 0.72 for sucrose and vitamin D and were mostly above 0.50 (median: 0.58).

Correlation coefficients for the validity of FFQs compared to WDRs are displayed in Table 4. The Pearson correlation coefficients between FFQ1-WDR and FFQ3-WDR for nutrients varied from 0.05 for vitamin A (P = .498) to 0.35 for cholesterol (P < .001) and −0.03 for vitamin A (P = .757) to 0.45 for thiamin (P < .001; median: 0.23 and 0.35), respectively. The partial correlations were 0.01 for vitamin A (P = .917) to 0.35 for vitamin D (P < .001) and −0.04 for vitamin A (P = .646) to 0.40 for manganese (P < .001; median: 0.14 and 0.25) between FFQ1-WDR and FFQ3-WDR, respectively. The correlation coefficient values did not change remarkably when de-attenuation factors were considered 0.01 for vitamin A to 0.40 for vitamin B12 and −0.05 for vitamin A to 0.41 for manganese (median: 0.17 and 0.26) for FFQ1-WDR and FFQ3-WDR, respectively. Furthermore, the ICC for FFQ1-WDR association and FFQ3-WDR association ranged from 0.09 for vitamin A (P = .261) to 0.49 for cholesterol (P < .001) and −0.05 for vitamin A (P = .621) to 0.58 for selenium (P < .001) and manganese (P < .001; median: 0.30 and 0.46), respectively. The sex-stratified analyses are also provided in Supplementary Tables 5 and 6.

When considering if the methods agreed for individuals using Bland-Altman method, the differences in log-transformed nutrient intake between the FFQs and the 27-day food records were plotted against the log-transformed mean nutrient intakes of the two methods for energy, protein, carbohydrates, and fats (Bland-Altman plots; Figure 3). The points are scattered above zero in most plots, suggesting that the FFQ provides a higher intake compared with the food record. In addition, there was a trend of decreasing accuracy with increasing protein and fat intake, as the scatter plots show overdispersion at higher intakes.

The contingency tables in which the quartiles of the dietary intakes of nutrients assessed using WDRs and the third FFQ are simultaneously provided against each other, are described in Table 5. The analyses revealed that 21.3% to 42.1% of individuals were classified in the same quartiles for potassium and selenium, respectively (median: 32.3%); and 34.6% to 48.2% for α-tocopherol and fat intake were categorized in adjacent quartiles, respectively (median: 42%). Except for vitamin A (11.9%), β-carotene (13.4%), α-tocopherol (12.2%), and iron (10.4%) the extreme misclassification was lower than 10% for other nutrients (median: 6%).

The correlation coefficients obtained from the triads method, calculated using correlation coefficients between FFQ, WDRs, and the biochemical measurements are demonstrated in Table 6. The validity coefficients between different measurement methods and the calculated true intake were considered as weak for FFQ (ρ QI: 0.13) and biochemical measurement (ρ BI: 0.16) and high for WDRs (ρ RI: 0.9) for vitamin C. The validity coefficients for calcium were considered as high for the questionnaire (ρ QI: 0.62), and biochemical measurement (ρ RI: 0.62), and moderate for WDRs (ρ BI: 0.21). The validity coefficients for magnesium were evaluated to be high for the questionnaire (ρ QI: 0.89) and moderate for biochemical assessment (ρ RI: 0.31) and WDRs (ρ BI: 0.22). Also, for zinc, the validity coefficients were considered as moderate for FFQ (ρ QI: 0.56) and WDRs (ρ BI: 0.32), and high for biochemical assessment (ρ RI: 0.66). The validity coefficient for FFQ and the correlation between questionnaire and biomarker were considered as the upper and the lower limit of the validity coefficient between FFQ and the true intakes, respectively. Therefore, the lower and upper limits ranged from 0.02 to 0.13, 0.13 to 0.62, 0.20 to 0.89, and 0.21 to 0.56 for vitamin C, calcium, magnesium, and zinc, respectively.

In the present study, we examined the validity and reproducibility of a 178-item multiple-choice SQ-FFQ which was assessed in a long-term clinical trial. The present results demonstrated a reasonable relative validity concerning WDRs for energy and all nutrients, except for vitamin A and β-carotene (median 0.46). The agreement between these 2 methods was reasonably acceptable (median 76.2%) and the median correlation between FFQs was 0.56 for all nutrient intakes. The present study tried to include a reasonable number of participants. Furthermore, to reduce the random error due to within-individual variation, both energy-adjusted and de-attenuated correlation coefficients were calculated. In addition, 2 blood samples were collected with 3-months intervals to reduce the influence of measurement errors.

It is proposed that measuring the dietary intakes using multiple DRs that are not dependent on memory and has a great specificity in describing foods is a suitable choice to be used as a reference method in validation studies.^4,25 Biochemical markers are also used in epidemiological studies to measure the participants’ status regarding specific nutrients or dietary compounds.^26,27 Previous studies indicate high correlations between dietary intake and some biochemical markers.^28,29 It should be noted that disease and homeostatic regulations might affect biomarkers’ status; furthermore, biomarkers should be assessed several times to show the long-term dietary intakes. These problems might reduce the applicability of biochemical markers to be used as the sole indicator of dietary intakes.²⁹ It is suggested that validation studies would provide a better insight if they compare FFQs with both DRs and biomarkers.²¹ Therefore, we used 27-days WDRs which have the least correlated error,⁴ as a reference to compare the energy and nutrients intakes from the questionnaire and biochemical markers.

We observed a general overestimation of nutrient intake using FFQs in comparison with WDRs. It is probably due to the seasonal availability of food items like fruits and vegetables, the misconception of portion size, and a long list of food items. In line with our results, other validation studies also reported that the FFQs, as compared with food record or 24-hour recall, overestimate the nutrient and energy intake.^30-32 Likewise, Considering that types of bread and rice are stapled foods, the overestimation of carbohydrates intake was found in another validation study in Iran.¹³ It is proposed that compared with reference methods, FFQs estimate higher intakes for most of the nutrients particularly when FFQ exceeds 100 food items.^33,34 We also observed the mean daily intake of nutrients is higher in men compared with women (Supplementary Tables 1 and 2). Sex differences in reporting energy intake exist and women were more likely to underreport energy intake.³⁵ Furthermore, according to sex differences in the food portion size, the sex-specific typical portion weights are recommended to be used instead of standard portion size.^36,37

The range of reproducibility of our questionnaire was 0.43 for thiamin to 0.73 for vitamin D for adjusted data which is comparable to other validation studies.^{7,12,13,30,31} According to the reports of a comprehensive review, the time interval in the validation studies varied from 2 hours to 15 years.³⁸ We chose 3-month intervals between the FFQs and tried to administer them at the same time as blood sample collection to diminish the difficulties for participants. The participants were asked not to change their diet during the study period.

Although the mean daily nutrient intake estimates between FFQs were not significantly different for the majority of the nutrients and energy intakes, the third FFQ showed a better correlation with WDRs, perhaps because of the learning bias that can result from participants learned how to answer the questions in the same way as previous questionnaires or WDRs, or change in participant’s diet.³⁹ Moreover, FFQ3, administered at the end of the study could comprise all WDRs in the period of the study which might explain the better correlation. In addition, we observed the higher median ICC in men between nutrients assessed by FFQs (0.59 for men and 0.58 for females; Supplementary Table 2) or between FFQs and WDRs (0.27 for men and 0.24 for women; Supplementary Table 3), which is in line with other reports.^40,41 As men tend to be unconcerned about their daily diets, it might have been easier for men to complete the FFQ, which requires simplified dietary habits.⁴¹

We expected that the random error correction for within-individual variation increases the correlation values. However, similar to the finding from other studies, the de-attenuation correlations were not substantially different from noncorrected estimates.^13,30 A large number of DRs (27 days) or the low within-individual variation compared to between-individual variation might explain this similarity.³⁰

Energy adjustment appears to improve correlation coefficients and diminish the measurement errors in the FFQ instrument.⁴ However, along with the finding of other studies,^13,42,43 using energy adjustment in our study, the median correlation coefficient of nutrients tends to lower the correlation values. It seems that the low between-individual variation in nutrients’ intakes measured by WDRs has led to lower correlation coefficients after adjustment.⁴⁴

The FFQs are mainly used to rank individuals based on their dietary intake and this is important in obtaining correct risk estimates of diseases.^4,5,45 The present study demonstrated that about 33% of participants were classified in the same quartiles using FFQs and WDRs. Furthermore, above 70% of participants were classified to the same or adjacent quartiles which are in agreement with other validation studies. Furthermore, the present study demonstrated that the proportion of complete disagreement was in the range of 3% to 13.4% (median 6%). These results were in line with other studies that used quartiles to classify their participants and were conducted in Asian adults.^31,46

As biomarkers represent the quantitative measurements and not rely on subjects’ memory which is the main source of bias in dietary assessment methods,²¹ we also used the serum biomarkers in our validation study. Using the method of triads, the correlation between estimated nutrient intakes using third FFQ and WDRs and measured biomarkers was calculated. Although the validity coefficients of a nutrient are not common to compare between studies because of differences in sample size, duration of studies, the number of food items, food consumption which is culture-specific, and intrinsic variability of biomarkers (bioavailability and metabolism of nutrients),^4,47 the FFQ validity coefficients for all biomarkers except for vitamin C (0.13 for vitamin C, 0.62 for calcium, 0.89 for magnesium, and 0.56 for zinc) were considered as moderate and high which is similar to findings from Mc Naughten et al (0.50, 0.63, 0.45, 0.62),²² and Andersen et al (0.58, 0.51).⁴⁸ In addition, Mirmiran et al¹³ found that the range of validity coefficient (ρ QI) was 0.21 to 0.95 (TLGS) which is in line with our results (0.02-0.89).

The realistic correlation coefficients of validation studies tend to be in a range of 0.5 to 0.7.⁴ In the TLGS, the mean Pearson correlation coefficient and the mean intraclass correlation coefficient between twelve 24-hour dietary recalls and FFQ for men were 0.53 and 0.59, and for women were 0.39 and 0.60 in energy-adjusted values.¹³ In the Golestan cohort study, the correlations coefficient between twelve 24-hour dietary recalls and the mean of four FFQs ranged from 0.49 to 0.82 and the intraclass correlations were between four FFQs varied from 0.66 to 0.89.¹² The validity correlation coefficients were reported to be lower in the present study compared to the previous Iranian studies. We observed that the median Pearson correlation coefficient and median intraclass correlation coefficient between 27-day WDRs and SQ-FFQ were 0.35 and 0.46. It should be noted that the previous investigations had used 24-hour dietary recalls for examining the validity which both rely on memory and this might increase the correlation coefficients by error.⁴ This is while our study used 27-day WDRs for validity assessment in the Iranian population for the first time which is not the same in the sources of bias.⁴ The range of correlation coefficients in our study was similar to studies previously conducted in Asia^31,49-53 which compared FFQs with DRs. They found that the range of correlation coefficients of nutrient intakes between FFQs and WDRS was 0.06 to 0.81 and the median ranged between 0.3 and 0.5. It should be noted that serving sizes and foods consumed in Asian regions are different; furthermore, meals are served as family-style and the family members share their foods. Thus, it might lead to a low perception of portion size when reporting their dietary intake using FFQs.³¹

The present study has some limitations that should be considered. First, the same portion size was used for both sexes which may result in substantial errors in the estimation of nutrient intakes. Second, as no complete Iranian FCT exists, we used the USDA FCT to calculate the energy and nutrient intakes for the majority of foods. This point might not affect the correlation coefficients and the assessment of misclassifications, however, might lead to biased absolute intakes. Furthermore, the same FCT was used to calculate the dietary intakes reported using FFQs and WDRs. This might lead to higher correlation coefficients. Moreover, our study was conducted on patients with diabetes and their spouses; therefore, the generalization of our findings to the general population should be done with caution. As the present study was conducted in the context of a clinical trial aimed to examine the effect of different plant oils on cardio-metabolic outcomes, the reproducibility and validity of FFQs might be prone to bias for dietary fatty acids. Therefore, we removed the validity and reproducibility statistics for different dietary fatty acids.

In summary, this study found that the present 178 item SQ-FFQ has overall acceptable levels of validity for assessing the dietary nutrients intake except for Vitamin A and β-carotene. Furthermore, the level of reproducibility for all nutrients was acceptable. Thus, the SQ-FFQ used in this study seems to be a useful instrument to measure the dietary nutrients in epidemiological studies conducted in Yazd province, central Iran.

The authors wish to thank all participants who voluntarily participated in the study. The authors also appreciate Shahid Sadoughi University of Medical Sciences. Furthermore, we are grateful to the research council of Nutrition and Food Security Research Center for their scientific support. The authors express appreciation for the executive collaboration of the diabetes research center of Shahid Sadoughi University of Medical Sciences, Yazd.