Feasibility of an online consensus approach for the diagnosis of cognitive impairment and dementia in rural South Africa

2.1 Study cohort

HAALSI Dementia is an ongoing prospective cohort study on a stratified subsample of 635 HAALSI participants aged 50 and older in Agincourt, a rural South African community. The study design and methods have been previously described.¹² In brief, HAALSI Dementia collects detailed neuropsychological and functional assessments, informant interviews, and neurological and clinical evaluations to enable cross-cultural comparison and cross-calibration with international HCAP and HRS studies. Furthermore, HAALSI Dementia is strategically oversampled for respondents categorized as higher risk for dementia to allow future estimation of dementia prevalence in the parent HAALSI cohort.

The first wave of data collection was completed in January 2020. Data were reviewed by an expert consensus panel to assign diagnosis of cognitively normal, mild cognitive impairment (MCI), or dementia, based on the NIA-AA criteria.⁶

This study was approved by the University of the Witwatersrand Human Research Ethics Committee (ref.M190443), Harvard T.H. Chan School of Public Health Institutional Research Ethics Board (ref. 18-1459,19-1396), and Mpumalanga Provincial Research and Ethics Committee.

2.2 Online consensus interface: design

Our online consensus website was modeled after LASI-DAD¹¹ and MYHAT³ web portals. Both studies showed the online consensus system to be an efficient, feasible, and valid approach for the clinical diagnosis of dementia. Following the same format and structure as LASI-DAD, our web interface entailed two main pages: assessments and ratings.

Raters independently reviewed all the data on the assessments page, including demographic, cognitive, neurological, and informant measures (Figure S1 in supporting information). Next, raters navigated to the ratings page and worked through the NIA-AA criteria to submit their final diagnosis (Figure S2 in supporting information).

Research participants were anonymous and identified only by a unique case ID. Raters were provided with unique rater IDs to access the web portal securely. The web portal randomly selected and assigned cases for each rater to review. Cases were reviewed one at a time, and any selections made by raters on the web page were autosaved. All recorded data from the site were autogenerated in a database and securely exported to STATA for data analysis and management.

2.2.1 Assessments: cognitive battery, informant interview, and clinical examination

All assessments were administered and captured via tablets using Computer-Assisted Personal Interviewing (CAPI) software. The respondents’ demographic information (age, sex, education) and limited health history were made available on the website. Any narrative text or remarks recorded by the nurses or field workers during visits were translated and displayed on the page. Data on administered cognitive measures, informant scales, and neurological findings were cleaned and managed in STATA and imported securely to the online assessments page (Figure 1). Full details on individual cognitive, informant, and neurological measures have been published.¹²

HAALSI Dementia cognitive measures and scores were shown under the respondent's assessments column and were categorized by their principal domain as follows: (1) general cognitive status: Mini-Mental State Examination (MMSE),¹⁴ days of the week,¹⁵ Telephone Interview for Cognitive Status (TICS),¹⁶ Community Screening Inventory for Dementia (CSID),¹⁷ and Clinical Dementia Rating (CDR) semi-structured interview;¹⁸ (2) episodic memory: immediate and delayed word recall,¹⁹ word list recognition,¹⁹ logical memory (story recall) immediate,²⁰ delayed and recognition;²⁰ (3) executive function and attention: motor sequences,²¹ go no/go,²² similarities and differences,²³ and Raven's standard Progressive Matrices;²⁴ (4) visuospatial/spatial memory: constructional praxis,¹⁹ constructional praxis recall,¹⁹ spatial working memory,²⁵ symbol cancellation²⁶ and clock draw;²⁷ (5) language: Boston Naming Test,²⁸ semantic fluency,²⁹ Token Test,³⁰ and phoneme (letter) fluency.³¹ The respondents’ assessments column also included a “mood” subheading, which outlined the respondent's self-reported measure of depression based on the Center for Epidemiologic Studies Depression (CES-D) scale.³²

In the absence of normative population data for our cohort and in an effort to aid the raters with their diagnostic decisions, the following items were also shown for each cognitive measure: (1) weighted means to provide a crude estimate of the participants’ performance relative to the average performance of individuals aged 50 and over across the parent HAALSI cohort, (2) percent of the sample missing the cognitive test, and (3) percent of the sample who performed at minimum and maximum score range. To assess cognitive decline over time, raters were also shown cognitive scores from the word recall task administered during parent HAALSI waves 1 (2014–2015) and 2 (2018–2019). For cognitive tasks that required participants to draw or write on the tablets (constructional praxis, clock draw, and MMSE), raters were able to see and evaluate the actual drawings completed by participants (Figure S3 in supporting information).

As part of the cognitive assessment, respondents also completed a reading assessment to assess basic literacy (letter and number recognition) through secondary level reading skills. Literacy scores were displayed alongside the self-reported education level to assist raters with the interpretation of cognitive test results.

The informant section summarized key demographic and background information (relationship to respondent, years known, frequency informant sees respondent) and presented all informant scales including Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE),³³ Blessed Dementia Rating Scale,³⁴ CSI-D Cognitive Activities Questionnaire,¹⁷ 10/66 Dementia Research Group Informant Questionnaire,³⁵ HRS Activities Questionnaire,¹³ and Clinical Dementia Rating (CDR) semi-structured interview.¹⁸ To help raters navigate through the informant data, different elements from the aforementioned scales were organized under the following headings: (1) respondent's mental status for information on changes in respondent's cognitive abilities, judgment, problem solving, and personal care activities; (2) respondent's activities for changes in Activities of Daily Living (ADLs), Instrumental ADLs (IADLs), leisure activities, as well as a general overview of respondent's social engagement and community affairs; and (3) respondent's behavior for any behavioral and mood changes reported by the informant (Figure S1).

All clinical data were reported under the health history column on the assessments page. Clinical findings were summarized as normal/abnormal for each of the following evaluated neurological exam components: face (muscle weakness, swallow, and tongue mobility), speech, upper and lower limbs mobility, balance, gait, vision, hearing, and any other observations noted by the nurse. Raters were also able to click on any of the neurological exam components for a more detailed overview of clinical findings. Other reported information included blood pressure, pulse, grip strength, and timed up and go readings as well as information on relevant medical history and medications (Figure S1).

2.3 Consensus panel

The raters panel consisted of US and South African clinicians with expertise in evaluating individuals with possible cognitive impairment or dementia. Experts on the panel included neurologists, geriatricians, neuropsychologists, as well as researchers with specialized training in the field of dementia. Selected raters had expertise in the clinical and research settings of cognition and dementia, in diverse populations, as well as experience with the neuropsychological assessments and diagnostic criteria used in the study.

All raters completed an online training session on how to navigate the web portal and submit ratings. During training, raters were also required to independently submit ratings for a number of pilot cases and provide feedback on their experience with the portal. Revisions to the web portal were made during the training phase only. Additionally, a detailed manual was provided to all raters, including guidance on the consensus portal, detailed descriptions of cognitive tests and other measures, and an overview of the diagnostic criteria used.

For the online consensus review, each case was reviewed by three raters, randomly chosen at the time of review. Cases were completed at raters’ own pace, so some raters completed more cases than others. All three raters needed to agree on the final NIA-AA diagnosis (cognitively normal, MCI, or dementia). Cases for which there was disagreement on the final diagnostic category among the three independent reviewers were discussed during biweekly consensus conference calls.

Consensus conference calls, which were led by a moderator, were open to all raters on the committee, not just raters who originally reviewed the case, and typically had between three to six raters in attendance. For each discrepant case, the moderator presented all relevant information including the neuropsychological testing profile, informant interview, and clinical observations from the neurological exam to the consensus panel. Members of the consensus panel were also able to log in and see the details of the case currently under discussion. The moderator then focused the discussion on the specific item(s) that led to disagreement among the three reviewers and allowed all consensus members to discuss the case and agree on a final diagnosis (Figure 1).

2.4 Statistical analyses

Distributions of diagnostic outcomes (cognitively normal, MCI, dementia) by age and sex were summarized with frequency counts and percentages. Differences in cognitive measures between diagnostic outcomes were assessed using chi-square tests and one-way analyses of variance (ANOVA), where appropriate. Overall concordance rate was measured as overall agreement between each rater's initial diagnoses (the ratings first entered in the online portal) divided by the final diagnoses (based on consensus agreement via the portal or consensus conference).

An unweighted Cohen's kappa coefficient (κ) was calculated to account for agreement expected to occur by chance alone.^{40, 41} Cohen's kappa coefficient (κ) was also used to assess the level of agreement among raters by key NIA-AA diagnostic criterions such as impairment based on objective testing and/or informant report, and impairment in social and occupational function, as well as by NIA-AA diagnostic cognitive domains (memory, executive, language, visuospatial functions, and behavioral changes).

Kappa statistics were interpreted using the Landis and Koch method in which strength of agreement was defined as poor (< 0.00), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00).⁴²

Concordance between rater's diagnosis and final diagnostic outcomes were also compared by rater specialty. Specifically, using the final consensus diagnosis as the gold standard outcome, we coded the diagnosis rating as 1 “cognitively normal,” 2 “MCI,” or 3 “dementia” and calculated the average difference score between each rater's assigned rating and the final consensus rating.

All analyses were performed using STATA 16 (StataCorp) software by authors M.G. and D.T.B.

3.1 Participants’ diagnostic outcomes, baseline characteristics and cognitive profile

A total of 635 cases were reviewed by a panel of 11 consensus members (six US and five South African) and assigned a diagnosis of cognitively normal, MCI, or dementia. From July 2020 through October 2021, each case was randomly assigned to three consensus members for independent review. Consensus panel members were compromised of neurologists (n = 5), neuropsychologists (n = 2), geriatricians (n = 2), and dementia researchers (n = 2) with a mean experience of 12 years (range, 6–25 years) and an equal number of men (n = 6) and women (n = 5) on the panel. On average, it took raters between 3 and 5 minutes to review each case, with more complex cases taking approximately 10 to 15 minutes to complete. After the independent individual ratings, 298 cases (47%) showed disagreement in the final diagnosis among the three raters. Discordant cases were resolved during biweekly conference calls and were mostly decisions between cognitively normal and early MCI, or late MCI and dementia.

The mean age in our cohort was 69.87 (standard deviation = 11.55) and the majority of study participants had no formal education (55%). Final diagnostic categories included 314 cognitively normal (49%), 184 MCI (29%), and 137 dementia (22%) cases, which reflects our initial sampling framework to oversample for respondents categorized as higher risk for dementia. The distribution of diagnoses by age and sex is presented in Table 1.

We also compared mean cognitive scores by diagnostic group for 631 participants (four participants were missing the cognitive battery). Overall, participants diagnosed with MCI and dementia performed worse on all cognitive measures (Table 2). Differences in mean cognitive scores between groups were significantly different from zero for all cognitive tests constituting the memory, executive, language, and visuospatial domains (Table 2). Visuospatial tasks, especially those requiring drawing, had high rates of missingness; hence, interpretation of these results may be tenuous.

3.2 Rater reliability: agreement between individual ratings and final diagnoses

As each case was reviewed by three raters, a total of 1905 individual diagnostic ratings were made. Overall agreement between each rater's initial diagnosis and the final diagnosis (based on the consensus agreement via the portal or consensus conference) was 78.3% (1494/1905). The average κ statistic was 0.50, suggesting a moderate level of agreement between raters’ individual ratings and the final diagnostic outcomes (Table 3). Upon investigating agreement level by diagnostic outcome (at final determination), we observed a higher level of overall rater agreement with cognitively normal (k = 0.61) and dementia (k = 0.57) diagnoses and conversely lower agreement with MCI (k = 0.34). Kappas for individual raters ranged between 0.34 and 0.96, but measurement of inter-rater reliability was not possible due to differences in number of cases reviewed by each rater (Table 3).

To better understand the reasons for disagreement between raters, we investigated the percent agreement between raters for main NIA-AA diagnostic criterions (cognitive impairment and social and occupational functional impairment) among cases that triggered consensus conference calls. In general, of those cases that were adjudicated between MCI and dementia, we found that discordance among raters was mostly due to difficulty in assessing social and occupational functioning (Figure 2A). Specifically, the three raters almost always agreed on whether the participant had cognitive impairment (91%), but disagreement on the social and occupational functioning criterion was responsible for > 50% of cases sent to adjudication. Another reason for discordance among raters was the absence of normative data; cases between cognitively normal and MCI were sent to adjudication conference calls mainly because of disagreement on the presence of cognitive impairment (>85% of cases; Figure 2A). Upon further exploration of the cognitive impairment criterion by cognitive domain, we found the highest level of disagreement (74%) among raters particularly for tests within the visuospatial domain, followed by the memory domain (70%; Figure 2B).

We additionally compared variations in diagnosis of cognitive impairment by rater's professional specialty. We classified raters into two categories: neurologists/geriatricians (seven raters) and neuropsychologists/dementia research psychologists (four raters). An average score of zero indicated congruency between the rater's individual rating and the final consensus rating, while an average score above zero indicated overdiagnosis of MCI/dementia and an average score below zero indicated underdiagnoses of MCI/dementia. Neurologists/geriatricians were more likely to overdiagnose MCI/dementia, averaging positive difference scores compared to neuropsychologists/dementia research psychologists. One neuropsychologist had an average difference score below zero, indicating leniency toward reporting normal cognitive function for study participants who had a final assigned consensus diagnosis of MCI (Figure 3).