Introduction

More than 300,000 unique healthcare available in major app stores and much work has addressed the feasibility and efficacy of these apps [1]. Despite promising results, the impacts of a health app could be lost if users reject using the app. Hence the question: what are factors that inhibit or enhance the actual use of these apps? Although Unified Theory of Acceptance and Use of Technology (UTAUT) [2,3,4] shows factors influencing technology acceptance in various domains, less is known about these factors in the context of health apps. Therefore, we report a case study set the insomnia treatment domain.

Insomnia is a sleep disorder where people have trouble with falling asleep or staying asleep. About 10 − 15% of adults report symptoms of insomnia associated with daytime consequences, and it is estimated that 6 − 10% of adults meet the criteria for an insomnia disorder [5]. Currently, Cognitive Behavior Therapy (CBT) is widely recognized as an effective non-pharmacological treatment for insomnia. Van Straten et al. [6] reported that a CBT for insomnia (CBT-I), either its components or the full package, was effective for treating insomnia.

CBT is a short-term psychotherapy that helps patients change self-limiting beliefs, develop new behaviors, and manage emotions. CBT-I techniques include relaxation, sleep restriction, stimulus control, cognitive therapy, and sleep hygiene. Relaxation exercises help the person to relax and to attain a state of calmness. Sleep restriction involves controlling time in bed based on a person’s sleep efficiency to restore the homeostatic drive to sleep. On the other hand, stimulus control aims at restoring the person’s positive association of the bed and bedroom with sleep. Cognitive therapy aims at changing dysfunctional beliefs and attitudes, whereas, sleep hygiene education aims to increase awareness of the behaviors and environmental factors that promote sleep.

Internet-delivered CBT-I has been developed to improve the availability of personal therapists by promoting adherence at a low cost and providing both long-distance communication and tailored interaction. Ritterband et al. [7] showed large improvement effects as a result of fully automated CBT-I treatment. Given advancements in mobile technology, an effective-CBT-I app could offer an opportunity to enhance the advantages and coverage provided by other formats of CBT-I, enabling real-time and personalized monitoring, assessment, and interventions [8, 9]. In particular, our developed SleepCare app was a stand-alone mobile phone delivered CBT-I and packaged with a sleep diary, a conversation module, a relaxation exercise, a sleep-restriction exercise, and sleep hygiene and education [8,9,10,11,12,13]. Horsch et al. [13] reported moderate significant effects of using this app according to insomnia severity index (ISI d = -0.66) and sleep efficiency (d = 0.71), which were also maintained at a three-month follow-up.

To the best of our knowledge, no studies have evaluated the acceptance and use of CBT-I apps. To understand factors that may explain the adoption of a CBT-I app, this paper presents an analysis of users’ behavioral intentions to use the app and their use-behavior, and underlying factors that could explain variation between users on these factors. The analysis was based on data obtained as part of a randomized controlled trial. It focused only on the experimental condition in which people with insomnia used the SleepCare app for seven weeks (n = 89). The Partial Least Square Structural Equation Modeling (PLS-SEM) method was chosen for this study due to its ability to handle heterogeneous data with a small sample size [14].

Research model

With CBT-I, individuals engage deliberately in activities that are expected to reduce their insomnia symptoms. They make conscious attempts to affect behavior that is not under volitional control, such as falling asleep or waking up. Their behaviors are shaped by beliefs towards the changed behavior and the support tools. This process can be depicted as a model of factors that determine the acceptance of app-based CBT-I and the related behavior (Fig. 1).

Fig. 1
figure 1

Research model

Full size image

The research model was developed based on the extended model of the Unified Theory of Acceptance and Use of Technology (UTAUT2) [3], i.e. a single theory combining many theories often used to understand technology use, which includes: the Theory of Reasoned Action, the Theory of Planned Behavior, Self-Determination Theory, Innovation Diffusion Theory, the Motivation Model, Social Cognitive Theory, the Model of Personal Computer Utilization, and the Technology Acceptance Model. Since its introduction, the UTAUT model [2] has been applied in multiple domains [4]. The meta-analysis using the original UTAUT model yielded medium to large associations between the UTAUT determinants and behavioral intention [15]. This model has often been extended beyond its original remit to encompass determining factors of behavioral intention, namely performance expectancy, effort expectancy, and social influence [3, 4]. In UTAUT2, Venkatesh et al. extended the model to include facilitating conditions, hedonic motivation, price value, and habit as determinants of behavior intention [3]. Table 1 gives an overview of the adaptation of these determinants for this study.

Table 1 Factor affecting behavioral intention and use-behavior
Full size table

According to the UTAUT2 model, people’s behavioral intention is a key mediating determinant for behavior under volitional control. Therefore, the factors considered influencing the intention in this study are limited to those factors that people can explore consciously, and hence people can be queried about them. These factors are expected to shape people’s intentions to engage in specific behavior and are therefore regarded as predictors for intentions—or in this case, the decision to use a CBT-I app. While including hedonic motivation, i.e. the user’s positive feeling about using the app, this study excluded two factors of UTAUT2: price value and habit, because the participants of the study did not have to buy the app and the seven-week limitation of interacting with the app was not regarded long enough for these participants to form a habit. Instead, the study considers three additional determinants because of previous findings. They are: (1) self-efficacy, as it has a strong relationship with users’ intention to use healthcare technologies [19]; (2) trust, as it influences users’ intention to use health information systems [20]; and (4) anxiety, as it was a significant predictor for users’ intention to use information technologies [21, 22].

For technology use, however, individuals’ perceptions of the available resources and support to help them use the technology are also important [2, 3]. Hence, UTAUT2 includes facilitating conditions as a second determinant [3]. The meta-analysis of Khechine et al. [15] reported medium associations between the UTAUT determinants, i.e. facilitating conditions and behavioral intention, and use-behavior.

In the CBT-I delivered by the SleepCare app, user behaviors consist of exercise behaviors (cf. main activities) and app-related activities (cf. supporting activities) [8, 9]. The former refers to actions that are usually related to the advised exercises, e.g. sleep restriction and relaxation exercise; and the latter is related to actions which the users engage in with the app, e.g. journaling using the sleep diary, and having consultations or dialogues with the app. Therefore, this study had defined one construct to capture these user behaviors: use-behavior.

Method

To identify significant influence within the relationships included in the model (Fig. 1), the present study analyzed 10 hypotheses (Table 2).

Table 2 List of hypotheses tested in the study
Full size table

The data was collected as part of a randomized controlled trial. The goal of the trial was to demonstrate the app’s efficacy in a sample of patients with relatively mild insomnia to test the proof of principle before investigating it in a more severely affected population. Although the trial had both app- and waitlist-conditions, only data collected from participants who received CBT-I treatment was examined in this study. In this section, the demographics of participants, a description of the app, the measurement instruments, the procedure of the study, and the data analysis are presented.

Participant

The participants in this study were recruited from August 15 to October 21, 2015, via websites, social media, online advertisements, flyers, and a press release. 151 participants out of an initial group of 639 voluntarily recruited individuals were included based on the inclusion and exclusion criteria (Fig. 2, [23]). The inclusion criteria [13] were: (1) difficulty with initiating or maintaining sleep for at least 30 min a night, for at least 3 nights a week, for at least 3 months, causing clinically significant distress or impairment in daily functioning, under the criteria for the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) diagnosis of insomnia [24]; (2) stable medication use; (3) being eighteen years or older; and (4) having a valid email address, being connected to the internet, and in possession of an Android mobile phone (operating system version 4.1 or higher). The exclusion criteria were: (1) total sleep time < 5 h on average as reported in a 7-day sleep diary; (2) Insomnia Severity Index (ISI) score lower than 7 [25]; (3) earlier treatment with CBT-I; (4) started other psychotherapy in the last six months; (5) self-reported diagnosis of schizophrenia or psychosis; (6) alcohol or marijuana abuse (more than three glasses of alcohol a day for at least 21 days a month or use marijuana more than once a week); (7) possible sleep apnea (determined with a subscale of the SLEEP-50 questionnaire [26], cut-off ≥ 15); (8) shift-work; (9) pregnancy or breastfeeding; or (10) symptoms of depression (determined with a subscale of the Centre for Epidemiological Studies Depression (CES-D) scale [27, 28], cut-off ≥ 27).

Fig. 2
figure 2

CONSORT flow diagram of recruitment, reasons for exclusion, experimental compliance, modified version of [13]

Full size image

118 participants installed the app on their mobile devices, but only 89 participants filled in the tailor-made Sleep App Acceptance Questionnaire. Table 3 shows the demographic data of these participants.

Table 3 Demographic characteristics of participants in the Sleep App Acceptance Questionnaire based on [13]
Full size table

System description

The SleepCare app [8, 9, 11,12,13] was based on previously published protocols (e.g. [29, 30]). It offered a seven-week CBT-I program, which consisted of six weeks of CBT-I exercises (as recommended by [6]) plus an additional one week for managing the participants’ adherence. The interface of the app was composed of a set of personalized conversations and specialized modules that formed a coherent structure of input and output facilities [9]. The conversations included introductions to the exercises, tailored advice for the individual user to do the exercises, appointments for consultations, shared decision-making about exercise-related assignments, evaluations of progress in the therapy, and feedback based on the current state of the user [11]. The app was fully automated.

The SleepCare app offered a sleep diary, a relaxation exercise, a sleep-restriction exercise, and sleep hygiene and sleep education. These exercises were in Dutch, tailored to the participants. The sleep diary was introduced on the first day of the therapy and could only be done for the current day. The relaxation exercise was introduced on the fourth day. The user was advised to do the relaxation once a day, but they could do it as often as they wanted [31]. After a week or when the condition that the user had filled out at least six sleep diaries and had an average sleep efficiency of less than 85%, the app introduced the sleep-restriction exercise. For this exercise, the app was equipped with an algorithm to calculate the ideal and maximum time in bed, creating boundaries for negotiation [13]. Finally, sleep hygiene and sleep education was presented as tips and facts that aimed for increasing the efficacy of the other exercises, in text format on a dedicated screen.

Measurement

Questionnaire measures

A tailored questionnaire, i.e. Sleep App Acceptance Questionnaire, was developed to measure the following constructs: Performance Expectancy, Effort Expectancy, Social Influence, Self-Efficacy, Trust, Hedonic Motivation, Anxiety, Facilitating Conditions, and Behavioral Intention. Appendix A shows the list of items used in the questionnaire, translated into English. Each item was rated on a 7-point Likert scale (from 1 = strongly disagree to 7 = strongly agree) including a “not applicable” option which the participants could select if the item did not apply to them.

Behavioral measures

Recorded interaction data of the participants were processed to measure one use-behavior construct that consisted of three indicators: (1) the number of days the sleep diary was completed (SD-DO); (2) the number of completed conversations (CONV-DO) and (3) the number of days the participant did at least one relaxation exercise (RE-DO).

Procedure

Participants gave online informed consent and filled in the questionnaire asking about the inclusion and exclusion criteria (including ISI) and demographics. The participants who met the criteria received a link to an online sleep diary by e-mail. For seven successive days, emails for the diary were sent at 6 a.m. and a reminder was sent at 10 a.m. Participants who reported an average total sleep time of more than five hours were included in the later stage of the study and randomly assigned to the app-condition or the waitlist-condition. The app-condition participants were allowed to install the app, while the waitlist-condition had to wait seven weeks before they were also allowed to enter the app-condition and install the app.

Three weeks after starting to use the app, all participants in the app-condition received another ISI questionnaire and the Sleep App Acceptance Questionnaire. At the same time, they received additional questionnaires to measure their dysfunctional beliefs about sleep and their motivation to use the app. At the end of seven weeks, all participants in the app-condition received a request to complete additional questionnaires, which also included the final ISI questionnaire. Again, they received additional questionnaires to measure their dysfunctional beliefs about sleep, anxiety, and depression symptoms. They were also asked to fill in a seven-day online diary to measure sleep variables such as sleep efficiency. After completing the online diary, the participants received a three-month follow-up questionnaire and another online diary assignment.

Note that the analyses of ISI, the dysfunctional beliefs about sleep, anxiety and depression symptoms, and the online diary assignments completed by the participants have been reported elsewhere [13].

Data analysis

In this study, two types of PLS-SEM tests were conducted: (1) single-path tests that examined the relationships between two latent variables in the model separately, and (2) multiple-path tests that examined all the relationships in the model as a whole. The tests were analyzed using SmartPLS 3.0 [32]. Nonparametric bootstrapping-based 95% confidence intervals were computed and used to determine the significance of estimates. Data, scripts, and analysis files are online available [33].

Additionally, mediation analysis [34] was conducted to examine the mediation of the Behavioral Intention in the relationship between significant determinants (of Behavioral Intention) and Use-Behavior. Behavioral Intention was hypothesized to play a mediating role between variables early and later in the causal chain. For example, the Behavioral Intention construct clarifies how beliefs about expected performance influence the usage of the app.

Minimum sample size

Initial sample size estimation was primarily guided by the objective of comparing intervention and waiting list conditions of the trial. Fortunately, data collected in the intervention condition also abided to recommendations for PLS-SEM (Table 4), although not for the more traditional covariance-based SEM. The latter would require a recommended minimum sample size of 200 [35] as it pursues different objectives to this study, i.e. to test and confirm a prior theory. Instead, PLS-SEM with its explorative nature was used to examine relationships between the latent variables as hypothesized. The 89 samples collected falls within the range of minimum sample sizes recommended by various methods for PLS-SEM (Table 4), when pursuing a statistical power of 0.80 for detecting at least a medium-size effect of R2 (with 0.05 probability of error).

Table 4 Recommended minimum sample size for PLS-SEM
Full size table
Fig. 3
figure 3

The bootstrapping results of the multiple-path analysis (n = 89)

Full size image

Data characteristics

We considered an item in the questionnaire results as an outlier concerning its representativeness for the application domain if the number of participants who rated the item as ‘not applicable’ was more than Q3 + 1.5*IQR, with Q3 as the third quartile and IQR as the interquartile range of the ‘not applicable’ frequency across the questionnaire item measured. This led to the exclusion of items BI4, SI3, SE2, HM6, FC3, and FC4 from the analysis. For the remaining items, ‘not applicable’ was treated as a missing value. Only items BI1, BI2, BI3, and BI7 were answered by all participants. This resulted in 90 (2.3%) missing values with two participants out of 89 (5.6%) were missing all their behavioral data. The missing values were replaced using multivariate imputation based on a Bayesian linear regression method using the R version 3.3.3 mice package version 2.25 [38]. A detailed description of the measurement variables (n = 89) can be found in Appendix B.

Latent variable assessment

In this study, all variables were considered as reflective latent constructs and were assessed based on four criteria, as shown in Table 5. The assessment resulted in the removal of EE5-R, SI5-R, TR2-R, HM7-R, AN2-R, and AN3-R from the model (Appendix B).

Table 5 Assessment methods of reflective constructs
Full size table

Results

Single path tests

Only Anxiety was not found to show a significant association with Behavioral Intention (Table 6). The other determinants, i.e. Performance Expectancy, Effort Expectancy, Social Influence, Self-Efficacy, Trust, Hedonic Motivation, and Facilitating Conditions, revealed associations with Behavioral Intention with medium (0.15 ≤ f2 < 0.35) to large effect sizes (f2 ≥ 0.35). Furthermore, the cross-validated redundancy values provided support for the models’ predictive relevance (Q2 > 0). Both Behavioral Intention and Facilitating Conditions were significant determinants for Use-Behavior. The association of Use-Behavior with Behavioral Intention could be classified as large, whereas with Facilitation Conditions was medium.

Table 6 The bootstrapping results of the single-path analyses (n = 89)
Full size table

Multiple-path test

The multiple-path test result in Fig. 3 shows only Hedonic Motivation was a significant determinant for Behavioral Intention (H6). Furthermore, as in the single-path test, the analysis found that Behavioral Intention (H9) and Facilitating Conditions (H10) were both significant determinants for Use-Behavior. Only the association of Use-Behavior with Behavioral Intention could be classified as medium, while the other relationships revealed a small effect size (0.02 ≤ f2 < 0.15). Appendix B presents the results in a table format.

Mediation analysis

The mediation analysis result showed that Behavioral Intention was fully mediating the relationship between Hedonic Motivation and Use-Behavior (Fig. 4). The mediation by Behavioral Intention on the relationships seems to be consistent with the research model. Appendix B presents the results in a table format.

Fig. 4
figure 4

Effects of the relationship between Hedonic Motivation and Use-Behavior with mediation (n = 89)

Full size image

Discussion

Fig. 5 presents the main findings of the study combining the single path analyses (grayed out) and the multi-path analysis (presented in bold). The findings of the single path analyses supported the viability of the model in assessing factors affecting behavioral intention to use the CBT-I app. The results showed that Performance Expectancy (H1), Effort Expectancy (H2), Social Influence (H3), Self-Efficacy (H4), Trust (H5), Hedonic Motivation (H6), and Facilitating Conditions (H8) could explain variation in the intention to use the app. However, the related multiple-path test revealed that the explanatory power of these individual factors did not go beyond the Hedonic Motivation factor, i.e. people’s positive feelings about the app, which accounted for 61% of the variance in Behavioral Intention between individuals. This result supported the inclusion of Hedonic Motivation in UTAUT2 [3]. Wang et al. [41] also found that the positive feelings about the system outperformed these traditional factors when explaining behavioral use among renal patients using a self-management support system.

Fig. 5
figure 5

Conceptual model of the findings: the results of single-path analyses (grayed out) and multiple-path analysis (bold)

Full size image

In contrast to the results of Venkatesh et al. results [3], the findings of the multiple path analysis showed that Performance Expectancy, Effort Expectancy, Social Influence, and Facilitating Conditions did not significantly predict Behavioral Intention. One consideration could be that both this study and Wang et al. [41] used data obtained in the context of a trial. This is likely to suffer from a sampling bias. Participants who volunteer for this type of study may have prior beliefs that such an intervention would be beneficial for them, and confidence in their ability to use the technology to accomplish the tasks provided during the study. Consequently, this may have reduced the predictive power of the traditional UTAUT2 predictors. It seems that the hedonic motivation aspects are crucial when people have the freedom to choose. Comments made by the participants at the end of the study confirmed this. Some participants mentioned that they had fun and felt good using the app, as it helped them understand their sleep patterns. Others indicated that they needed more personalized features implemented in the app to help with their sleep problem. Without these aspects, as explained by Davis et al. [42], the expectations of the app’s performance and the social environment may influence people’s intention to use the technology for the first time. People may simply not know if they will like the app before they actually use it. In both this and Wang et al. [41] studies, however, determinants for behavioral data were collected when people had obtained some actual user experience.

Consistent with UTAUT2, Anxiety did not have a significant influence on people’s Behavioral Intention, and therefore no support was found for H7. One explanation could be inferred from the contextual factors, such as the substantial proportion of highly educated participants (i.e. about 73% were university graduates) and early middle-aged individuals, which together might indicate a demographics more at ease with using technology in this context.

The analysis also showed that Behavioral Intention (H9) and Facilitating Conditions (H10) were significantly associated with Use-Behavior, together accounting for 32% of the variance in people’s app interaction behavior. Both behavioral intention and facilitating conditions are important as they are predictors for people’s commitment to exercise behavior, in this case, journaling sleep diaries, having dialogues with the app, and doing the relaxation exercise. Furthermore, it revealed that Behavioral Intention fully mediated the impact of Hedonic Motivation on Use-Behavior. This finding was following other theories of behaviors which have long recognized Behavioral Intention as an important mediator in the relationship between behavior and other factors, such as attitudes or subjective and perceived behavioral control [3, 15, 17, 42].

In practice, the findings of this study suggest that the design and the implementation of a CBT app should take into account not only mobile phone-delivered CBT usability, social influences, and communicating potential benefits, but also the emotional experience of using the app. Some strategies for enhancing the emotional experience of app users were proposed by the participants at the end of the study. Very prominent were suggestions to automate diary entry and personalize the features developed in the app, such as the variation of exercises, music, and the voice used in the relaxation exercise, and also to personalize the conversation contents and the information displayed on the interface. Such improvements to the app were anticipated to increase the positive feeling of the users about the app. Other strategies have been proposed [8,9,10], which included usable and attractive design approaches for improving therapy adherence. Moreover, when offering therapeutic services on a mobile phone, it is also important to consider facilitating conditions, i.e. people’s perceived knowledge about using the app, and also the phone’s ability to run the app smoothly. Given the extensive variation in mobile phone models and different versions of operating systems in use [43], developers might be forced to find a balance between an attractive design and a solution with low demand for computational resources.

To fully appreciate the findings, understanding the study’s limitations is necessary. First, although to the best of our knowledge the data collected about mobile phone-delivered CBT-I was unique, the sample size was relatively small considering the number of factors included. A larger sample size would allow a confirmatory factor analysis and covariance-based structural equation modeling for objective estimation of the proposed model fit. Other limitations are related to the data collected in the study, which could put constraints on the generalizability of the findings and may cause a lack of consensus with other studies in the same context. The data collected by the questionnaires only provided snapshot information about beliefs three weeks after using the app. For example, the beliefs people held before downloading the app were not considered. Moreover, these questionnaires were tailor-made and adapted specifically to the purpose of the study. Another limitation is related to the representativeness of the Dutch sample for other countries, and the fact that a disproportionately large section of the sample consisted of highly educated participants. Furthermore, a substantial number of participants rated some of the questionnaire items, such as Behavioral Intention, Hedonic Motivation, Self-Efficacy, Social Influence, and Facilitating Conditions related to the mobile technological aspects, as not applicable. In addition, the assessment of latent variables led to the removal of six questionnaire items (observed indicators) that were negatively formulated. Reconsidering these items is therefore advisable for the future. Lastly, the technical issues that occurred during the trial made it impossible for some participants to continue to the next conversation. The occurrence of this problem was monitored and solved when needed. In these cases, a new conversation was manually planned in the database for these participants, and an email with instructions to update the app was sent to these participants.

Conclusion

This study examined factors affecting users’ behavioral intention to use a cognitive behavioral therapy for insomnia delivered by a fully automated mobile app and the app-usage behavior. The study revealed that hedonic motivation (to use the app) was associated with people’s intention to use the app. As the behavioral intention was considered as an indication of the users’ acceptance, the results suggested that hedonic motivation modulated acceptance and in turn, together with facilitating conditions, affected their use-behavior with the app. These findings warrant more research into computerized persuasion strategies that support the hedonic motivation of behavioral change app users.