Persuasive communication systems: a machine learning approach to predict the effect of linguistic styles and persuasion techniques

2.1 Input features for persuasion systems

We start by describing the input features used in a persuasion system and the scale we used to collect them. Several investigations have yielded information about the importance of attitude assessment for persuasion. O’Keefe (2015) states that people’s attitudes (i.e. values, personality traits, feelings and emotions) are the special interest of a persuader, because attitudes represent stable evaluations that can influence behaviour. Furthermore, the importance of alignment between attitudes and the persuasive message has been emphasised, noting that this functional matching is crucial for the persuasion process (Petty and Cacioppo, 1986). In addition, O’Keefe (2015) argued that attitudes and the belief behind the attitude could determine people’s actions, e.g. what products people buy, what policies people endorse, what hobbies they pursue. Furthermore, O’Keefe argues that the systematic study of persuasion requires assessing people’s attitudes – with attitudes representing the persuasive target.

With many studies suggesting the importance of assessing and measuring people’s attitudes, we reviewed the literature to ascertain useful variables/input features for the development of ML models that can predict persuasive message impact based on user attitude and demographic data. The underlying hypothesis is that the complex relationship between a user’s profile and their response to different persuasion techniques can be learnt by ML algorithms. The data used for building persuasive profiling models commonly includes demographic information (e.g. gender, age and postcode), whereas some studies also incorporate personality traits (Shumanov et al., 2021; Matz et al., 2017; D’Souza Tay, 2016b; Anagnostopoulou et al., 2017; Spielmann et al., 2016). A variety of attitude assessment and data collection techniques have been used in studies – for example, data can be supplied voluntarily in exchange for internet services. Furthermore, current trends in research on data collection provide scope for implicit measure assessment. Examples of this type of data collection include digital footprints (e.g. search history, site usage, click data)(Gencoglu et al., 2015), Facebook likes (Marengo et al., 2020; Azucar et al., 2018) and Twitter posts (Setiawan and Wafi, 2020). Data harvested from digital sources is beyond the scope of this paper – for a review, see Laperdrix et al. (2020), Pugliese et al. (2020). This line of research provides evidence that the quantification of user activity on social media platforms is being explored to measure the effectiveness of targeted persuasive messages (Bossetta, 2018; Farseev et al., 2015; Farseev and Chua, 2017). In this study, participants data were collected via the survey described in Section 3.1. The personality traits and attitude scales are collected using the TIPI and the DAS scale, that we present in the remaining of this section.

2.2 Persuasion techniques

Persuasion techniques are a set of linguistic features that can modify the intended core statement without necessarily changing the content of information delivered in the message (Holtgraves and Lasky, 1999). Persuasive messages carry embedded persuasion techniques and linguistic strategies such as language style, variability, intensity, tag words, framing and rhetorical devices (Xu and Tan, 2020; Renaldo, 2017; Kaur et al., 2013). These types of persuasive messages aim to elicit the peripherical route of persuasion/System1 (instinctive and emotional thinking) (Kahneman, 2013; Petty and Cacioppo, 1986), rather than the central route (conscious and logical thinking)(Dove, 2021). Persuasive messages are inevitably biased – caused by the rhetorical and linguistic features of persuasive language. When designing persuasive messages, it is common to use relatively informal styles such as imperatives, contractions, clipping and subject/auxiliary omissions (Labrador et al., 2014).

A wide array of persuasion techniques is adopted across a myriad of domains. Researchers often select a technique to use based on the persuasion target or the content of the message, or instead use techniques that are customarily used in their domain. In this paper, we investigate and apply a range of techniques that are standard in fields including psychology, rhetoric and marketing. Persuasive strategies are extensively studied, and much research can be found which includes relevant definitions and descriptions of the persuasion techniques we have used – for a review (Dillard and Pfau, 2002; Myers, 2012; McGuigan, 2011; O’Keefe, 2015; Carey, 1996; Harris, 1997).

In this study, we have used persuasion techniques that elicit the peripheral route of persuasion (also known as System 1) and embedded each technique within a statement relating to the benefit of third-level education. The selected techniques include appeal to finances and logic (Aristotle, 2015; Aristotle and Cooper, 1960), rhetorical devices (anaphora, antanagoge, epistrophe, rhetorical question) (Harris, 1997), cognitive theories (flattery, awareness words, illusion of superiority, priming/semantic priming) (Fogg and Nass, 1997; O’Keefe, 2015; Stengel, 2002) and Cialdini’s Principles of authority and social proof (Cialdini, 1987). A summary with a brief description of persuasion techniques employed in this work is provided in Table 1.

In this research, to acquire the input features for the supervised learning algorithms, a survey was designed (Section 3.1) that captures demographic and attitude information i.e. personality (TIPI) and belief system-based variables (DAS) this is our input matrix. The response variable is a score given by participants which indicates the level to which they felt influenced by a presented statement. The statement contained a core message such as the benefits of education and an embedded persuasion technique. The participant input data and statement scores were then used to develop and evaluate ML models.

Figure 1 shows a proposed design matrix for modelling persuasion. The design matrix consists of two main parts: the independent variables (user profile data) and the dependent variable (the score each user gives to a persuasion technique. Source: Authors work.

2.3 Machine learning

ML is a discipline that models data by integrating and applying methods from fields such as computer science, mathematics, statistics, data mining and distributed systems. The study of the link between users’ profiles and persuasive strategies has been widely confirmed in marketing and advertising research (Hirsh et al., 2012; Clark and Çallı, 2014; Plummer, 2000). However, most of the studies made use of conventional statistical models for predicting the effect of persuasion techniques given the user’s profile (Pangbourne et al., 2020; Thomas et al., 2017; Wang et al., 2019). Also, it has been noted that most studies tend to focus on a few persuasion techniques, for example Cialdini’s principles of persuasion (Cialdini, 1987), disregarding the large array of persuasion techniques available in the literature (McFarland and Dixon, 2019). Although, the interest in using ML in persuasion research is growing exponentially, e.g. (Wang et al., 2019; Lukin et al., 2017a; Shmueli-Scheuer et al., 2019a, 2019b), still there are relatively few proposals that formalise user modelling through ML methods.

Compared to conventional statistical models, ML has the capability of capture nonlinear relationships between input data (for example, personality traits, emotions, values, demographical information) and the associated output (e.g. target linguistic styles/persuasion techniques). A relatively limited number of works in the field of PC and advertising have used ML methods, specifically for the prediction of linguistic styles and persuasion techniques, despite ML methods have been used in multiple fields and industries including experimental psychology and behavioural change science. The adoption of such methods more widely in the field of PC would represent a promising line of research with potential to discover more powerful and performant models. A summary of recent publications, methodologies and their goals, along with reported applications is provided in Table 2.

3.1 Survey design

To gather the input features for the supervised learning algorithms, a survey was designed that captures demographic and attitude information (i.e. TIPI, DAS scores). To measure the influence of the embedded linguistic styles/persuasion techniques present in statements on participants, a Net Promoter Score (Reichheld, 2003) was adopted wherein the participant indicates their level of agreement with each statement – survey participants score each statement on a scale from 1 (no effect) to 10 (very convincing). In this study the persuasion context was in the domain of education. Statements contained a core message aimed to persuade about the benefit of pursuing a third level education – with multiple variants being presented that each had an embedded persuasion technique accompanying the core message content. These self-reported scores (called influence scores) formed the response variable representing the ground truth for our supervised ML models. Given that participants may score each statement variant (with different embedded persuasion techniques) differently, any variation in these scores indicate an individual’s relative preference for the linguistic style/persuasion technique.

The survey was deployed using Qualtrics software (qualtrics.com) in August 2020. It contained four sections, i.e. demographic information, TIPI personality traits (Gosling et al., 2003), DAS attitude information (Burns, 1981) and the persuasion tasks (presented statements for rating). The full questionnaire can be found in Appendix. Participants were recruited via the crowd-sourcing marketplace Prolific (prolific.ac.uk). Inclusion criteria required participants to be native English speakers or non-native with high proficiency. Three attention questions were included in the survey to ensure that participants were engaged. Additionally, participants were provided with instructions requesting them to answer all questions as honestly as possible. They were also given the instruction: “Please answer each of the questions as you feel right now”. The survey was answered by 1,061 participants with data from 1,022 participants ultimately being used. A total of 39 submissions were excluded, as these participants either failed two of the three attention questions, completed the survey in an unreasonably short time or did not fully complete the survey.

To assess the reliability of test scores, we used Cronbach’s α which is an internal consistency estimate that is derived from classical test theory and indicates a theoretical measure of reliability. Internal consistency estimates can be used to measure item homogeneity, i.e. the correlation between items on a test that are intended to measure the same construct (Henson, 2001). Cronbach’s α test was conducted by using the SPSS statistical package (IBM SPSS version 26). For the TIPI data, Cronbach’s α was 0.355, suggesting a low level of internal consistency when considering the Vellis scale (Vellis, 2003). Gosling (2017) notes that the TIPI test is expected to possess a low Cronbach’s α. Tavakol and Dennick (2011) highlight that Cronbach’s α levels are affected by the length of the test – if the test length is too short (as in the TIPI case) the value of alpha is reduced. Comparatively, the DAS test results indicated a high level of internal consistency on our data (Cronbach’s α = 0.786).

The survey contained sets of statements that were very similar, i.e. the same content, but a different embedded persuasion technique, the presentation of statements was randomised to counteract the impact of repetition of these similar statements on participants. As noted in other studies (Hassan and Barber, 2021; Unkelbach, 2007), repetition could contribute to higher ratings as repeated statements are perceived as more trustful. The persuasive statements (Table 3) have a similar length and were reviewed by two independent researchers and pre-piloted with a sample of 30 respondents before the survey was deployed. The respondents provided feedback on the clarity and intelligibility of the persuasive message. Table 3 shows the list of statements with an embedded persuasion technique, and Figure 2 presents the different phases and processes for our experiments, i.e. the steps involved in the survey development and design support the theoretical foundation of the ML models.

3.2 Machine learning modelling methodology

Multiple models were developed and tested using the same methodology. Each model uses user profiling data to predict the influence level indicated by a user for a given statement. Statements can carry the same core message but have a different embedded persuasion technique. As such, there is a model for each statement variant. The target variable (ground truth) was initially collected as values from 1 to 10 as users indicated the level of influence that they felt a statement had on them. As used in other studies (Pangbourne et al., 2020; Torgo and Gama, 1996), the effect was modelled as a binary variable indicating a participant’s level of agreement with a statement, i.e. an influence level was specified as low or high. We converted the survey influence scores into a binary variable; 1–5 coded as “0” and 6–10 coded as “1”, indicating a low and high level of agreement, respectively. Once converted, the binary target variable resulted in an unbalanced data set.

Unbalanced data distributions could significantly affect the performance of learning algorithms, equal representation of all the classes in the data set is desired (Zeng et al., 2021), and each model (statement variant) was balanced using the random under sampling technique. This decreases the number of majority class instances by randomly removing data from the original data set (Lemaître et al., 2017). This can help to mitigate bias towards the majority class which may result in poor performance on the minority class and artificially high accuracy estimates (Taghizadeh-Mehrjardi et al., 2020).

A common requirement in ML modelling is data scaling of the input variables. Some ML classifiers will not work without standardisation. In our data set, all input variables including TIPI/DAS variables, reversed-scores and non-reversed-scores were scaled. Additionally, in ML modelling is important to compare the performance of multiple ML algorithms under different pre-processing techniques. PCA is a dimension reduction tool that can dramatically impact the performance of ML models. PCA is used to condense a large set of variables to a small set that still contains most of the information in the large set (Witten and Witten, 2017).

For each of the statement variants investigated, a set of ML models were developed. Three different learning algorithms were tested, i.e. support vector machines (SVMs), gradient boosted machines and auto-sklearn. Using the full data set, training and test sets were formed by random allocation in a 70%:30% split, respectively (Figure 3). Parameter tuning was conducted on the training set to find a good set of model hyper-parameters. This was performed using 10-fold cross-validation with a randomised search of the full parameter search space. Area under the receiver operating characteristic (ROC) curve (AUC) was optimised during the cross-validation tuning process. Final models were then developed using the full training data set and the optimal parameter values discovered in the cross-validation process. These models developed on the full training data were used to make predictions on the independent test set data.

4.1 Analysis of variance

An ANOVA analysis was performed (as described in Section 3.1.2) to investigate whether there were significant differences in influence scores for each of the statement variants furnished to survey participants. Each statement variant relates to a separate ML model. Results of the one-way ANOVA test indicate that the means of the influence scores across statement variants (different embedded persuasion techniques) were significantly different at significance level p < 0.05.

The distribution of the means of the influence scores [F (11,12252) = 86.587, p = 1.028e-189] suggest that the means of the influence scores were significantly different. Hence, results from ANOVA provide evidence that survey participants perceived the language variation/linguistics style embedded in the statements (Statements – Table 3). Data was normally distributed for each group, as evaluated by Shapiro–Wilk test (w = 0.999, p = 4.28e-33). A post hoc Tukey test showed that persuasive techniques groups differed significantly at p < 0.05.

Results of the ANOVA test support the hypothesis that the means of the influence scores across statement variants (different embedded persuasion techniques) were significantly different, given that the core content was the same across statements. It is important to note that these results do not consider whether survey participants chose to give low or high scores to statements because the context domain of statements or the underlying beliefs. There is evidence to suggest that the content of the message is important in predicting user response. According to Ajzen (2002) and the theory of planned behaviour, persuasion as a process has shown to be dependent on context and on peoples' beliefs and values. Persuaders can be successful when eliciting specific salient beliefs (attitudes towards the behaviour). As a proof of concept, this study focussed in a very broad topic such as education. For our further studies, we will test an array of persuasion techniques in different communication contexts.

Figure 4 shows the mean distributions of the influence scores for each of the statement with an embedded persuasion technique.

4.2 Best models

To assess the quality of predictions on the test set, various metrics were considered to understand each model’s performance. We observed considerable variation in results across the different models i.e. statement models with different embedded persuasion techniques. While some models had an accuracy just above the random guess threshold, others reached reasonable performance (∼70% AUC), in line with or better than comparable state-of-the-art works (Wang et al., 2019; Lukin et al., 2017; Shmueli-Scheuer et al., 2019b).

The results in Table 4 show that the best SVM model (antanagoge) achieves generally higher performance than the best LGBM models (Table 5). For example, the best SVM model has an AUC, balanced accuracy, sensitivity and MCC of 0.71, 0.63, 0.72 and 0.27, respectively. Comparatively, the LGBM model (Flattery) which achieves the highest AUC of 0.66 has a balanced accuracy, sensitivity and MCC of 0.61, 0.66 and 0.22, respectively. It is notable that the more automatic approach to ML modelling (Auto-Sklearn) yields the least efficacious results in general. Interestingly, the top 5 models by AUC score for each of the 3 investigated algorithms include the statement models Flattery, Anaphora and Antanagoge. Only 3 of the 36 models outlined in Tables 4–6 use the low dimensional input data representation (i.e. the 35 DAS questions that were condensed to 7 values and the 10 TIPI questions condensed to 5 values). This may indicate that the loss of information when summing and averaging the DAS and TIPI data, respectively, may have limited the models’ ability to find a predictive pattern – given that the majority of models selected used the raw data.

Results pertaining to each ML algorithm (SVM, LGBM and Auto-Sklearn) applied to the various statement models (core message with embedded persuasion technique) are shown in Tables 3, 4 and 5. As explained in Section 3.2.1, there were 48 model combinations for each ML algorithm – here we have shown results relating to the best data set and pre-processing combination (best of four) for each persuasion technique.

4.3 Model performance at high/low influence scores

Even though our investigated models were binary classifiers, i.e. they predict a low or high influence score, the underlying discrete influence scores with values ranging 1–10 were available. This allowed us to investigate how sensitive models were at extreme scores (near 1 and 10) and at mid-range scores (around 5 and 6) where the participant’s influence level straddled the low/high binary threshold. As such, we were interested to know to what extent participant’s felt that a statement had a small, medium or strong influence on them.

Figure 5 shows the average accuracy of the 12 LGBM classifiers by influence score (a similar trend is present for SVM). The graph shows how the classifiers obtained good accuracy (approximately 75%) when the effect of persuasion was at the extremes, i.e. little to no effect and high effect (influence scores close to 1 or 10, respectively), while the classifiers performed poorly when the persuasive effect was mid-strength (influence scores around 5 and 6). For the highest-performing techniques such as antanagoge, the classifier reached an accuracy of close to 85% for extreme (high or low) influence scores – for example, this means that in most cases when the actual influence score was 1, the classifier predicted the low influence class and, alternately, when the actual influence was 10, the classifier predicted the high influence class. The results shown in Figure 5 are positive, as most of the classification mistakes were minor, occurring in situations where the user indicated a mid-strength reaction to statements/techniques, i.e. there were proportionally less examples of extreme classification mistakes.

4.4 Analysis of features importance

Feature selection is an important process that can achieve a number of objectives including the identification of a subset of variables, the reduction of noise, filtering out irrelevant features while maintaining or even improving the predictive capability of the data (Guyon and Elisseeff, 2003). Feature selection is particularly relevant to the experiment presented in this paper as implementation of an effective persuasion system on a large scale would likely only be feasible using a few highly predictive user data points, e.g. those demographic, personality or belief system questions that have the largest predictive utility. Earlier studies have suggested feature selection can be used to reduce the number of questions that users need to answer, in order to build a useful profile (Kaur et al., 2021).

There are many types of feature selection including filter, wrapper and embedded methods. Additionality, feature transformation methods such as PCA can be used to reduce the dimensionality of data – however these methods do not necessarily yield an explicit set of features and can be less valuable in understanding how features drive a model’s decision (Doherty et al., 2022). As such, in this study we have adopted a filter method (LGBM feature importance) to compare the impact of each feature – the importance scores can be used to rank each feature. For each persuasive technique we assigned an ordinal score to the features, giving a score of 6 to the most important feature, 5 to the second and so on, down to a score of 1 for the 6th most important feature for that technique. We then aggregated the scores of each feature across all the techniques.

The ten features (individual questions from the survey) with the highest importance score were the demographics questions (gender and age), the TIPI questions 5 (Openness), 1 (Extroversion) and 4 (Emotional stability) and the DAS questions 18 (Perfectionism), 22 and 24 (Entitlement), 11 (Achievement) and 26 (Omnipotence). These findings confirm that demographic information (gender and age) can have a strong effect on predictions, in line with previous research (Kaptein, 2015). Interestingly, the results show a robust effect of DAS in predictions, providing support for the hypothesis that negative and positive emotions of an individual are key factors of the influence process (as explained in Section 1). Figure 6 shows the importance score by psychological dimension and that many DAS components outperformed the TIPI ones, providing evidence that the information collected by the DAS scale did add value to the predictions. We then grouped the importance scores of each question by the 12 psychological dimensions (5 from TIPI and 7 from DAS). For instance, the Openness score was the average of the score of the two individual TIPI questions 5 and 10. TIPI components are shown in dark grey and DAS components in light grey (see Figure 6).

4.5 Model performance with and without dysfunctional attitude scale information

The AUC averaged over all statement models (i.e. the 12 persuasion technique models) is shown in Figure 7 for both models that used only demographic and TIPI data versus models that used demographic, TIPI and DAS data.