The Islamophobia Index: Exploring the Challenges in Establishing Reliability for a Content Analysis Instrument Evaluating Islamophobia in Media Texts

Mujib  Abid

In a wide variety of studies over the past two decades, the media has consistently been identified as a significant social institution implicated in normalizing and disseminating anti-Muslim prejudice. Identifying and combating Islamophobic discourses has been a challenging proposition, however, due to difficulties in systematically identifying and evaluating “Islamophobia” within texts. Islamophobia is a complex and contested phenomenon that defies easy classification within the boundaries of terminology which would normally be employed to describe hostility or prejudice based on, for example, race or religion. This article outlines some of the challenges involved in defining and categorizing Islamophobic discourses through an exploration of the process of constructing, evaluating, and applying a unique content analysis instrument, the “Islamophobia Index,” to media texts, using data sets drawn from the Australian news media. We critically reflect upon the methodological limitations of systematic quantitative studies, the importance of qualitative and interpretive approaches that take into account researcher subjectivities, and ultimately reposition and repurpose this project as a mixed-method study. In conclusion, potential applications for the Index, including non-media textual analysis, are considered.

182 ISJ 6(2) The Islamophobia Index: Exploring the Challenges in Establishing Reliability for a Content Analysis Instrument Evaluating Islamophobia in Media Texts Leticia Anderson, Shima Shahbazi and Mujib Abid Southern Cross University ISLAMOPHOBIA STUDIES JOURNAL VOLUME 6, NO. 2 Fall 2021, PP. 182–206. Published by: Islamophobia Research and Documentation Project, Center for Race and Gender, University of California, Berkeley Disclaimer: Statements of fact and opinion in the articles, notes, perspectives, and so on in the Islamophobia Studies Journal are those of the respective authors and contributors. They are not the expression of the editorial or advisory board and staff. No representation, either expressed or implied, is made of the accuracy of the material in this journal, and ISJ cannot accept any legal responsibility or liability for any errors or omissions that may be made. The reader must make his or her own evaluation of the accuracy and appropriateness of those materials. 182 183 The Islamophobia Index: Exploring the Challenges in Establishing Reliability for a Content Analysis Instrument Evaluating Islamophobia in Media Texts Leticia Anderson, Shima Shahbazi and Mujib Abid Southern Cross University In a wide variety of studies over the past two decades, the media has consistently been identified as a significant social institution implicated in normalizing and disseminating anti-Muslim prejudice. Identifying and combating Islamophobic discourses has been a challenging proposition, however, due to difficulties in systematically identifying and evaluating “Islamophobia” within texts. Islamophobia is a complex and contested phenomenon that defies easy classification within the boundaries of terminology which would normally be employed to describe hostility or prejudice based on, for example, race or religion. This article outlines some of the challenges involved in defining and categorizing Islamophobic discourses through an exploration of the process of constructing, evaluating, and applying a unique content analysis instrument, the “Islamophobia Index,” to media texts, using data sets drawn from the Australian news media. We critically reflect upon the methodological limitations of systematic quantitative studies, the importance of qualitative and interpretive approaches that take into account researcher subjectivities, and ultimately reposition and repurpose this project as a mixed-method study. In conclusion, potential applications for the Index, including non-media textual analysis, are considered. INTRODUCTION Islamophobia has emerged in the past two decades as a serious social problem within contemporary Western societies, marked by increasing anti-Muslim racialization, prejudice, and discrimination impacting negatively upon Muslim individuals and communities. During this period, the media has consistently been identified as a significant social institution implicated in normalizing and disseminating Islamophobic discourses (Ahmed and Matthes 2017; Akbarzadeh and Smith 2005; Dabashi 2011; Duderija and Rane 2019; Poynting and Perry 2007; Rane, Ewart, and Martinkus 2014). Systematically identifying and combating Islamophobic discourses within the news media has been a challenging proposition, however. This is partially due to the challenges in developing standardized frameworks for evaluating and comparing “Islamophobia” within texts, as it is difficult to classify the phenomenon within the boundaries of terminology that would normally be employed to describe hostility or prejudice based on, for example, race or religion. In this article, we report upon the results of an attempt to assess the use of an index for evaluating and comparing Islamophobia within texts, with the main intended genre of analysis being news media texts. This process represented an opportunity to evaluate and improve the coding criteria and identify any issues within our methodology. Through our discussion we also illustrate in detail the process by which we evaluated and coded each article. Although the project initially focused on quantitative methods, including statistical analysis, challenges in establishing reliability and validity of the Index led to an extended process of refining the 184 ISJ 6(2) instrument, ultimately leading to a reflective, qualitative review of the process on the part of the researchers. We position this aspect of our study as indicating the centrality of subjectivities in academic engagement with a charged and lived phenomenon such as Islamophobia. We set out to offer an account of the mediatization (and over-mediatization) of Islamophobia and its impacts before critically examining existing literature on conceptualizations of Islamophobia. We next offer a report on the development and utilization of the Islamophobia Index, and analysis of challenges encountered in this process. Finally, we expound upon our results, utilizing a mixed-methods approach that combines our systematic quantitative approach and the subsequent qualitative review of this process to critically reflect upon our research process and build a case for the utility of this method. THE IMPACTS OF MEDIATIZED ISLAMOPHOBIA There is presently a burgeoning body of research documenting the serious negative impacts of Islamophobia. These find expression in myriad ways, with research to date primarily documenting the experiences of Muslim minority communities in the Western world, including Australia, the geographical locus for this particular study (Rane, Ewart, and Martinkus 2014). Research on manifestations and experiences of Islamophobia in non-Western countries is still an emerging field of research (Ahmed and Matthes 2017; Osman 2017), as we shall explore further in our discussion. Islamophobic abuse and attacks directed at Muslims or those perceived to be Muslims in Western nations have risen steeply in the past two decades and impact many individuals and communities on a regular or ongoing basis, although media reporting has also begun to focus more on recent mass Islamophobic attacks (e.g., the 2017 Finsbury Park, 2017 Quebec City, and 2019 Christchurch tragedies). Among those who are most often exposed to Islamophobic attacks, including verbal and physical abuse, are women— often accompanied by their children (Abu-Ras, Suárez, and Abu-Bader 2018; Allen 2014; APPG 2018; Hopkins 2016; Iner 2017). Due to their hypervisibility, Muslim women of color experience Islamophobia at the intersection of their gender, race, and faith, which adds other layers to the forms of oppression targeted at them. The negative impacts of the hyper-surveillance and securitization of Muslims in Western societies that gained momentum through the War on Terror and continue into the present are an emerging focus in recent academic research (AbdelFattah 2019; Brown 2018; Dabashi 2011; Fox and Akbaba 2015; Morsi 2017; Selod 2018). The nexus between exclusionary, anti-Muslim political discourses and their amplification through the mainstream media marks a critical juncture for the exploration and problematization of anti-Muslim prejudice. Islamophobic discourses in Western media can have direct negative impacts upon sense of belonging, safety, psychological health, and wellbeing for Muslims (Brown, Brown, and Richards 2015; Bull and Rane 2018; Hargreaves 2016b; Kunst, Sam, and Ulleberg 2013; Kunst et al. 2012; Samari, Alcalá, and Sharif 2018). In one study conducted within different European contexts, simply encountering Islamophobic discourses in the media was correlated with high levels of psychological distress for Muslim participants: “perceptions of belonging to a group that is feared in society has itself a distinct effect on Muslim minorities’ health and identification, regardless of whether individuals personally experience discrimination in their daily lives or not” (Kunst, Sam, and Ulleberg 2013, 235). Recent research in the Australian context by Nahid Kabir (2019) indicates that Islamophobic media representations can also contribute to increased marginalization of vulnerable young people and influence their engagement with extremist propaganda. The widespread implication of mainstream media sources in reinforcing, perpetuating, and disseminating a range of negative stereotypes about 185 Islam and Muslims is amply documented within the literature (Ahmed and Matthes 2017; Duderija and Rane 2019; Esposito and Iner 2019; Ewart and O’Donnell 2018; Poynting and Perry 2007). As Brown, Brown, and Richards (2015) highlighted in their study of the negative impacts of mediatized Islamophobic discourses upon international students in the UK, “mental health and wellbeing can only flourish in a context of basic civil, political, socio-economic and cultural rights,” yet this possibility is undermined by the experience of constant exposure to mediatized Islamophobia (51). In addition to the direct harms caused by this exposure, the role that negative media representations play in the normalization and dissemination of Islamophobic discourses is of particular concern. The impact of negative media representations in increasing Islamophobic attitudes and actions is difficult to quantify; as O’Donnell, Ewart, and Walding (2018) note, “the extent to which the mainstream news media acts to influence the opinions of various publics is contested terrain that continues to develop theoretically” (5). Reception studies of the impact of negative news discourses on anti-Muslim beliefs and activities therefore remains an important direction for future research in this field. O’Donnell, Ewart, and Walding have highlighted that there are limited large scale media effects studies on attitudes toward Islam and Muslims with a few exceptions, such as Shaver et al.’s (2017) analysis of New Zealand social attitudes data. This study found a reliable association between increased media exposure and higher levels of anti-Muslim prejudice. Although researchers in this study found this effect was consistent for respondents from across the political spectrum, and O’Donnell, Ewart and Walding found that the media served as the main source of information on Muslims for only half of a representative sample of Australians, other evidence suggests the possibility that the impacts of Islamophobic media representations may be particularly salient and significant for individuals who already hold prejudice toward Muslims and Islam. Randa Abdel-Fattah (2018), for example, in her ethnographic research in Australia, notes “the pervasive and relentless impact of political and media discourse in training people to be affected by Muslim bodies” (144), among committed Islamophobes and “everyday” Australians (i.e., people who did not self-identify as Islamophobic) that she interviewed (179). Given the demonstrated impacts of Islamophobia, and the implication of the mainstream media with its evolution and impact, further research on Islamophobic representations in the media is certainly warranted. Nevertheless, the field has now grown in such a way that critical reflection on how this should most effectively and appropriately be conducted is necessary. The next section therefore highlights the challenges in framing investigations of Islamophobia through consideration of some of the ongoing challenges over definition and operationalization of the concept of Islamophobia in academic research, with a focus on media research. DEFINING AND OPERATIONALIZING THE CONCEPT OF “ISLAMOPHOBIA” Research and debate on Islamophobia has grown significantly in the past two decades “not only for political reasons, but also because it attempts to label a social reality—that Islam and Muslims have emerged as objects of aversion, fear, and hostility in contemporary liberal democracies” (Bleich 2012, 181), yet “Islamophobia” remains a contested term. Rather than simply a “psychological fear of Islam,” the term is generally utilized to describe “the practical hatred of Muslims” and manifestations of negative affect, including prejudice and discrimination (Hargreaves 2016a, 14). Consensus appears to be emerging that despite continued contestation over the term, since “Islamophobia” has achieved widespread social and academic currency, working on refining definitions and understanding of what “Islamophobia” signifies and means in various settings is likely to be more of a focus for future studies in this area (Lean 2019; Love 2017). 186 ISJ 6(2) When we move from considering contestation over the term itself to its definition, we similarly find widely divergent understandings of how best to define Islamophobia. As Love (2017) emphasizes, Islamophobia is a “complex problem that involves a wide range of social issues . . . religion, gender, class, sexuality – all these and more play crucial roles in the reproduction of Islamophobia” (18). One of the earliest and most influential definitions of Islamophobia was developed by the Runnymede Trust, a prominent British race equality think-tank. Their landmark 1997 report defined Islamophobia as “an unfounded hostility toward Islam, and therefore fear or dislike of all or most Muslims” (Runnymede Trust 1997, 4). This definition, and variants of it, have tended to be somewhat constrained by the focus on “fear” or “phobia” as the defining element of expressions of Islamophobia, and have not necessarily kept pace with emerging expressions of what we can broadly term Islamophobia; in particular the increasingly accepted insistence by critical scholars on the need to understand Islamophobia as a racialized form of discrimination and prejudice (Cainkar and Selod 2018; Meer and Modood 2019). Nevertheless, as Julian Hargreaves (2016a) has noted in his review of efforts to define Islamophobia, the Runnymede definition of Islamophobia has been by far the most heavily utilized definition within academic research as well as popular debate. We suggest that one of the reasons for the longevity of the Runnymede definition is its accompaniment by eight statements about different categories of “closed” views toward Islam which expanded significantly upon the brief outline of Islamophobia given in the formal definition (Runnymede Trust 1997, 5–11). These statements go some way to providing a more detailed framework for understanding and identifying Islamophobia, despite some issues with the characterization of “closed” versus “open” views toward Islam. These statements each describe various characteristics of Islamophobia, including “key attributes of the concept and, in particular, the perceptions of Islam and Muslims that underpin the types of anti-Islamic and anti-Muslim sentiment purported to cause negative outcomes for Muslim communities” (Hargreaves 2016a, 13). As Hargreaves (2016a) observed in his survey, this framework has generated significant attention from scholars, including attempts at adaptation into research projects, suggested modifications, and as a provocation for further constructive critique. This is despite the fact that what he calls “the eight-point list” uses terms that “are almost entirely abstract and do not provide researchers the practical tools needed to measure the extent of ‘Islamophobia’” (14). Given such acknowledgments of the limitations of the Runnymede framework, somewhat surprisingly, competing definitions have not gained predominance. In some cases this is due to the unwieldiness of alternatives and comparable difficulties in operationalizing them for the purposes of systematic research, as per Chris Allen’s thoughtful and pioneering yet very lengthy definition of Islamophobia (Allen 2010, 190). Researchers have therefore repeatedly returned to the Runnymede definition and framework, in particular noting the potential utility of “the eight-point list” in capturing key elements of the evolving phenomenon of Islamophobia across the past two turbulent decades. Abdalla, Ewart, and Rane (2010), for example, emphasized that “any genuine attempt by the media or journalists to challenge and eliminate prejudice in reporting against Islam and Muslims would require an appreciation of these identifiers [of Islamophobia]” (235). Significantly, the Runnymede Trust has recently moved to update its definition to emphasize the processes of the racialization of Muslims as a core element of contemporary Islamophobia. In response to criticisms of the elision of race within the emphasis on “fear” in Runnymede’s original definition of Islamophobia (see Cainkar and Selod 2018), on the twentieth anniversary of the landmark report’s release, Runnymede redefined Islamophobia as “antiMuslim racism” (Runnymede Trust 2016). In altering the definition of Islamophobia, Runnymede emphasized the continued value of the descriptive framework of Islam which accompanied their definition: 187 The 1997 Islamophobia report more or less got most things right. In the 1997 report, Runnymede offered two analytic frameworks. The first contrasted “open” and “closed” views of Islam, with eight distinctions clarifying the differences. These included monolithic vs. diverse, separate vs. interacting, enemy vs. partner, manipulative vs. sincere and discrimination defended vs. discrimination criticized. On balance, this framework holds up very well as an analysis of views of Islam today. (Runnymede Trust 2016) This is where our research comes in. Although we welcome the redefinition of Islamophobia as outlined above, we also still see value in the broad Runnymede framework, and consider that it provides sufficient scope for extending understandings of Islamophobia to accommodate an appreciation of the processes of racialization and racism inherent in this phenomenon. We maintain there is a need to differentiate between racism and racialization; that is, an understanding of how racialization works as an institutional practice in activating latent Islamophobia through systematic exclusionary practices ought to be accounted for within a definition of Islamophobia as well as prejudicial and discriminatory attitudes and actions that are directed toward individuals or groups that are identified (correctly or erroneously) as “Muslim” via phenotypical and cultural markers. In this article, we posit that it is possible to develop ways to work with the eightpoint framework for the purposes of conducting media analysis of Islamophobia in ways that enable grappling effectively with racism and racialization—although our research also highlights important limitations. The original Runnymede framework did include short examples and a multi-paragraph discussion on each of the eight distinctions between open and closed views toward Islam (Runnymede Trust 1997, 5–11). However, these examples are explained in a brief conversational style and are highly specific in terms of references to the late 1990s UK context. They are not especially helpful, for example, for researchers seeking to operationalize the framework for analysis of media coverage from more recent years, or for different regional contexts, when the terms of the debate have evolved significantly. Nevertheless, the longevity of the eight key indicators themselves, remarkably, still persists. In responding to the call for the formulation of more rigorous methodological approaches to the study of Islamophobia (Bleich 2012, 46), this study therefore considers approaches to updating and developing the Runnymede eight-point framework for contemporary analysis of Islamophobic representations. As part of our process of investigating whether it might be possible to develop “a meaningful aggregate index of Islamophobia” for media analysis (Bleich 2012, 46), we decided that a promising example that warranted further investigation was Anderson’s (2015a) “Islamophobia Index.” This study utilized the Runnymede definition and eight-point list to construct an aggregate Index which was used to evaluate Islamophobic discourses in Australian Federal Election reporting during the mid-late 2000s. Since the turn of the century, Australia has been the site of high levels of social contestation in regard to existing and potential Muslim minorities, and has also been the locus for a significant volume of research on Islamophobia (Ahmed and Matthes 2017). As an Anglophone, Western-oriented, and white dominant society located in the Asia-Pacific region, Australia continues to grapple with the legacies of colonialism, the increasingly diverse demographics of the country, and relationships with regional neighbors (Paul 2014). The ambivalence of Australia’s engagement with Western and non-Western cultures and politics—paradoxically invested in yet distinct from both “the West” and Asia—presents a worthwhile case study to examine further in regard to Islamophobia studies. 188 ISJ 6(2) THE ISLAMOPHOBIA INDEX Anderson (2015a) compared articles published in two high circulation Australian broadsheet newspapers in the month prior to the 2004 and 2007 Federal elections, respectively. The eight indicators of “closed” and “open” views toward Islam featured in the Runnymede definition of Islamophobia were used to elicit which frames dominated the narrative within specific news articles (n = 373). Each article was scored as open, closed, or neutral on each of the eight frame descriptors (monolithic/diverse; separate/interacting; inferior/different; enemy/ partner; manipulative/sincere; Western actions defended/criticized; Discrimination defended/ criticized; Islamophobia normalized/problematized—see Appendix A). The scores on each of these eight dimensions were then aggregated to produce an index of the level of Islamophobia within each article, which enabled comparison of patterns in representation across time and different publications through supplementation with additional content and critical discourse analysis upon selected subsets of the data. The findings reported in this article indicated that Anderson’s Islamophobia Index could be effectively applied as an evaluative criteria across relatively large datasets “to highlight and pinpoint significant shifts in representational patterns,” namely a significant decrease in Islamophobic reporting in the 2007 election campaign compared to the 2004 election campaign, and publication-specific differences in Islamophobic discourses (Anderson 2015a, 262). However, intercoder reliability was not reported, and applications other than media content and discourse analysis were not explored. Although the results from this study were promising, in order to further develop this instrument it would be imperative to first test the reliability of the Index in terms of reproducibility: “in content analysis, reproducibility is arguably the most important interpretation of reliability” (Krippendorff 2004, 414). This current article therefore reports on an attempt to extend Anderson’s (2015a) research through exploring the possibility of developing the Islamophobia Index into a reliable measure. Ultimately, we also wished to explore whether it could be utilized by researchers from different disciplinary contexts to evaluate Islamophobia in a range of texts, not just media texts, but the initial phase of this project was focused on testing the reliability of the Index before exploring other applications. We now turn to explore the methodology for this project in detail. METHODOLOGY The aim of this study was to test whether an acceptable level of intercoder reliability rating and internal consistency could be achieved using the Islamophobia Index (Anderson 2015a). The key to establishing the reliability of a coding instrument is to be able to demonstrate acceptable levels of agreement between independent researchers using that instrument: “the more observers agree on the data they generate, and the larger the sample of units they describe, the more comfortable we can be that their data are exchangeable with data provided by other sets of observers, reproducible, and trustworthy” (Hayes and Krippendorff 2007, 78). If statistical analysis confirmed that different coders could apply the evaluative criteria with acceptable levels of agreement on given texts, then we could be more confident that the Index had satisfactory statistical reliability as a measure of Islamophobia within data sets. We determined that the most appropriate approach to conduct reliability testing of the Index was to develop comprehensive coding instructions, recruit multiple coders, train the coders, and then statistically analyze their coding of a set sample of texts. As texts typically support multiple interpretations, thorough preparation for coding assists in ensuring that content analysis will be focused effectively on the phenomena we expect it to focus on and that results are 189 more likely to be reproducible by other researchers at different points in time and different locales (Krippendorff 2009). To evaluate whether we had been effective in this endeavor, we determined that we would conduct analysis of coding responses in Statistical Package for Social Sciences (SPSS), including testing pairwise correlations between coders, and testing for internal consistency on each dimension of the Index. We would also be testing the reliability of coding across the full set of coders. Although there is a wide range of indexes of reliability, we chose to use Krippendorff’s alpha (α) as it was designed specifically for evaluation of communications studies content analytical data: it “generalizes across scales of measurement; can be used with any number of observers, with or without missing data; and it satisfies all of the important criteria for a good measure of reliability” (Hayes and Krippendorff 2007, 78). Sample Selection The data set analyzed in this article was compiled through Factiva online archive searches on articles published in two high circulation Australian broadsheet newspapers, The Sydney Morning Herald (SMH) and The Australian, during a time period of one week prior to the 2004 and the 2007 Australian Federal Elections, inclusive of the day of the election (exact time frames October 2–9, 2004, and November 18–24, 2007), following Anderson’s (2015a) rationale for the selection of this specific sample.1 Articles from these publications during these time periods were screened on the basis of their inclusion of the keywords “Islam OR Islamist OR Islamism OR Muslim OR Muslims” and a further screening ensured the article was substantively about Islam or Muslims, following Richardson’s conventions for sample selection in analysis of representations of Muslims in the media (Richardson 2006). This process resulted in a total of 94 articles (8—SMH 2007; 21—SMH 2007; 26—The Australian 2004; 39—The Australian 2007). In order to create a subset of articles to conduct a comparative coding exercise on, as representative a selection of articles as possible was chosen from across the two different publications and two different pre-election periods. Through a combination of publication date and subject matter seven articles were selected from each data set, giving a total of 28 articles for analysis (7—SMH 2007; 7—SMH 2007; 7—The Australian 2004; 7—The Australian 2007). In selecting our data set, we attempted to include at least one article per day from each publication within the seven day time periods. Where there were no articles published on a particular day, an additional article was selected from the next day. The articles were predominantly traditional “hard news” stories, but also included one Feature article, one article from the World section and one from the Local news section for each data subset. Where possible, articles that addressed similar subject matter in different publications in the same election period were selected. For example, one article on the EU’s reservations toward accepting Turkey as a member state was chosen from each of the two publications in the 2004 period, and one article from each of the two publications on a racist election leafleting scandal in Sydney from the 2007 data sets, in order to permit some preliminary comparative comments to be made on the representation of the same issue in the different sources. Coding Process In order to evaluate the reliability of the Islamophobic Index, multiple coders from different disciplinary backgrounds were recruited to utilize the Index to evaluate the set of sample articles.2 Coders were provided with explicit coding instructions (including a detailed definition 190 ISJ 6(2) of Islamophobia, detailed instructions on how to code the articles, and a sample pre-coded article), a codebook (Excel spreadsheet), and a set of the 28 deidentified news articles (with publication, date and byline details all removed). Coders were trained in a one hour face-to-face meeting where the coding instructions were methodically explained, and the sample pre-coded article was discussed in detail. Coders were instructed to independently code each article according to the Islamophobic Index based on the eight-point list described above (referred to in the coding instructions as “frame descriptors”). Coders were directed to allocate each article a score of –1 (closed), 0 (neutral) or +1 (open) for each frame descriptor (see Appendix A). For example, a news story that directly referred to Islam as a “barbaric” religion would be scored with –1 on the third frame (Islam represented as “inferior”). If the article also indicated that Islam was presented as a genuine religious faith rather than a political ideology, it would be scored with +1 on the 5th frame (Islam represented as a “sincere” faith). If the article did not explicitly address one or more of the frames, or contained a relatively balanced mix of open and closed views for a particular frame, then coders were instructed to score the article as 0 (neutral) on that dimension. Following Anderson (2015), the scores across the eight dimensions for each unit of analysis which each coder entered in their codebook were then aggregated automatically into a single “summed score” for each article. Summed scores ranged from –8 through 0 to 8. For the purpose of this study, articles were categorized on the basis of their summed/aggregate score as being predominantly characterized by “closed” views toward Islam and Muslims if they received an aggregate score of –8 > –3; “neutral” if they received an aggregate score between –2 > 2, and predominantly characterized by “open” views toward Islam if they received an aggregate score of 3 > 8.3 We defined “agreement” between different coders as when the aggregate scores for individual units of analysis (i.e., each news article) indicated that coders independently concurred that the unit should be classified overall as “open,” “neutral,” or “closed” in terms of its patterns of representations of Islam and Muslims. This gave some flexibility for individual differences of interpretation on the disparate eight dimensions of the Index, yet gave firm boundaries on how to determine if an article was overall characterized more by a prevalence of open, closed, or neutral views. Since quality coding instructions are critical to establishing reliability for an instrument, following our initial coding exercise with multiple coders, the first author undertook informal discussions with coders regarding their results. As Krippendorff (2009) emphasized, the development of quality coding instructions typically undergoes several iterations until the instructions are reliable (351), so following preliminary data analysis and collation of feedback from coders we proceeded to development of a second iteration of the coding instructions. Based on verbal feedback from coders about how best to improve the coding instructions, the redevelopment of the coding instructions by the authors of this article focused on providing sets of relevant and timely examples for each of the eight frame descriptors (Appendix B). We then conducted a limited retesting of agreement between three coders using the expanded coding instructions to recode the same set of articles. We now report on the results and key findings from our study. RESULTS The data was analyzed using the Statistical Package for Social Sciences (version 24.0). Since Krippendorff’s alpha is not calculable in SSPS, a KALPHA macro was utilized to assist us in completing our analysis (Freelon 2017). Discriminant validity between all coders was checked via a factor correlation matrix. Correlation values between all the coders were below 191 Table 1 Correlation among six coders Coder 1 Coder 1 Coder 2 Coder 3 Coder 4 Coder 5 Coder 6 Coder 2 Coder 3 Coder 4 Coder 5 Coder 6 1 .460** .516** .407** .128 .208** 1 .355** .300** .176** .288** 1 .280** .253** .193** 1 .057 .123 1 .226** 1 **Correlation is significant at the 0.01 level (2-tailed) 0.7, indicating satisfactory discriminant validity (more than 0.7 specifies a common variance). The level of covariance with a 99% confidence interval between coders was, however, highly variable. The Pearson correlation analysis shows that all coders were significantly and positively correlated with each other except Coders 1 and 5, Coders 4 and 5, and Coders 5 and 6, respectively. In terms of higher variance, Coder 1 and 3 (r = 0.516, p < 0.001), Coder 1 and 2 (r = 0.460, p < 0.001), and Coder 1 and 4 (r = 0.407, p < 0.001) were more strongly correlated than Coder 2 and 3 (r = 0.355, p < 0.001), Coder 2 and 4 (r = 0.300, p < 0.001) and Coder 2 and 6 (r = 0.288, p < 0.001) (see Table 1). The overall Cronbach’s alpha coefficient for all six coders was 0.77. This implies that the internal consistency of all the coders in the initial phase of testing was acceptable, almost reaching the level of “good reliability,” which is 0.800 (Manning and Munro 2007). Also, for “tentative conclusions,” Krippendorff advises α ≥ 0.667 as the minimal acceptable level of reliability (Krippendorff 2009, 354). Cronbach’s alpha is appropriate as a measure of the reliability of an aggregate measure across coders/observers (Hayes and Krippendorff 2007) and indicates that the Index was effectively evaluating the construct of “Islamophobia” in given texts. Krippendorff’s alpha test was also used (Hayes and Krippendorff, 2007) to estimate the intercoder reliability; however, the low alpha value (α = 0.3491) indicated that there was poor agreement between the six coders. We therefore had some mixed evidence toward establishing the validity and reliability of the Index: “a measuring instrument is said to have validity if it measures what it purports to measure; it is said to have reliability if it gives the same results consistently” (Janis 2009, 358). The results from this initial round of testing were encouraging but indicated that further iterations of the development of the instrument would be required to achieve more robust evidence of validity and reliability. When we considered the data in-depth, an interesting pattern emerged. The relationships between results for some coders were clearly much stronger than for others. When we compared the correlation of Summed Scores only for Coders 1, 2, and 3, Coders 1 and 2 had the highest level of covariance with a 99% confidence interval (r = 0.742, p < 0.001). After that, the next strongest relationship was found between the scores of Coder 1 and Coder 2 with a 95% confidence interval (r = 0.465, p < 0.001) followed by Coder 2 and 3 with a 95% confidence interval (r = 0.410, p < 0.001) (Table 2). In addition, the Krippendorff’s Alpha Reliability Estimate between Coder 1 and Coder 2, for example, was much higher than for other coders (α = 0.701), approaching the 0.800 threshold that would indicate good intercoder reliability. When we reviewed the disciplinary background of Coders 1, 2, and 3 and their relationships with each other, it became apparent that the more central the study of Islam and Middle Eastern culture and politics to the coder’s discipline and the more time the coders had spent informally discussing Islamophobia and the coding instructions, the higher the correlation and inter-rater reliability between their coding. 192 ISJ 6(2) Table 2 Cronbach alpha and correlation matrix among three coders Coder 1 Coder 2 Coder 3 α Coder 1 .602 .609 .851 1 .742** .440* Coder 2 Coder 3 1 .431* 1 **Correlation is significant at the 0.01 level, *Correlation is significant at the 0.05 level The three coders for whom the highest pairwise correlations and inter-rating reliabilities were recorded (the authors of this article) then worked collaboratively on redeveloping the coding instructions, including developing specific examples of each category, then recoded the articles. We do not report statistical analysis results from this round, because the observers were not coding independently due to discussing the coding instructions in-depth together and had also been through a previous attempt at coding the articles. What we found, however, was that the process of working on revising the coding instructions and recoding the articles raised a range of new questions and issues for our research, potentially with implications for other researchers in this field. In particular, we found that our experiences highlighted the importance of conversation/dialogue and privilege/positionality in exploring and defining a concept as contested as Islamophobia. We therefore faced difficult decisions regarding the next steps for our research. One route for further research was to attempt to replicate the study with new coders (and potentially a new sample of texts), using the redeveloped and expanded coding instructions. In this way we could work toward more robust statistical evidence for the validity and reliability of the Islamophobia Index. While this remains a potential future direction for our research, the qualitative aspects of our findings, which pointed at and underscored the complexities of coming to a common agreement on definitions and understandings of Islamophobia, have emerged for us as equally important issues which require deeper examination. We therefore turn now to explore these issues in further detail. DISCUSSION In discussing our development of the detailed examples for the revised coding instructions, we will refer in detail to the comparison of one paired sets of articles from the data set and to the recoding of these articles in the second phase of the study. Within the same time period, clear divergences of the treatment of the same issue in the different newspapers were revealed through this analysis. During the 2004 period, for example, the data set included one article from The Australian and one from The SMH on Turkey’s European Union bid. “EU puts Turks on a 10-year Time Line,” published in The Australian, was rated as “closed” (scores in the recode –4, –4, and –3) (Rothwell 2004), while “Europe Cautious in Talking Turkey,” published in The SMH was rated as “neutral” (scores in the recode of 1, 1, and 0) (Fray 2004). This is a quite substantial difference between two articles on the same subject which were published just one day apart. We describe in some detail below the reason behind the differential scores in the two articles, to illustrate the coding method as well as explaining some of the distinctive features of differential approaches to the same news “story.” The first article, by The Australian’s Middle East correspondent Nicholas Rothwell (2004), is on the surface a fairly balanced, neutrally worded article. Rothwell faithfully recorded a range of objections to Turkey’s EU membership without using heated, inflammatory language; 193 he also notes dispassionately that “there are some strong pro-Turkey EU members,” namely the UK. But at no point did Rothwell question the Islamophobic and xenophobic attitudes which underpinned much of the antagonism at that time to Turkey’s membership bid. Rothwell justly mentions Turkey’s poor human rights record and continued abuses in the policing system as obstacles to EU membership. However, when he describes “widespread concern” over “the growing Islamic minorities [in Europe] and the threat of terrorism,” as well as the potential for “a vast flow of migrant workers into Western Europe,” Rothwell did not provide any critical analysis of whether such concerns are well-founded. By not critiquing objections to Turkey’s membership in the EU on the grounds of anti-Muslim fears, Rothwell subtly implied that such concerns were legitimate. Rothwell also records the unprecedented provision of permitting the EU to close its borders to Turkey at any time, if Turkey’s bid was successful, without critically reflecting upon the xenophobia and double standards underlying such a proposal. Rothwell himself refers to Turkey as “the Muslim nation,” reinforcing the idea that Islam is monolithic and unchanging, and in the context potentially primitive and aggressive. His listing of objections to the membership of Turkey also reinforces this idea by implying that Muslim terrorists, Muslim minorities in Europe, and Muslim nations are all one within this monolithic entity. Rothwell signally failed to critique the role of Islamophobia in the furor; by tacitly accepting the assumption that there was some kind of unbridgeable gap between “the West” and Turkey, opposition to Turkey’s EU membership on the basis of its “Muslimness” was legitimized and normalized. In terms of the detailed coding in the recoding exercise for this article, unanimous negative scores on Frames 4 and 7 and largely negative scores on several other Frames were not counterbalanced sufficiently by neutral coding on Frames 6 and 8, resulting in a final aggregate score of “closed” for this particular article. By contrast, the second article, penned by The SMH’s London correspondent Peter Fray (2004), openly acknowledged and challenged some of the attitudes underlying opposition to the EU’s bid. Fray wrote that “fears are widespread across Europe and run the gamut from blatant Islamophobia to legitimate issues about human rights.” Fray discussed objections to Turkey’s membership under four headings: financial, human rights, employment, and religion. Fray went through these objections in turn and weighed up to what extent the concerns were valid, concluding that the financial and human rights objections were significant and justified and that these concerns would theoretically have been able to be addressed during the ten-year time-frame that was set at the time to manage Turkey’s bid. Fray noted in regard to “employment,” however, that the EU’s suggested provision for blocking migration from Turkey raised concerns that Turkey was only being offered “second-class membership,” and moreover, that the proposal appeared to “breach the EU’s own principle of free movement of labor.” On the topic of religion, Fray noted “growing fears that Europe, especially France, is being overrun by what many consider an alien culture.” Nevertheless, Fray emphasized, “the prospect of Turkey joining the EU one day remains a potentially powerful symbol . . . the country’s geographical location, history and culture make it the perfect bridge between East and West.” This vision of the symbolic value of admitting Turkey to the EU, if the Turkish government could address the legitimate objections to its admission, is a long way from Rothwell’s mechanical and unchallenging recitation of both legitimate and more problematic objections to Turkey’s membership bid. Yet, the story still fell into the use of tropes such as the dichotimization of “East and West” and uncritical use of loaded terms such as “alien.” In terms of the detailed coding for this article, positive scores on Frames 5–7 balanced out the largely negative scores on Frames 1–3, resulting in a final aggregate score of “neutral” for this article. 194 ISJ 6(2) Issues in Redeveloping the Descriptions of the Coding Instructions In developing detailed examples of each of the frame descriptors as the basis for the recoding of these articles, we were effectively engaging in producing an updated description of the Runnymede eight-point list, which, as noted above, is highly specific in terms of the 1990s time period and UK-focused context. At times, agreement between Coders 1, 2, and 3 on examples of “closed” or “open” views toward Islam and Muslims for some of the frames was readily apparent, and it was relatively straightforward to write examples that were suitable for the texts we were analyzing. For Frame 1 (monolithic/diverse), for example, we were able to quickly come to a consensus on the wording for examples of “closed” views, including: diverse communities or states referred to reductively and defined solely through reference to the Muslim aspects of their identity, such as “the Muslim nation Turkey,” and “radical or extremist groups implied as representing or collectively endorsed by all or most Muslims.” The equivalent examples of “open” views on this frame we worded as: “emphasis on diversity of beliefs, ethnicity, language and other markers of cultural difference within Muslim communities and states” and “clear distinction made between the beliefs and actions of some individuals or organizations and that of the majority of global Muslim communities” (see Appendix B). Other examples were more challenging and required lengthy discussions between researchers on this project, sometimes partly as a result of problems inherent within the original Runnymede descriptions of some of the frames. The brief Runnymede (1997) description of Frame 1 (monolithic/diverse), for example, described “open” views on this frame as: “Islam represented as diverse and progressive, with internal differences, debates and development.” When trying to develop examples of each of these descriptors, we had to come to terms with the issues inherent in describing “open” views toward Islam as predominantly being reducible to capturing a description of “progressive” Islam. Similarly, the Runnymede description of Frame 3 (inferior/different) included the description of “Islam represented as inferior to the West—barbaric, irrational, primitive, sexist” (Runnymede Trust 1997, 5). In defining “open” views on this frame, how then could we avoid mirroring problematic terminology inferring cultural inferiority through references to “rationality”? As Abdel-Fattah and Krayem have demonstrated, “the war on terror has split the racialized category of ‘Muslims’ into ‘good’ and ‘bad’ or ‘moderate/mainstream’ and ‘radical’ . . . the ‘radical’ personality is demonized, marginalized and pitted against the ‘moderate’ and ‘progressive’ personality” (Abdel-Fattah and Krayem 2018, 434). Morsi (2017) similarly highlights the way that the moderate/radical binary of Muslimhood in the Western world ultimately casts both the moderate and the radical as racialized identities which emerge from a hegemonic construct of whiteness, within which white/Western standards of “rationality” remain the yardstick against which Muslimhood is measured and evaluated (5). In struggling to escape the binaries of loaded, polarizing terms like “rational” and “progressive,” we therefore became collectively aware of a paradox that pushed us to consider how the repertoire of (English) scholarly terms limits us as researchers of the phenomenon of Islamophobia. At one point, Author 2 and Author 3 broke off the discussion in English to discuss a range of other possible terms we could use in other languages which they shared, to avoid the traps we came to perceive in using terms like “progressive,” “rational,” “secular,” “modern.” We did not come to a resolution on alternative terms outside of the English scholarly repertoire we could use in this particular study, but tried to be cognizant of these dilemmas in crafting the examples for each dimension, and took this issue on board as a significant point that requires further attention for future studies in this area. Rather than using the language of “rationality” when developing an example of “open” views on Frame 3 (inferior/different), therefore, we 195 developed the phrase “outlining of capacity of Muslim individuals and communities for excellence in a variety of fields and emphasis on the ability to make significant contributions to effective problem solving.” This vignette therefore illuminates a significant issue that we identify in defining and measuring Islamophobia: if we try to define what is not Islamophobic, we also need to avoid the trap of constructing only Westernized/secular/“rational” Muslims as “acceptable”: i.e., those conforming to Western liberal expectations. As Morsi (2017) observed, we can see in academic language a foreclosure of how “the Muslim” can even be defined: does defining an image of Islam and Muslims that is not Islamophobic force all Muslims to don the “mask of the moderate” (152)? The definition of Muslimhood through reductive terms that rely upon Western conceptions of “objectivity” and “acceptability” can create reductionist categorization that silences and excludes many of the ways in which Muslim identity is expressed. This can include those who do not consider themselves to be practicing Muslims and do not automatically consider Islamophobia as a form of oppression that impacts upon them, yet are nevertheless categorized by others as “Muslim” through the process of racialization arising from identity markers such as their name, phenotype, or clothing. The reductionist label of the “ideal moderate” is language that is not only used in academia, but also within mediatized discourses which see Muslim participation in public debate strictly controlled and policed, and also impacts upon communication between and within Muslim communities in the West (Anderson and Gatwiri, 2020). Such incidents and discussions in our research process therefore led us to delve deeply into our own positionality in regard to the issues we are seeking to describe and define. We are scholars from different cultural and disciplinary backgrounds: an Australian-born white female social scientist; a female critical feminist theorist from Iran; and a male social scientist from Afghanistan who is now an Australian citizen. In comparison to the wider initial pool of coders, we had in common more experience in researching matters related to Islam and Muslim cultures and societies, and more time spent informally discussing experiences and observations of Islamophobia in social and political discourses, yet still due to our different positionality within Australian society we had distinct and at times highly divergent perspectives on how to describe and define Islamophobia effectively. Based on our experience, we contend that those who have engaged with the material critically for longer periods are more able to observe and expose concealed and mainstreamed tropes. Furthermore, the Australian context renders expressions and articulation of Islamophobia unique, often in relation to its attitude toward immigration from non-Western nations, which has implications for researchers with divergent backgrounds. This is an important issue for future media research to consider; a recent study of transnational news coverage in Australia, for example, found that the same media articles were evaluated differently by research assistants depending on their background. When read by an Australian-born journalist, the news stories were considered predominantly neutral (13.5% considered negative in tenor). In contrast, the same articles evaluated by a research assistant “born overseas from a culturally diverse background” reported that “over 35% of the stories seem to evoke negative emotions because of a lack of balance in the information included in the stories and sources quoted in the story” (Rodrigues and Paradies 2017, 9). Australian anxieties over Muslim migration and settlement have led to increasingly extreme securitization policies over the past two decades, including mandatory off-shore detention of asylum seekers (Boochani 2019). Such policies are indicative of the paradoxical position that Australia contradictorily occupies, in locating itself in the “Anglosphere” as a settler-colony while maintaining an ambivalent attitude, of simultaneous fear and hope, toward its actual geographic location, the Asia-Pacific region (Johnson, Ahluwalia, and 196 ISJ 6(2) McCarthy 2010, 59). These tensions were highly evident in the samples of articles analyzed within this study, with a high number of articles focused upon securitization in regard to Muslims from countries in the broad Asian region, including Turkey (as detailed in the analysis featured above), Indonesia and Malaysia. Australia therefore remains highly influenced by prevailing attitudes from the West even as, Ann Capling reminds us, it is more “engaged with Asia than ever before” (2008, 601). This historical and cultural backdrop, we maintain, ought to inform attempts at theorizing Australia’s position vis-à-vis Asia and implications for applications of Australian-based research in the emerging field of study of Islamophobia within Asia, particularly in regard to the study of media representations. Ahmed and Matthes (2015) found that studies on media representations of Islam and Muslims from the Asian region constituted less than 15% of total research in a large meta-analysis, with Australian research constituting a similar proportion of total research (12.5%) despite the comparative differences in population and geographical area. They concluded that there was “an urgent need to break the existing geographical bias and investigate under-explored countries” such as in the Asian region, further advocating for crossnational comparative work and cross-disciplinary approaches to such research. Osman (2017) similarly concluded that there is an urgent need for further research on the expression and mediatization of Islamophobia in Asian contexts. We contend that research utilizing critically reflective teams of diverse disciplinary and national backgrounds, as in our study, presents a strong opportunity to form the basis for future research teams capable of addressing some of the issues identified by Ahmed and Matthes. What our research demonstrates is that through collaborative dialogue, we were able to synthesize our diverse perspectives and produce a set of coding instructions that were appropriate, within our context, for relatively reliable and valid measurement of Islamophobia within a given set of media texts. Future research in this area, we suggest, should similarly work to establish the contextual and positional relevance of the instrument. As Bleich (2012) noted, there are substantial difficulties apparent in attempts to develop a “meaningful aggregate index of Islamophobia” which would be appropriate for multiple geographical, social and disciplinary contexts. However, we do contend that in this study we have made significant progress toward the goals of evolving a promising approach to developing “a common conceptual language and more consistent tools for measuring Islamophobia” as Bleich advocated for (46). CONCLUSION AND FUTURE DIRECTIONS Understanding the prevalence of anti-Muslim discourses within media reporting has been a significant preoccupation of researchers broadly engaged with Islamophobia for good reason. The media has repeatedly been singled out as a social institution that contributes significantly to the promulgation and normalization of Islamophobic praxis and discourses, and has been clearly linked to harm to Muslim individuals and communities. This is particularly well documented for Muslim minorities living permanently or temporarily in Western societies. Yet, in the focus on the media, other areas within which Islamophobic discourses are also promulgated and circulated have received less attention, for example, studies analyzing Islamophobia within policy documents (Jivraj and de Jong 2011) or life writing like (auto) biography and literary memoirs (Jelodar et al. 2013). Therefore, one of the aims for future research building upon this project will be to explore possibilities for utilizing an “Islamophobia Index” for the evaluation of anti-Muslim discourses in non-mediatized texts. 197 Future research on media representations of Islamophobia—in its increasingly opaque, mainstreamed, and institutionalized forms—should focus on how to address and improve representations. As recent debates (see Anderson 2015b; Dunn 2019; Ewart and O’Donnell 2018) on progressing this direction of research indicate, “intervening and changing media representations will be an agonistic politics, requiring an ongoing struggle” (Dunn 2019, 269). Part of this struggle will include evolving debates on not only how to effectively identify and evaluate Islamophobic discourses and representations, but also to be able to measure and evaluate progress in meeting the aims of improving representations. One important point that our attempt to test the reliability and validity of an instrument for this purpose was the need which we perceived for ensuring that this work is done by diverse teams of researchers, who take time to explore divergent understandings of expressions of Islamophobia in terms of the positionality of those enacting and experiencing its manifestations, in order to effectively research the phenomena and refine methodologies for its measurement. The traditional positivist approach to research has emphasized the separation of the personal from the process of research and the cultivation of an omniscient, detached researcher position. In our experience, this approach is not suited to developing effective methodologies for the incisive study of Islamophobia. In putting forward this critical evaluation and development of one such methodological approach, we have sought to make an impactful contribution to future research in this area. Given the pervasiveness of Islamophobic representations within media, its many forms of communicative practices continue to require critical (and innovative) methodological approaches. As such, it is imperative that future directions for research on Islamophobia and the media include studies of evidence-based best practice for the education and training of media professionals, and audience studies to evaluate the impact of Islamophobic media representations. Research in the former of these fields is gaining traction, as represented particularly by the recent groundbreaking Reporting Islam research project based in Australia (Ewart and O’Donnell 2018), but audience studies are an area where there is still a paucity of recent, relevant work (Saeed 2007). In these fields of research, just as much as in descriptive content and discourse analytic studies, there is still nevertheless a need for clearly recognized standards and tools for the evaluation and comparison of Islamophobic discourses within media texts. Beyond this, there are other potential applications for such standards, such as enabling a clear articulation of standards for best practice in media coverage on issues related to Islam and Muslims, or the comparison of audience responses to texts deemed to include higher or lower levels of Islamophobic discourses. Future research could also include exploring the potential for utilization of the Index in fields other than communication studies. Studies on race and whiteness and the ways in which such constructs impact upon the expression and experience of Islamophobia, and are intrinsically involved in anti-Muslim cultural and institutional practices, could utilize an adapted form of the Index. In applying such an Index in different disciplinary fields, however, the approach of characterizing representations of Islam as positive, negative, or neutral may face some limitations in discovering the implicit meanings hidden in different linguistics layers such as in more metaphorical texts. That said, the researchers are aware of these limitations and in future research hope to be able to address them through a multidisciplinary approach and encouraging scholars in different fields with various research backgrounds to contribute to future iterations of this project. As such, future research in this field would benefit from critical engagement with transnational teams of scholars. APPENDIX A—FRAME DESCRIPTORS (ADAPTED FROM RUNNYMEDE TRUST 1997, 5) Frame Descriptor Closed views of Islam Open views of Islam 1. Monolithic/diverse “Islam” represented as a single monolithic bloc, static and unresponsive to new realities. “Islam” represented as separate and the Other—(a) not having any aims or values in common with other cultures (b) not affected by them (c) not influencing them. Superiority of “the West” emphasized through representations of Islam as inferior to “the West”—especially as barbaric, irrational, primitive, and sexist. “Islam” represented as violent, aggressive, threatening, supportive of terrorism, engaged in “a clash of civilizations.” “Islam” represented as a political ideology used for political or military advantage. Critiques of “the West” are rejected out of hand, particularly when the criticism is voiced by Muslims. Hostility toward “Islam” used to justify discriminatory practices toward Muslims and exclusion of Muslims from mainstream society. Anti-Muslim hostility accepted as natural and “normal.” “Islam” represented as diverse and progressive, with internal differences, debates and development. “Islam” represented as interdependent with other faiths and cultures—(a) having certain shared values and aims (b) affected by them (c) enriching them. “Islam” represented as distinctively different, but not deficient, and as equally worthy of respect. 2. Separate/interacting 3. Inferior/different 4. Enemy/partner 5. Manipulative/ sincere 6. Criticism of West rejected/considered 7. Discrimination defended/criticized 8. Islamophobia seen as natural/problematic “Islam” represented as an actual or potential partner in joint cooperative enterprises and in the solution of shared problems. “Islam” represented as a genuine religious faith, practiced sincerely by its adherents. Critiques of “the West” and other cultures are considered and debated. Debates and disagreements with “Islam” do not diminish efforts to combat discrimination and exclusion. Critical views of “Islam” subjected to critique and anti-Muslim hostility framed as problematic. APPENDIX B—FRAME EXAMPLES (DEVELOPED BY AUTHORS IN RECODING EXERCISE) Frame Descriptor Closed views of Islam and Muslims Open views of Islam 1. Monolithic/ diverse Examples “Islam” represented as a single monolithic bloc, static and unresponsive to new realities. —Diverse communities or states referred to reductively and defined solely through reference to the Muslim aspects of their identity, such as “the Muslim nation Turkey”; —Muslims or Muslim majority states represented as closed or resistant to change, development and non-Muslim ideals and values; —Radical or extremist groups implied as representing or collectively endorsed by all or most Muslims; —Muslim Majority nations homogenized into a binary of “Sunni states” and “Shiite states,” e.g., “Middle Eastern Conflict” reductively defined as a sectarian conflict; —Homogenization of aspects of Islamic faith, practice and jurisprudence, such as reification of “Sharia” (implied to be a consistent, universally accepted code or dogma within a monolithic Islam). “Islam” represented as separate and the Other—(a) not having any aims or values in common with other cultures (b) not affected by them (c) not influencing them. —Islam represented as categorically different to other religions or cultures, which cannot be constructively compared or contrasted with other world religions; —Discounting or refutation of the idea that Muslims might hold overlapping values or morals with other cultures; —Sharp distinction drawn between “Judaeo-Christian” or “Christian” culture and Islamic cultures. —Imputation that religion is the identity marker of highest salience for all Muslim individual or Muslim Majority states and that it supersedes all other aspects of identity including ethnicity, language, gender, class; —Discounting or not mentioning indigenous or grassroots movements or institutions which express shared values such as Islamic feminism, religious secularism, Islamic democratic models, reformed legal codes; —Islam presented as incompatibly with or fundamentally oppositional to secularism. “Islam” represented as diverse and progressive, with internal differences, debates and development. —Emphasis on diversity of beliefs, ethnicity, language and other markers of cultural difference within Muslim communities and states; —Muslim individuals, communities or Muslim majority states represented as open to and interacting with a wide diversity of actors, ideas and values from other cultures; —Clear distinction made between the beliefs and actions of some individuals or organizations and that of the majority of global Muslim communities; —Complex socio-political history of different Muslim majority states and communities acknowledged or referenced; —Different components of Islamic belief and practice referred to appropriately and contextually, e.g., acknowledging the diversity of opinions and practices within different Islamic law codes or different approaches such as that of Ahmadiyya, Sufist, or diasporic communities. “Islam” represented as interdependent with other faiths and cultures—(a) having certain shared values and aims (b) affected by them (c) enriching them. —Islam represented as a distinct religious tradition, which is nevertheless comparable to and has commonalities with other major world religions; —Affirmation that many Muslims hold values or morals which are shared by other cultures, such as a commitment to democratic principles or respect for human life; —Acknowledgment that Islam will be just one aspect of the identity of an individual, community or state, which will vary in degree of salience at different times and in different places; —Exploration of connections between different Abrahamic traditions and other world religions; —Reference to or quotations from Muslim individuals or organizations articulating shared common values and representing efforts to develop and promote indigenous or grassroots movements which are capable of enriching local and global cultures; —Shared aspirations of Muslims and people of other backgrounds emphasized, e.g., education, health, justice. 2. Separate/ interacting Examples (Continued) (Continued) Frame Descriptor Closed views of Islam and Muslims Open views of Islam 3. Inferior/different Islam represented as inferior to “the West”—barbaric, irrational, primitive, sexist. —Muslims and Muslim communities exoticized and described exclusively in ways that emphasize visible markers of “primitive” or “tribal” identity, such as traditional clothing, hair or head coverings; —Representation of localized cultural practices such as genital mutilation and honor killings or mainstream ideologies such as the patriarchal oppression of women as characteristically Islamic; —Implied low expectations outlined for Muslim individuals or nations, e.g., Muslims represented as unable to behave or think in a thoughtful or measured way and contrasted with representations of individuals and states from “the West” as “rational,” “mature,” “peaceful” etc.; —Lack of acknowledgment of complex geopolitical realities affecting Muslim communities; —Assumption that dysfunctional or corrupt political regimes are intrinsic to Muslim majority nations and generalized to represent the whole society and its systems; —Excessive use of demonizing, victimizing and emotive language to portray the inferiority of Muslim subjects. “Islam” represented as violent, aggressive, threatening, supportive of terrorism, engaged in “a clash of civilisations.” —Muslim individuals or states represented as incapable of genuine partnership or not mentioned as active and effective partners in local and global matters such as climate change, conflict resolution processes (Palestine–Israel, for instance), or gendered violence; —Excessive emphasis on terrorists/fanatics/extremists as representatives of Islam and diversity of Muslim communities and contradictory opinions not mentioned or discounted; —Excessive use of demonizing, victimizing and emotive language, for example, to emphasize the impacts of terrorist acts and to suggest a causative relationship between Islam and violent political acts; —Repeated references to episodes of historical conflict implying “age-old” irreconcilable differences, e.g., the Crusades. Islam represented as distinctively different, but not deficient, and as equally worthy of respect. —Wide array of Muslim individuals and communities represented, such as quotes from or descriptions of Muslims in diverse settings such as contemporary urban environments. —Localized cultural practices discussed in a contextually appropriate manner which avoids the implication that certain practices are “inherently Islamic”; —Outlining the capacity of Muslim individuals and communities for excellence in a variety of fields and emphasis on the ability to make significant contributions to effective problem solving; —Recognition of geopolitical realities affecting Muslim communities, including the negative legacy of colonialism and military intervention in Muslim majority regions of the world. —Nuanced discussions of socio-historical context influencing the preponderance of poverty, corruption and conflict within Muslim majority regions of the world. —Use of objective and neutral language and descriptors to describe what has been done and could be done by people. Examples 4. Enemy/partner Examples “Islam” represented as an actual or potential partner in joint cooperative enterprises and in the solution of shared problems. —Muslim individuals or states represented as engaged in genuine partnerships or as active partners in solving shared problems, such as seeking to modify legal codes in order to tackle gendered violence; —Description of or quotations from Muslims from a broad array of backgrounds and professions, engaged in activities and projects beneficial to their immediate and broader community; —Use of objective and neutral language to describe what has been done and could be done by people and how they are forming diverse partnerships, coping with adversity, or succeeding with different ventures; —Reference to historical and contemporary episodes or epochs illustrating peaceful coexistence or engaged debates between Muslim and non-Muslim individuals or communities. 5. Manipulative/ sincere Examples “Islam” represented as a political ideology used for political or military advantage. —Reductive representations of complex conflicts, such as the implication that conflicts in the Middle East are predominantly proxy wars perpetrated by Iran or Saudi Arabia; —Explicit or implied claims that Islam is not a genuine religious faith, such as emphasis on the duplicity of Muslims implying Muslims are concealing political intentions behind the cloak of religious observances, such as suspicion or hostility toward halal certification; —Repeated emphasis on Islamic political movements in the absence of mention of lived experiences of faith; —Ahistorical references to contemporary political landscapes and expression of suspicion toward social movements of protest characterized as inherently Islamic. 6. Criticism of West rejected/considered Critiques of “the West” and rejected out of hand, particularly when the criticism is voiced by Muslims. —Defense of policies that can be unjust and oppressive and impact disproportionately on Muslims, including military interventions, sanctions, anti-terrorism regimes; —Attacks on individuals and organizations which criticize the foreign policy of hegemonic Western states; —Existence of Islamophobia questioned and the relationship of Islamophobia to racism denied; —Lived experiences of Islamophobia denied and implied to be unfounded —“Anger” or “bitterness” of Muslim individuals or communities over the prejudice and discrimination they claim to have experienced emphasized and framed as irrational and unfounded. “Islam” represented as a genuine religious faith, practiced sincerely by its adherents. —Complex and contextualized discussions of the background of conflict situations, which attempt to disentangle the contribution of religious difference from other aspects influencing the situation; —Description of or quotes from varied members of Muslim communities outlining the lived experience of Muslims as members of diverse faith communities; —Discussion or description of different aspects of Islamic faith important in the day-to-day life of Muslims living in diverse Muslim communities around the world; —Attempts to contextualize and provide an appropriate discussion of political movements and social movements; —Quotes from political leaders or members of the bureaucracy of Muslim majority nations or Islamic organizations balanced with quotes from more representative and “ordinary” members of faith communities. Critiques of “the West” and other cultures are considered and debated. —Provides an opportunity for criticism of policies that impact unfairly and disproportionately on Muslim individuals, communities, and states to be heard; —Consideration given to concerns, especially when voiced by Muslims, that foreign and domestic policy of hegemonic Western states has serious negative impacts upon Muslim communities; —Serious weight given to evidence of the extent of Islamophobia and its impacts upon Muslim individuals and communities; —Attempts to explore and seriously consider purported links between experiences of Islamophobia, racism, marginalization and their relationship to political movements and “radicalization.” (Continued) (Continued) Frame Descriptor Closed views of Islam and Muslims Open views of Islam 7. Discrimination defended/criticized Example Hostility toward Islam used to justify discriminatory practices toward Muslims and exclusion of Muslims from mainstream society. —Justification of policies that target whole Muslim communities, such as refugee and immigration bans or antiterrorism regimes, through reference to terrorist attacks by individuals or small unrepresentative groups; —The dehumanization of Muslims endorsed as a way to justify discriminatory practices; —Hostility and attacks directed toward measures that could support better social cohesion and inclusion, e.g., better legal protections against Islamophobia; —Denial of racialization as a component of Islamophobia. Anti-Muslim hostility accepted as natural and “normal.” Debates and disagreements with “Islam” do not diminish efforts to combat discrimination and exclusion. —The existence of individuals and groups which justify violent acts in the name of Islam acknowledged, but attempts to distinguish between policies that seek to contain legitimate threats and policies which affect Muslims and Muslim communities indiscriminately; —Attempts to source a diversity of opinions on contentious matters within specific Muslim communities; —Highlighting of efforts to tackle discrimination and prejudice; —Recognition of the impact of racialization in expressions of Islamophobia directed toward Muslims and perceived Muslims or perceived expressions of Islam. Critical views of “Islam” are subjected to critique, lest they be inaccurate and unfair, and anti-Muslim hostility framed as problematic. —Anti-Muslim sentiments and generalizations uncritically discussed or repeatedly quoted without qualification; —Direct reference to anxiety about Muslims or Islam or terrorism as “normal,” “mainstream,” or “commonsense”; —Downplaying of claims of discrimination. —Attempts to critically engage with Anti-Muslim sentiments and generalizations such as Islamophobic statements from prominent politicians or community members; —Self-reflection by author or quoted sources evident, which illustrate critical reflection on biases and latent prejudice; —Critical engagement with key pillars of Islamophobia, such as questioning the purported links between Islam and violence. 8. Islamophobia seen as natural/ problematic 203 ENDNOTES 1 These data points were of particular interest as during the 2004 Federal Election campaign Islamophobic rhetoric deployed in political discourse was relatively unchallenged by mainstream Australian media, but the reporting on the 2007 Federal Election campaign featured more critical lenses being applied to such rhetoric, therefore providing key points of diverging discourses in political discourse worthy of deeper analysis (Anderson 2015). 2 Coders were recruited from an informal research network at an urban Australian university composed of early career researchers interested in exploring different ways of evaluating Islamophobia within a variety of texts. Coders were all from different disciplinary backgrounds (including International Relations, Human Geography, Linguistics, Peace and Conflict Studies, and Comparative Literature), with varying levels of experience researching matters relevant to the study of Islam and Muslim societies. 3 This is a departure from the rating scale utilized by Anderson (2015), which utilized a bipolar 9-point Likert style rating scale to assign descriptors ranging from “extremely closed” (aggregate Index score of –8 > –7) through to “extremely closed” (aggregate Index score of 7 > 8). The nine different descriptors used in this rating scale did not enable clear definitions of agreement on the degree of open and closed views in given units of analysis which made it more difficult to determine acceptable levels of statistical agreement between different coders, thus leading to our decision to utilize just three divisions for aggregate scores. REFERENCES Abdalla, Mohamad, Jacqui Ewart, and Halim Rane. 2010. “A Way Forward?” In Islam and the Australian News Media, Halim Rane, Jacqui Ewart, and Mohamad Abdalla eds. Melbourne: Melbourne University Press: 228–237. Abdel-Fattah, Randa. 2018. Islamophobia and Everyday Multiculturalism in Australia. London: Routledge. Abdel-Fattah, Randa. 2019. “Managing Belief and Speech as Incipient Violence: ‘I’m Giving You the Opportunity to Say That You Aren’t.’” Journal of Policing, Intelligence and Counter Terrorism, 14, no. 1: 20–38. Abdel-Fattah, Randa, and Mehal Krayem. 2018. “Off Script and Indefensible: The Failure of the ‘Moderate Muslim.’” Continuum, 32, no. 4: 429–443. Abu-Ras, Wahiba, Zulema E Suárez, and Soleman Abu-Bader. 2018. “Muslim Americans’ Safety and Well-Being in the Wake of Trump: A Public Health and Social Justice Crisis.” American Journal of Orthopsychiatry, 88, no. 5: 503–515. Ahmed, Saifuddin, and Jörg Matthes. 2017. “Media Representation of Muslims and Islam from 2000 to 2015: A Meta-Analysis.” International Communication Gazette, 79, no. 3: 219–244. Akbarzadeh, Shahram, and Bianca Smith. 2005. The Representation of Islam and Muslims in the Media. Melbourne: Monash University. Allen, Chris. 2010. Islamophobia. London: Ashgate. Allen, Chris. 2014. “Exploring the Impact of Islamophobia on Visible Muslim Women Victims: A British Case Study.” Journal of Muslims in Europe, 3, no. 2: 137–159. Anderson, Leticia, and Kathomi Gatwiri. (forthcoming). On “getting Yassmined”: How the Australian Media Polices the Bodies of Women of Colour. In Minority Women And Western Media: Challenging Representations and Articulating New Voices, M. Bashri and S. Ahmed eds. Lanham, MD: Lexington Books. 204 ISJ 6(2) Anderson, Leticia. 2015a. “Countering Islamophobic Media Representations: The Potential Role of Peace Journalism.” Global Media and Communication, 11, no. 3: 255–270. Anderson, Leticia. 2015b. “Reflecting on Representation [Book Review].” Australian Journalism Review, 37, no. 1: 195–196. APPG. 2018. Islamophobia Defined: The Inquiry into a Working Definition of Islamophobia. All Party Parliamentary Group on British Muslims, London. https://static1. squarespace.com/static/599c3d2febbd1a90cffdd8a9/t/5bfd1ea3352f531a6170c eee/1543315109493/Islamophobia+Defined.pdf. Boochani, Behrouz. 2018. No Friend But The Mountains. Sydney: Picador. Bleich, Erik. 2012. “Defining and Researching Islamophobia.” Review of Middle East Studies, 46, no. 2: 180–189. Brown, Katherine E. 2018. “Introduction: Radicalisation and Securitisation of Muslims in Europe.” Journal of Muslims in Europe, 7, no. 2: 139–145. Brown, Lorraine, Joanne Brown, and Barry Richards. 2015. “Media Representations of Islam and International Muslim Student Well-Being.” International Journal of Educational Research, 69: 50–58. Bull, Melissa, and Halim Rane. 2018. “Beyond Faith: Social Marginalisation and the Prevention of Radicalisation among Young Muslim Australians.” Critical Studies on Terrorism: 1–25. Cainkar, Louise, and Saher Selod. 2018. “Review of Race Scholarship and the War on Terror.” Sociology of Race and Ethnicity, 4, no. 2: 165–177. Ann Capling, Ann. 2008. “Twenty years of Australia’s engagement with Asia.” The Pacific Review, 21, no. 5, 601–622. Dabashi, Hamid. 2011. Brown Skin, White Masks. London: Pluto. Duderija, Adis, and Halim Rane. 2019. Islam and Muslims in the West: Major Issues and Debates. Cham: Palgrave Macmillan. Dunn, Kevin M. 2019. “Reporting Islam: International Best Practice for Journalists [Book Review].” Journal of Intercultural Studies, 40, no. 2: 269–274. Esposito, John, and Derya Iner, eds. 2019. Islamophobia and Radicalization: Breeding Intolerance and Violence. Cham: Palgrave MacMillan. Ewart, Jacqui, and Kate O’Donnell. 2018. Reporting Islam: International Best Practice for Journalists. London: Routledge. Fox, Jonathan, and Yasemin Akbaba. 2015. “Securitization of Islam and Religious Discrimination: Religious Minorities in Western Democracies, 1990–2008.” Comparative European Politics, 13, no. 2: 175–197. Fray, Peter. 2004. “Europe Cautious in Talking Turkey.” The Sydney Morning Herald (Sydney), Fairfax, October 8, 2004. Freelon, Deen. 2017. “Recal for Ordinal, Interval and Ratio Data (Oir).” dfreelon.org. Last modified May 22, 2017. Accessed May 1, 2019. http://dfreelon.org/utils/recalfront/ recal-oir/. Hargreaves, Julian. 2016a. “Islamophobia: Reality or Myth?” PhD, Lancaster University. Accessed April 1, 2019. http://eprints.lancs.ac.uk/80451/1/2016hargreavesphd.pdf. Hargreaves, Julian. 2016b. “Risk and Resilience in British Muslim Communities.” Ethnic and Racial Studies 39, no. 14: 2601–2620. Hayes, Andrew F., and Klaus Krippendorff. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures, 1, no. 1: 77–89. 205 Hopkins, Peter. 2016. “Gendering Islamophobia, Racism and White Supremacy: Gendered Violence against Those Who Look Muslim.” Dialogues in Human Geography, 6, no. 2: 186–189. Iner, Derya. 2017. Islamophobia in Australia, 2014–2016. Charles Sturt University & Islamic Sciences and Research Academy, Sydney. https://arts-ed.csu.edu.au/__data/assets/ pdf_file/0009/2811960/csu-islamophobia-in-australia-report14-16.pdf. Janis, Irving. 2009. “The Problem of Validating Content Analysis.” In The Content Analysis Reader, Klaus Krippendorff and Mary Angela Bock eds. Los Angeles: SAGE: 358–367. Jelodar, Esmaeil Zeiny, Noraini Md Yusof, Ruzy Suliza Hashim, and M.M. Raihanah. 2013. “Critical Pedagogy of a Post-9/11 Muslim Memoir.” International Journal of Applied Linguistics and English Literature, 2, no. 2: 216–222. Jivraj, Suhraiya, and Anisa de Jong. 2011. “The Dutch Homo-Emancipation Policy and Its Silencing Effects on Queer Muslims.” Feminist Legal Studies, 19, no. 2: 143. Johnson, Carol, Pal Ahluwalia and Greg McCarthy. 2010. “Australia’s Ambivalent Re-imagining of Asia.” Australian Journal of Political Science, 45, no. 1: 59–74. Kabir, Nahid. 2019. “Can Islamophobia in the Media Serve Islamic State Propaganda? The Australian Case, 2014–2015.” In Islamophobia and Radicalization: Breeding Intolerance and Violence, John Esposito and Derya Iner eds. Cham: Palgrave MacMillan: 97–116. Krippendorff, Klaus. 2004. “Reliability in Content Analysis.” Human Communication Research, 30, no. 3: 411–433. Krippendorff, Klaus. 2009. “Testing the Reliability of Content Analysis Data: What Is Involved and Why.” In The Content Analysis Reader, Klaus Krippendorff and Mary Angela Bock eds. Los Angeles: SAGE: 350–357. Kunst, Jonas R., David L. Sam, and Pål Ulleberg. 2013. “Perceived Islamophobia: Scale Development and Validation.” International Journal of Intercultural Relations, 37, no. 2: 225–237. Kunst, Jonas R., Hajra Tajamal, David L. Sam, and Pål Ulleberg. 2012. “Coping with Islamophobia: The Effects of Religious Stigma on Muslim Minorities’ Identity Formation.” International Journal of Intercultural Relations, 36, no. 4: 518–532. Lean, Nathan. 2019. “The Debate over the Utility and Precision of the Term ‘Islamophobia.’” In The Routledge International Handbook of Islamophobia, Irene Zempi and Imran Awan eds. New York: Routledge: ch. 1. Love, Erik. 2017. Islamophobia and Racism in America. New York: NYU Press. Manning, Mark, and Don Munro. 2007. The Survey Researcher’s SPSS Cookbook. 2nd ed. Frenchs Forest, NSW: Pearson Education. Meer, Nasar, and Tariq Modood. 2019. “Islamophobia as the Racialisation of Muslims.” In The Routledge International Handbook of Islamophobia, Irene Zempi and Imran Awan eds. New York: Routledge: ch. 2. Morsi, Yassir. 2017. Radical Skin, Moderate Masks: De-Radicalising the Muslim and Racism in Post-Racial Societies. London: Rowman and Littlefield. O’Donnell, Kate, Jacqui Ewart, and Shannon Walding. 2018. “Perceptions and Realities: The Main Sources of Information for Non-Muslim Australians About the Religion of Islam and Muslim People.” Journalism: 1–17. https://dx.doi. org/10.1177/1464884918808658. Osman, M.N.B.M. 2017. “Understanding Islamophobia in Asia: The Cases of Myanmar and Malaysia.” Islamophobia Studies Journal, 4, no. 1: 17–36. 206 ISJ 6(2) Paul, Erik. 2014. Australia as US Client State: The Geopolitics of De-Democratization and Insecurity. London: Palgrave. Poynting, Scott, and Barbara Perry. 2007. “Climates of Hate: Media and State Inspired Victimisation of Muslims in Canada and Australia since 9/11.” Current Issues in Criminal Justice, 19, no. 2: 151–171. Rane, Halim, Jacqui Ewart, and John Martinkus. 2014. Media Framing of the Muslim World: Conflicts, Crises and Contexts. Basingstoke: Palgrave Macmillan. Richardson, John E. 2006. “Who Gets to Speak? A Study of Sources in the Broadsheet Press.” In Muslims and the News Media, Elizabeth Poole and John E. Richardson eds. London: I.B. Tauris: 103–115. Rodrigues, Usha and Yin Paradies. 2017. Transnational News and Multicultural Australia: Cultural Diversity and News in Australia. Melbourne: Deakin University. Rothwell, Nicolas. 2004. “EU Puts Turks on 10-Year Time Line.” The Australian (Sydney), News Ltd, October 7, 2004. Runnymede Trust. 1997. Islamophobia: A Challenge for Us All. Runnymede Trust, London. https://www.runnymedetrust.org/companies/17/74/Islamophobia-A-Challenge-forUs-All.html. Runnymede Trust. 2016. “Islamophobia: 20 Years on, Still a Challenge for Us All.” https:// www.runnymedetrust.org/blog/islamophobia-20-years-on-still-a-challenge-for-us-all. Saeed, Amir. 2007. “Media, Racism and Islamophobia: The Representation of Islam and Muslims in the Media.” Sociology Compass, 1, no. 2: 443–462. Samari, Goleen, Héctor E Alcalá, and Mienah Zulfacar Sharif. 2018. “Islamophobia, Health, and Public Health: A Systematic Literature Review.” American Journal of Public Health, 108, no. 6: e1–e9. Selod, Saher. 2018. Forever Suspect: Racialized Surveillance of Muslim Americans in the War on Terror. New Brunswick: Rutgers University Press. Shaver, John H, Chris G Sibley, Danny Osborne, and Joseph Bulbulia. 2017. “News Exposure Predicts Anti-Muslim Prejudice.” PloS one, 12, no. 3: e0174606.

RELATED PAPERS

RELATED TOPICS

Log In

The Islamophobia Index: Exploring the Challenges in Establishing Reliability for a Content Analysis Instrument Evaluating Islamophobia in Media Texts

The Islamophobia Index: Exploring the Challenges in Establishing Reliability for a Content Analysis Instrument Evaluating Islamophobia in Media Texts

Related Papers

RELATED PAPERS

RELATED TOPICS