Promoting Classroom Engagement Through the Use of an Online Student Response System: A Mixed Methods Analysis

1 Introduction

Student engagement has been defined as how involved students are in their learning experience and how connected they feel to their classes, peers, and their institutions (Axelson and Flick Citation2010). Given the positive associations between student engagement, course satisfaction (Swan Citation2001; Wefald and Downey Citation2009), persistence (Berger and Milem Citation1999; Kuh et al. Citation2008), and academic success (Carini, Kuh, and Klein Citation2006), initiatives aimed at fostering student engagement in classrooms have become a primary focus for higher education institutions. As more and more technology becomes integrated into the student learning experience, educators have begun to explore technology-based initiatives as a means to enhance student engagement through facilitating active learning activities.

Previous research investigating the use of technology within higher education has demonstrated that using technology can enhance student engagement (D’Inverno, Davis, and White Citation2003; Poirier and Feldman Citation2007; Blasco-Arcas et al. Citation2013). The results of the 2016 Educause Center for Analysis and Research (ECAR) Study of Undergraduate Students and Information Technology Report (Brooks Citation2016) found that students believe that technology enriches their academic experience and is critical for academic success, indicating that students have welcomed the incorporation of technology-based learning strategies. Furthermore, recent reports have found that students are requesting classes to integrate more technology (Dahlstrom and Bichsel Citation2014; Brooks Citation2016). Hence, there appears to be a need for universities to incorporate more technology into their teaching practices to satisfy the needs of today’s students. However, despite computers and technology becoming an integral part of delivering education, how best to exploit this potential to enhance student engagement remains largely unknown.

Student response systems (SRSs) are one such technology that has been promoted as having the potential to enhance student engagement in the classroom. SRSs are electronic tools which enable teachers to engage with students during a class by asking questions that can be answered instantaneously by large groups of students. Such systems are increasingly being employed by teachers and research has found that using such tools during classes can help increase student engagement (Hall et al. Citation2005; Fies and Marshall Citation2006; Caldwell Citation2007; Kaleta and Joosten Citation2007; Trees and Jackson Citation2007), partly due to the increased anonymity provided to students when responding to questions (Freeman, Blayney, and Ginns Citation2006).

Until recently, SRSs were predominately employed via handheld devices (e.g., “Clickers”) that teachers had to set up and distribute to students at the beginning of classes. Students would use the handheld devices to respond to multiple-choice questions displayed via a projector (Kay and LeSage Citation2009). However, rather than handheld devices, free online student response systems (OSRSs) are now available. In addition to the financial benefits of using these OSRSs as opposed to the “clicker” systems, these systems can be accessed by students using their own mobile devices, provided that there is Internet access. To access an OSRS, teachers and students simply log on to a website or app on their own devices (e.g., a computer, laptop, tablet, or smartphone) to interact and respond to questions in real-time via the Internet.

An example of such an OSRS is Socrative (https://socrative.com/). Socrative enables teachers to create quizzes and other educational exercises that can help guide the focus of a particular lesson as well as generate discussions with students. The software empowers teachers to monitor and assess students’ responses and progress in real time by giving them immediate feedback. This enables the teacher to identify areas that students are struggling with and adjust the pace or focus of a lesson accordingly. For example, when teaching students how to produce a piece of output for a particular statistical technique, teachers can administer a set of questions via Socrative to gauge their students confidence in producing the output before moving on to the next activity, or if the results of the questions suggest it is necessary, dedicate more time to the current activity.

A previous controlled trial using a sample of engineering students reported mixed results for the OSRS, Socrative, finding that implementing the OSRS significantly increased student engagement, but had no significant effect on student performance (Dabbour Citation2016). Much of the previous research investigating OSRSs has predominately been feasibility-based though. Such studies have reported positive results with students typically reporting OSRSs to be easy to use and have a positive impact on their learning experience through enhanced levels of engagement and performance (Coca and Slisko Citation2013; Awedh et al. Citation2014; Dervan Citation2014; Mork Citation2014; Kaya and Balta Citation2016). Although the results from these previous studies are encouraging, these studies tend to be limited in terms of their generalizability due to the use of post-test, uncontrolled research designs with non-randomized samples.

The use of Socrative has been investigated across a variety of disciplines including physics (Coca and Slisko Citation2013), physiology (Rae and O’Malley Citation2017), science (Wash Citation2014), sports management (Dervan Citation2014), computing (Awedh et al. Citation2014), English language (Kaya and Balta Citation2016), economics (Piatek Citation2014), and engineering (Dabbour Citation2016). Statistics courses are another area that may benefit from using Socrative given its potential positive effect on the student learning experience and considering that course evaluations by students taking statistics units tend to indicate poor engagement (Gladys, Nicholas, and Crispen Citation2012). To the authors’ knowledge, only one study has previously investigated the effect of Socrative specifically for statistics students. Balta and Guvercin (Citation2016) found that the final grades of students enrolled in a statistics class who chose to engage with Socrative-based learning materials prior to their exam were significantly higher than the grades achieved by students who chose not to engage with the Socrative-based learning materials. Although this result is encouraging, the use of a non-randomized, post-test design means that we cannot confirm from this study that there is a beneficial effect for using Socrative, or if the difference in exam scores was due to underlying scholastic aptitude or motivation of the students who chose to engage with the OSRS. Hence, there is a need for further research exploring the use of Socrative specifically within statistics classrooms.

It is important that OSRSs are rigorously tested to ensure that they are having the intended effect and are improving (or at least maintaining) the learning experience for students. Hence, there is a need for further, more methodologically rigorous research investigating the benefits of using OSRSs in the classroom. The aim of the current study was to investigate the integration of technology in a higher education setting. Specifically, the study aimed to examine the impact of using an OSRS (Socrative) within experimental conditions for a cohort of intermediate undergraduate statistics students. This study used both quantitative and qualitative data to assess the impact of Socrative. Firstly, a quantitative analysis was used to investigate the hypothesis that when exposed to the OSRS students would report greater levels of engagement with the curriculum content than when taught without exposure to the OSRS. In addition, a qualitative thematic analysis was used to further investigate what students did and did not enjoy about using the OSRS.

2 Methodology

The sample comprised of on-campus undergraduate students recruited from an intermediate-level statistics unit enrolled at a higher education institution located in Melbourne, Australia. The statistics unit is a core unit in the psychological sciences courses of the university and introduces students to fundamental statistical areas of research design and linear models. Specifically, the unit examines how multiple regression and analysis of variance (ANOVA) can be used to analyze data.

The experiment was based on a 2 (Time: Weeks 1–3 vs. Weeks 4–6) by 2 (pedagogical approach: Standard curriculum vs. Standard curriculum plus Socrative) mixed crossover design. Tutorials were conducted on-campus (i.e., in-person in computer labs). Tutorials are conducted once a week for the duration of the semester. A typical tutorial involves a tutor conducting a class for 120 min with a small group of students. Each of the five tutorial groups (of approximately n = 15 for each tutorial group) were randomly allocated into one of two groups: the intervention group and the control group. The intervention group’s tutorial classes utilized Socrative (in addition to the standard curriculum activities) for the first three weeks of the semester (i.e., Weeks 1–3) before engaging with just the standard curriculum (i.e., no Socrative use) for the next three weeks (i.e., Weeks 4–6) of the semester. The control group’s tutorial classes were exposed to Socrative in the opposite order (i.e., tutorial groups completed the standard curriculum without using Socrative for Weeks 1–3 before using Socrative in Weeks 4–6) (see Table 1). Hence, all students were exposed to the standard curriculum for half of the term and spent the other half of the term being exposed to the standard curriculum in addition to using Socrative. Throughout this time, all students were exposed to the same textbook and learning materials, the same tutor (author three; LT), as well as the same lecture and tutorial content (see Table 2 for a description of each week’s content). The only difference between the groups was the timing that Socrative was utilized in tutorials. An advantage of this type of design is that each participant acts as their own control, thus, reducing the influence of any confounding covariates.

When exposed to the standard curriculum, students were presented with a statistical problem at the start of their tutorial sessions. This problem would relate to the content that was covered during an earlier lecture. The students would then work alongside their tutor to complete the presented problem. When students were exposed to Socrative, tutorials operated in a similar manner, however, Socrative was used in conjunction with the standard curriculum as a means to evaluate student learning. When students were presented with the initial question to solve, they were also asked to use Socrative to anonymously input their perceived ability in solving said problem (e.g., “Please rate your current knowledge of interpreting correlation output”). Students were instructed to use their computer to login to Socrative using a “room name” supplied by the tutor via the Student Socrative website (https://b.socrative.com/login/student/). The responses were instantaneously combined and displayed as a bar chart for the entire class to see via a projector (see Figure 1). The session would then proceed as per the standard curriculum, with the students working alongside the tutor to solve the problem. At the end of the tutorial session, the students were asked to complete the same Socrative question from the start of the tutorial, with the anonymous aggregate totals being displayed for the whole class to compare. Figure 1 provides an example of a question given to students during one of the tutorials. This example shows that 63% of students responded as having “Fair” or “Excellent” knowledge of the concept about to be tested in the tutorial and increasing to 94% responding “Fair” or “Excellent” after the tutorial. Similar results were observed for this type of question for each tutorial in this study. Socrative questions were created prior to each class and the same questions were used in each class that was being exposed to Socrative during that particular week (see, e.g., Appendix A questions).

Fig. 1 Comparing student’s perceived knowledge (pre-post) using Socrative.

Display full size

One week prior to their first tutorial, participants were invited to complete an initial survey consisting of measures that examined academic engagement. Academic engagement was measured using an adapted version of the 9-item Utrecht Work Engagement Scale (UWES-9) (Schaufeli, Bakker, and Salanova Citation2006). Participants rated each item on a 7-point Likert scale ranging from 1 (Strongly Disagree) to 7 (Strongly Agree). Scores were summed to produce a total engagement score with higher scores indicating greater levels of academic engagement. The UWES-9 has demonstrated good internal consistency and test-retest reliability (Schaufeli, Bakker, and Salanova Citation2006; Balducci, Fraccaroli, and Schaufeli Citation2010) and has been adapted to measure academic engagement in previous studies (Salmela-Aro et al. Citation2009).

The students then completed three weeks of either the Socrative or non-Socrative tutorial sessions, followed by a post-intervention survey consisting of the same measure described above (i.e., the academic engagement survey). Participants in the study group were also asked to respond to three open-ended qualitative items on the post-intervention survey. These items were: (1) What did you enjoy about using Socrative?; (2) What did you dislike about Socrative?; and (3) If you could change or improve anything about Socrative what would it be?

All students were required to complete the ethics-approved intervention activities during their respective tutorials, however, participation in the pre-post surveys was completed voluntarily. Any student who did not consent to participate in the research was not required to complete the surveys and their data was not included in the analysis. In total, eighteen valid survey responses were acquired for analysis. This research was carried out according to protocols approved by Swinburne University of Technology Human Research Ethics Committee (Project number: 2017/144).

3 Results

4 Discussion

The use of OSRSs is increasing within tertiary education providers with practice typically leading research on these tools. To bridge the gap between research and practice concerning the integration of technology in higher education, this study investigated the use of an OSRS (Socrative). The quantitative results provided insufficient evidence for an increase in academic engagement scores after being exposed to Socrative. However, the qualitative feedback received from students on the use of Socrative was positive.

The themes that emerged in response to the open-ended questions suggest that Socrative may be an appropriate tool to integrate more technology-based learning strategies that satisfy student requests for more interactive lessons (Dahlstrom and Bichsel Citation2014; Brooks Citation2016). Overall, students appreciated the introduction of novel active learning strategies into tutorial classes and perceived Socrative as having a positive impact on engagement by facilitating a fun and unique learning experience. While some in higher education are quick to dismiss such affective reactions to novel approaches to teaching as fun, it should be remembered that getting students enthused about learning statistics is a significant barrier that statistics educators are regularly faced with. Furthermore, given the challenges to fostering engagement in statistics classes due to students typically holding negative attitudes toward the discipline and perceiving the subject as an obstacle in the way of attaining their degree (Perney and Ravid, Citation1991; Gal and Ginsburg Citation1994), these results suggest that Socrative has potential to overcome issues specific to statistics units. This is encouraging news for educators who might be looking for novel ways to engage their students while delivering statistical-based content.

The qualitative findings also indicate that Socrative provides a welcomed opportunity for students to assess their perceived competence of course content learnt during class, and suggests that students appreciate being able to determine their level of comprehension in comparison to other students. This feedback was also beneficial for the teacher running the tutorial. The feedback provided from questions similar to the example shown in Figure 1 assisted the teacher to implement Just-in-Time Teaching (JiTT) by fine-tuning the tutorial activities to better meet the students’ needs. Using Socrative in this way has the potential for increasing the effectiveness of learning during tutorials.

This suggests that Socrative may be a worthwhile tool, particularly for statistics educators to consider using in online courses where students are often isolated from one another and are unable to make any comparisons between themselves and their peers. For example, online courses with a synchronous component (e.g., a weekly “live” online lecture) might consider using Socrative to run polls or quizzes, much like we did in our on-campus classes, to overcome isolation. Furthermore, the sense of anonymity afforded by SRSs (see Freeman, Blayney, and Ginns Citation2006) which also apply to OSRSs such as Socrative, may also see students more willing to express their opinions and participate in online class discussions. The novelty factor reported by students in the current study may also be true for online students which could also help with engagement in online classes. Future research should consider testing OSRSs like Socrative in online learning environments to investigate whether these tools are beneficial for online students and educators.

Students also reported that Socrative was easy to use. Given that modern students find browsing and navigating web pages to be a relatively simple task, it was to be expected that no major complications would arise from using a browser-based program. However, some students suggested that they would have preferred to engage in Socrative activities via an app rather than the web-browser. Socrative does offer an app that can be used by students in the same way that the web-browser is used. As such, it is suggested that educators consider enabling students to engage in Socrative via either the web-browser or the app.

The results from the qualitative analysis were not all positive, however, with some students questioning the relevance of the Socrative-based activities. This is a common critique noted in previous research investigating student response systems (see Aljaloud et al. Citation2015). This is a concern as students who find an activity irrelevant may quickly lose interest in the task and their attitudes toward the subject may become negative (Osborne, Simon, and Collins Citation2003). However, this problem is not unique to the use of SRSs with students in general desiring authentic learning activities that are applicable to the real world (Maina Citation2004).

Students did offer suggestions for how Socrative could be better used in the classroom, suggesting for it to be used as a revision tool. Previous research has suggested that using Socrative as a revision tool has the potential to help students revise and achieve higher exam marks (Balta and Guvercin Citation2016). Hence, in addition to using Socrative to assess student mastery of the content, educators should also consider creating multiple choice quizzes via Socrative assessing student knowledge relevant to each lesson in preparation of upcoming assessment tasks.

The present study was not without its limitations. Firstly, due to the small sample size, the inferential statistics should be interpreted with caution. Despite the researchers’ best efforts to maximize participation (e.g., discussion boards posts, E-mails, lecture visits), overall, students did not appear interested in participating in the research itself (i.e., completing the questionnaires), hence, the small sample size. This is a common limitation amongst studies investigating OSRSs (e.g., Coca and Slisko Citation2013; Awedh et al. Citation2014; Dervan Citation2014; Mork Citation2014; Balta and Guvercin Citation2016; Dabbour Citation2016; Kaya and Balta Citation2016).

A further limitation was the timing of the research. Students were only exposed to Socrative for three weeks of tutorial classes. This may not have been a long enough time for the OSRS to have an observable impact on engagement. Furthermore, students were administered the pre-post surveys within the first month of the teaching period. This is when engagement levels for students are typically at their highest (Stewart, Stott, and Nuttall Citation2011). Further study is needed to investigate the use of Socrative across multiple academic units (for a longer duration and across multiple time points) using large and heterogeneous samples to confirm a beneficial effect for using Socrative and for the results to be generalizable to the wider tertiary education sector. In addition to academic engagement, future research should look to quantify the effect of using OSRSs on student learning and course satisfaction.

By taking advantage of new educational technology, teachers can create a more active learning environment that assists students to have an enhanced learning experience (D’Inverno, Davis, and White Citation2003; Poirier and Feldman Citation2007; Blasco-Arcas et al. Citation2013). Hence, investigating and employing technology-based strategies that promote student engagement is an important initiative, especially when it comes to units like statistics, that can present educators with difficulties in regards to engaging students in the classroom. Taken together, the quantitative and qualitative results from the current study outline a clear path for future OSRS-related use in teaching and research on the effective use of these tools. It is hoped that the results from this study will encourage educators seeking to foster greater student engagement to consider incorporating OSRSs, like Socrative, into their teaching practices. However, before this technology can be widely adopted, further research is needed to confirm the presence and magnitude of the effect of OSRSs on the student learning experience. Further research is also needed to determine if and to what degree these improvements in engagement impact students’ performance as well as their attitudes and interest in statistics and related career pathways.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.